I'm not sure how to answer this.
We planted one in our back yard, and it grew beautifully up an oak tree, and deposited all its fruit in the neighbor's yard before dying.
A bird or squirrel must have dropped some seeds in our front yard, as this one was a volunteer which attached itself to a nearby tree and started giving fruit last year, quite a few years after it appeared. because all our citrus trees have citrus greening and aren't producing fruit, we have been casual about feeding them, and forgot to feed the passion fruit, which we decided would like citrus food, so the fruits aren't as full as we would like, but they have great flavour.
So no, not hard to grow, but they choose their place. You should be able to grow one where you are - we regard the $2.99 fruit we find in the store as a seed packet. Just remember it climbs like crazy. Let me know if you want some seed from last year's crop - I'll mail it to you. (I'm also contemplating planting seeds near the fences of the school I teach at.)