Yep I agree with Susanne. It is NOT at all possible. Worms and parasites live in the ground. I would do a fecal float on you goats every now and then and see how they are doing, I would not deworm them just because you think you should.
I have never had a worm problem either, but some day I just might, so I do fecals two to three times a year.
There are a few things that wild life like deer share with goats- also there are a few things like one kind of tapeworm that dogs and goats can share, etc. Deer can also share cocci that goats get according to my vet. But the worse thing is that some worms can remain encycted in animals for years.
I do think you have a lucky chance if your evaluation of your place is correct- I would do a fecal as mentioned above every once in awhile to check, but I would think that you might be luck enough to not have to worm much if at all. But no way would I just assume that.