There is the possibility of the goats getting salmonella from them. They usually don't share diseases though.
But how much space will all these animals have? Several pot belly pigs and some goats says to me that they will need at least a couple acres for their pasture and quite a good size shelter and house them separately. You really have to be very careful about not overcrowding. Overcrowding leads to serious parasite problems and sickness faster than anything. Also, if you are new to livestock, you do want to be careful not to jump in too deep. It is devastating to not be able to keep things under control and have losses. Especially if you don't have deep pockets and a good livestock vet.
One of my wethers is living with another goat and a few potbellies. The potbellies snarled when he got near to the food, so they will need separate feeding stations. There is also the question of the different dietary needs. You'd have to make sure what the pigs were eating would be safe for the goats, and vise versa.
I personally would not recommend keeping them together. When I first started out in goats, I had two pot belly pigs and kept them in with the goats for awhile. They need to be fed a little differently, but the biggest problem was them getting into their water. During summer months, I could never keep the trough clean. It was gross. They would get hot and into the trough they'd go...sit there and pee and poop in it and lounge in there all day. Adding a little pool or extra mud hole didn't work either...the goats would either drink out of it sometimes or the pigs would still get in the trough when they felt like it. They were a little food aggressive as well and would always hog (no pun intended) any grain or "special" food for the goats.
If you want pigs, they are fun, but are a lot of work and time. Mine were escape artists..and I have very strong fencing. But if they got bored or hungry, they'd start to damage the fencing trying to get out. Even hog panels. They also eat and poop a lot so keep in mind they'll cost you quite a bit of money and time. If you have a secure fence, plenty of time, and are prepared, it can be a fun and rewarding experience raising/breeding or having them as pets. But no, I wouldn't keep them in with the goats.