I find that my wethers are just as large or larger then intact bucks. They do take longer to get there but, they do. I think that the main reason some wethers don't grow to their potential is because people don't feed them to support that growth.
Wethers do not have the stress of rut to take their calories away like bucks. Everything that they take in is used for growing and gaining. Mine quit having spring growth spurts at about 3 years but, continue to grow slowly until about 6 years of age.
The gonads are not the only part of the body that produce testosterone, even does need testosterone to function correctly. While bucks do grow more quickly in the beginning, the wethers always catch up or exceed the bucks size in the end.
I haven't seen the wethers get as big as the bucks. We have had Nubians for years and the bucks grew to 300 - 400lbs. Where as the wethers only grew to 250 lbs. You have to be careful with feeding wethers and not get them too fat. If they get too fat they have joint problems.
You can castrate at any time in their life with no more danger then when they are young. As for size, with the pack goats vs breeding bucks, the difference is in mass. Breeding bucks put on more muscle and substance of bone. Wethers on the other hand grow taller. During the rut bucks will also amass extra girth in the neck much the same way deer and elk do. Its in preparation for the fights to come.