Handle away, I see nothing at all wrong with handling the chicks, or even the adults. I consider my chickens pets, but don't really raise them as pets, if that makes since, I handle them some as chicks, but I don't pick up and carry my chickens around, as adults they are all fairly friendly as far as coming close enough to step on my feet and hang out near me. I like them to be friendly enough that if I need to catch them for whatever reason I can.
I really like the chickens, some more then others, I have one big rooster that likes to hang out with me, if I sit outside he likes to sit with me, he'll pick the hay and stuff off my pants while we sit together. Here he is sitting w/ me while we waited for Jade to have her kids, everytime I would move he'd follow me and sit back down next to me.
Anyway that is my opinion, if you like holding them, go for it, there are always going to be people w/ opinions, not really right or wrong. I would imagine there are people that would be completely discussed by me (and others here I'm sure) showing affection to our goats!