I have a few bushes in my pasture and the goats nibble on them from time to time. They really don't want to really munch on them until all the grass is dried up and the leaves are turning in the fall. Even then not so much. I have never had a problem.
If they are macking them down then I might want to take them out, but if they are just taking a few bites here and there, I wouldn't worry about it. Bushes like these are sort of a last resort food. As long as there is plenty of other things to eat, they should leave them pretty much alone.
IF you are worried about them, then take them out!