Yes, all types of Acer Rubrum are toxic. The problem is that it is impossible to tell when they toxic. Broken branches and wilting leaves are dangerous, sometimes drought is a danger, fall and spring are their most poisonous times.
The naturally fallen leaves are not considered a problem. The best thing would be to fence around the trunk so that the goats can't reach the bark and any broken branches should be picked up immediately.