Even if the buck is with the does, it's no big deal because it's how you want to manage your goats. It's really not a big deal and you're not a bad owner for it!

Just because some people wouldn't put does and bucks together doesn't mean that it's wrong to do so. You may just have some trouble tracking down their due dates.
There are a few things you do need to take into consideration when keeping bucks and does together.
1- If your does are under a year old they could potentially be breed prematurely and it is fatally dangerous for the doe and kids. If the pregnancy goes well the doe may bout be a good mother due to the young age and less time to mature.
2- Does need to be separated when kidding until the kids are weened or strong enough to join the herd. Your buck could be mean to the kids this is something that needs to be considered.
3- It's a lot of stress on your buck to constantly be with does. They are constantly horny and it is rough on their heart. They also are crazy and harder to deal with because the are constantly wanting and thinking about that cute little doe across the way.
4- The does get tired and annoyed with constantly being "courted".
5-The doe may be bred too soon after pregnancy. This is could cause their health to deteriorate because of constantly popping out kids. It's almost like a puppy mill. It can be done but it's not humane or healthy for the goat to constantly pop out kids.
Those are just a few things to think about. I understand their are some circumstances where one may not have a place for a buck or a friend for him and the only way to have him is too keep them together. I'm not here to judge, only inform. My buck runs with my does. I know the risk, that's why when I get the extra money my buck is going to be surgically castrated and when I want my does bred I will get a stud.