It would have been better to start with a bottle baby for a single goat. But if the deed is done, go sleep in the pen a few days. Take a book. You want him to bond to you since he has lost his 'buddy'. And get him another animal to keep him company when you're not there. Take him on long walks to help work out the stress.
I have five goats. The fifth is designated to be my 'buddy' rather than paired with another goat. I take him to work and visit him on breaks and walk him at lunch. By making many small visits, he will get the idea that he doesn't have to panic when I am not in sight. I do it at home as well. He has free roam of the yard, not in the pen with the other goats. I visit him frequently and he can wait by the door or go about his business exploring. When by the door, he can see me through the glass.
After lunch yesterday, I forgot to tie him, and he stood quietly by the door waiting for me for nearly an hour before I discovered my mistake. The real fun of having pack goats isn't the packing. That is a mere side benefit. They each have their own personalities. As you get to know your goat you'll enjoy the time you spend with him even if you aren't packing.
Oh, and welcome to the forum.