I think the guy who mentioned families is right on the money with things.
To some people the ball game is just their job, they go in put in all their effort and want to go home.
I like all my co-workers, think they are great but I don't want to spend time with most of them after work. It has nothing to do with them, if I didn't work here anymore I would call them up to hang out.
The problem is that you only have so much time in the day and sometimes guys have to deal with their personal side first.