Taking an infant to a game is a novelty, something they can brag about when older by telling people they went to their first game before they were a year old. It isn't particularly safe for a vaiety of reasons either.
As far as older kids, some have better attention spans than others. I'll take my 7 year old daughter to 1 game this fall (it'll be a cheap game anyway thats all i can afford) and we'll see how she does. My wife will come and it will be with the agreement that if my daughter gets inconsolably hot/cold/bored, my wife will take her to get a snack, walk around or just meet me somewhere after the game.
Don't force your kids to endure a game, and have a back-up plan.