I don't think the reason you state is the true underlying reason for your preference.
Officials decision can be beffudling in any sport where you don't understand the game. In football there is a fine line between holding and legal OL play (not to mention defense). On nearly every single play, it seems a penalty could be called. An uninitiated viewer can't see anything beyond a jumble of crashing bodies, and I suspect most initiated viewers just ignore it also - focusing on the skill positions and the ball. I feel the same way you do about hockey, but I recognize that I feel that way because of my own ignorance, having never played and having watched less than other sports. Since i've played basketball and watched a lot of it I personally feel more comfortable with basketball refeering than any other sport.
Of course, officials will make errors in any sport, but if that ruins the game for you I'd recommend watching a few boxing matches and you'll feel much better about the lack of perceived clarity in comparison.