1. The environment: the attention span of Bay Area people starts with two baseball teams, two tradition-heavy pro football teams, hiking, camping, biking, driving Volvos and Priuses, the list goes on...the lifestyle does not cater to heavy interest in a college football team, especially in the quiet suburb that is Palo Alto. The fact Cal is a hit-or-miss program doesn't help the college game get traction.
2. Stanford has pretty well sucked about half the time over the last two decades, usually in a typical up-down-up-down seasonal cycle. Which kills the chance for the program to become guaranteed entertainment and destination viewing.
3. Stanford's alumni base is small, and distributed around the country and the world. They can't depend on a large local base to become long-term season ticket holders for the alma mater (like Michigan and Washington have among many other teams).
4. Why does it matter? Well I don't think Harbaugh takes it personally, but as he's noted it's a factor in the BCS slots so it matters to his program. Other than that I think attendance is just a proxy for how big the program can go and wants to go.