Fine, set a 50K minimum capacity. Most schools that don't meet this will have an NFL caliber stadium nearby. Those that don't (Boise) would till probably prefer playing in Seattle (8-9 hours away) than playing in Phoenix (18 hours away).
Or, since potential candidates will be known 4-5 weeks before a selection is made. Force schools with a shot at hosting to submit plans and bids with alternative stadiums if necessary (this happens in baseball). All these problems can be easily worked out if it weren't just an excuse to keep the bowl sites in the picture.