I live in Howard County near Baltimore. Inner Harbor is similar to the area around Woodward/Harmony/Stadiums, complete with several high-profile shootings in the Harbortown complex last summer. They DO have the National Aquarium and the Constellation, which is pretty cool.
The parts I DO like are Federal Hill and Fell's Point...both vibrant areas, but still a little bit of grit. We're always there on the weekends. Detroit really doesn't have anything like that (or Canton as well, combination of 1700-1800s architecture with restaurant/bar/shopping scene)
You haven't been to some of the bad parts of Baltimore. There are some truly terrifying places around Hopkins, and around northwest Baltimore. I went to Med School @ Wayne, and did a rotation @ Sinai-Grace in NW Detroit (another pretty damn scary area, my first ER night one of the ambulances got carjacked in the bay), and did fellowship @ Cleveland Clinic, right near East Cleveland. All areas are comparably bad.
I would love for Detroit to get on it's feet again. Hopefully, the impending Chapter 9 will allow the Emergency Manager to trim a lot of the legacy costs and modernize the public service contracts so the city doesn't have a public infrastructure to support 2 mil, while the population is 700K. Decrease taxes, better public services, continued improvements in safety, and most of all, better schools will help keep people in the area.