[Note: iPhone app is currently broken; that is the #1 priority in terms of fixes. Hope to have it up by Monday.]
This has nothing to do with Michigan football but the least I can do to help the greater health of the internet is to offer some measure of advice for people who find themselves hacked in the face.
I'm not an expert. Please read the comments for people disagreeing with me, as they may/are better at this than I am. But I just went through this and if you're in the same boat here's what happened with me and what I took from it.
Boatmurdered. BURN. ALL BURN.
"Last known good" may not be as good as you think. We have a backup. That backup overwrites itself on a nightly basis. Correction: that backup overwrote itself on a nightly basis. Going forward we wanted to be able to roll back up to a week.
However, we found out that would not have helped us here. Some of our infected files were last modified in early January. A "last known good" configuration from last weekend would have still featured multiple scripts with backdoors that Eastern European hackers could jump in.
We're still going to change our backup system so that it has more snapshots—an injection attack would be more susceptible to a DB rollback, I think—and we are going to have a billion and two backups of the actual code so that if, God forbid, something like this happens again we can have a reference point to pull forward stuff we customized and don't want to lose.
BURN. ALL BURN. I'm not pulling anything forward except select bits and pieces I can hand-inspect. The rest of it dies in a fire. I thought we were destroyed until my brother asked "how long would it take to recreate it from scratch?" This was the moment in the movie when the camera zooms out and the city becomes transparent. It would take… um… maybe a couple hours. The defining feature of a CMS is that everything is in the database. So if you're confident the database isn't the issue you can pick that out, raze the world, download and install all your crap, and not have to worry about finding every last piece of corrupted code. You're going to break a few things when the new versions of your modules don't work exactly as expected but it's way better than the alternative.
Then change your FTP password over SSH. And then, if you're paranoid (ie: us now), turn FTP off entirely for a while. We had to use plain FTP, which is not very secure, because for some reason enabling encryption turned directory listing into a cripplingly slow process. A reader had related an experience in which a corrupted local computer had been giving away FTP passwords, giving hackers direct access to the server. We're not taking any chances despite my incessant scanning.
Burn, all burn exception: we pulled the "files" folder forward despite it being too massive to check because it's all data and those folders are locked down by server permissions so they can't execute anything. Everything else was pored over.
Why we thought it wasn't the database. Well, one, we found plenty of stuff indicating the server had taken a direct hit in the form of scripts that included helpful comments like "webshell by oRb." We brought those shells up and didn't find any database functionality.
Also, injection attacks usually don't affect the entire site—they're more likely to be hostile code submitted by users (something Drupal is good about) that affect only the pages they're submitted on. The malware was being delivered via the CSS and JS files, which are amongst the few bits of the page you're reading that don't come from the DB. While the server corruption could have in turn hit the DB, we didn't see obvious avenues for that and all of the problems were segregated from said DB.
We're now watching it closely just in case, but the evidence pointed to something other than an SQL injection.
What to search for. This article is fairly comprehensive but I'd also suggest looking for "unescape" or the string "%3C%69%66%72%61%6D%65." If you run that through the unescape function you get "<iframe". What are the chances that's helpful code? Not so good.
Don't waste your time with "StopBadware." This is the site you get funneled to if you click the I'm-so-screwed button on the Google warning page. Their extremely awesome advice is to look for the bad things and remove them. They list scripts, redirects, and iframes as the main ways you transmit the bad things—okay, probably helpful—and then offer this up:
There exist several free and paid website scanning services on the Internet that can help you zero in on specific badware on your site. There are also tools that you can use on your web server and/or on a downloaded copy of the files from your website to search for specific text.
Awesome! Where are they? Which are the best ones?
StopBadware does not list or recommend such services, but the volunteers in our online community will be glad to point you to their favorites.
Fu. The "online community" at "badwarebusters" mostly consists of people screaming about erroneous hits. About four threads pop up per day and they can go days without a response. If you're looking to do something quickly it's useless.
That's annoying. This is the worst advice possible:
Once you have located the code that is causing the badware behavior, removing it is often as simple as deleting the offending code from all files in which it appears. Sometimes, it is easier, if you have a clean backup of your site’s contents, to re-upload all of the site’s files, though be careful about overwriting files that may have changed since your last backup.
They've just glossed over the difference between the offending iframe and the code that generated it. Backdoors are not mentioned. This section needs to be replaced with:
BURN. ALL BURN.
Whoever wrote it should be horsewhipped. The next section is about "preventing future infection" when the previous section has essentially advised a n00b who needs to be informed that scripts and iframes are bad, mmmmkay, that "removing the offending code" "often" solves the problem. False. Burn. All burn.
If you aren't already, sign up with Google's Webmaster tools. We first found out the aggregated JS file was an issue from them, and they periodically updated their findings to let us know we still hadn't killed the problems. Tip: if you're aggregating js and css you may want to stop for more precise identification of the end destinations.
These are not the sources. You have to find those, or just burn everything to the ground.
Don't get notifications other than security notifications. This site is now running dozens of Drupal modules, some of which actually have release changelists that read, in their entirety, "fixed typo X." After a while you stop checking just to see that some random module has done some stuff you don't care about, and then you don't know when certain modules are out of date. We're still not sure what the attack vector was but one of the main candidates was known, patched holes in Drupal. I went from weekly updates about everything to daily updates about security. Drupal shouldn't have other options.
Status. We're not entirely out of the woods yet but it's looking promising, and we have installed various alarms in the system to blare at us whenever anything unexpected (a file getting updated outside of the areas that's supposed to happen) goes down. Hopefully if there is another breach we will catch it long before anything starts getting delivered.
More yes, please. Given the current state of college football scheduling, where you have to have one real nonconference game and then you can schedule anything that will show up at your stadium down to the Albanian cricket circus, I've been in favor of expanding the conference schedule for years. So Adam Rittenberg's post on the possibility comes with some welcome quotes:
There are certainly pros and cons to increasing the number of league games, and Big Ten athletic directors expect to debate them in August during their next scheduled meeting in Chicago.
"Unless you’re really hot, fans are finding that some of the preseason games, they just don’t appreciate," Purdue athletic director Morgan Burke said. "They’d rather see you play every Big Ten opponent. If you went to nine games, you’d be bringing in one more Big Ten opponent, which would make your season-ticket package more attractive."
By radically increasing the amount of money people are expected to play with PSLs and mandatory donations and whatnot, schools have increased the pressure to have home schedules actually worth buying. Burke's actually in favor of ten(!) conference games, which will never happen.
The article also quotes Barry Alvarez in support and we know that Michigan has been pushing for more conference games for a few years now, so there's at least some chance the league will add another game. Another bonus of the extra conference game: if the Big Ten does go away from pure geography and creates a division that's Michigan-OSU-Alamo Party*, additional conference games will reduce the impact of any disparity. It also makes cross-division protected games (which I don't like) less necessary since you'll be playing two-thirds of the opposite division instead of half.
*(Which seems to have something of a consensus building around it. TOC threw in the towel, and once the blogs are united nothing can stand against them. If Penn State had a vote that might be a problem, but lol Penn State suffrage.)
If NASCAR counts as a sport… then solar car competitions, where you actually build the thing yourself, is like a double sport. Also Michigan's solar car team is consistently awesome. They're running the American Solar Challenge right now and, though it's fuzzy if they're actually winning, they think they're doing well:
After being tight with Minnesota this morning and afternoon, they had to pull off the road for what is rumored to be battery problems. We don't know the current location of any other teams, but we believe we are at least 15 minutes ahead of everyone but Stanford.
That was yesterday. They learned last night that Minnesota is now 40 minutes back and Missouri S&T, which is apparently big in solar cars, is 10 minutes back. The previous stage saw Infinium finish almost an hour in front of their nearest challenger. We should totally try to get this thing in the Director's Cup.
Goodbye, almost everyone. One of the tangential discussions that's entered the public consciousness after the QC/stretching violations at Michigan is "dang, there are a lot of dudes getting paid to not coach football." The NCAA is within its rights to reel these guys in somewhat, but this seems drastic:
Back in April when the Athletics Personnel and Recruiting Cabinet began seriously discussing legislation to curb the growing football and basketball staffs, there were two big questions: exactly how many noncoaching staff members would the teams be allowed and how would the legislation deal with attempts to build new offices in the athletic department?
The cabinet gave an emphatic answer to the former question, with a somewhat weaker answer to the latter. Bowl Subdivision Football would be limited to just four noncoaching staff members, while men’s and women’s basketball would be reduced to just one. In the Football Championship Subdivision, the limit would be two.
That's not four grad assistants, it's four staff members, period. The Bylaw Blog suggests this would see athletic departments devolve the many other roles undertaken by specific sport-specific staff into department-wide organizations that avoid this new regulation. The money is always going to flow somewhere. At some point the NCAA should get serious about booting I-A teams that can't manage 20,000 paid attendance per game into I-AA. The real problem here is that teams like Michigan and Eastern Michigan are being addressed by the same sets of laws when they have zero resemblance to each other.
The elusive and wonderful. Six Zero's regular series profiling some of the characters who hang out around here has an exclusive look at youtube hero Wolverine Historian. Most surprising to me was WH's age:
Wangler to Carter. Hello Heisman. Bo singing the Victors. In your expert opinion, what is the single most iconic video clip of Michigan football?
There have been many, many memorable moments over the years. But I think Wangler to Carter from Homecoming 1979 is probably the most iconic video clip of Michigan football. I was born 4 months after that game was played so I obviously have no personal memories of it. But the video speaks for itself. One last play, Carter dancing into the end zone, the crowd going insane, Bo jumping up and down, Bob Ufer screaming, “Oh my GOD!!! Carter scored!!!” and Lee Corso having a stroke on the Indiana sideline.
Given the vast breadth of WH's tape collection, I would have ballparked his date of birth sometime around 1817. Instead he is younger than me.
Merrill watch. Not in the scary way. The first round of the NHL draft is tonight and should see defenseman Jon Merrill taken. There will also be a goalie taken, and this will be lame. But back to Merrill:
"I honestly want to get drafted, but it's not that big of a deal," Merrill said in a phone interview Tuesday. "It's tough not to hear about (mock drafts) or see things, but I really don't care that much about it.
"First pick or the last pick, you have the same opportunity to play in the NHL."
For the paranoid, there's no hint in of a Merrill defection anywhere in the article. The remainder of the draft will be more interesting as far as the composition of the team goes: CCHL forward Alex Guptill is eligible and has made some comments about deciding what he wants to do after he talks with the team who drafts him. He could spend a year in the USHL, possibly with fellow 2011 commit Lucas Lessio, or defect if the Kings or some other team run by paleolithic folk grabs him. He should go somewhere in the middle rounds.
The final word on SEC vs Big Ten. Sure, they may have won a zillion national titles but this is the Big Ten's position on vuvuzelas:
The Big Ten has specific policies that do not allow irritants or noisemakers, so vuvuzelas would not be allowed. Below is the specific language from our football game management manual.
This is the SEC's:
This instrument, no matter how irritating to some, will not be banned from SEC games this upcoming season, according to the SEC. The instrument of choice in South Africa, which may or may not catch on here in the states, can be brought into stadiums across the league.
Big Ten wins forever. Not that I imagine there will be a ton of vuvuzelas at SEC games. There will be three incidents where vuvuzelas are brought into the stadium, then gingerly extracted from parts of the anatomy plastic horns were not meant to tread, before everyone gets the idea.
Not technically World Cup content. This is about soccer but the larger point is excellent:
One of the hard things about forming an outlook on the World Cup is that when an event gets this much attention, the flow of commentary is so fast and broad that every possible angle is exhausted and trivial positions develop a kind of insubstantial politics. Conventional wisdom starts to seem like an ideology, and if you’re not careful, your own feelings about what happens will be dictated by where you want to stand in relation to that ideology rather than by what you actually think. There’s a pundit position, a cognoscenti backlash, an uber-cognoscenti counter-backlash, and so on till after midnight. Your heart and the stadium get farther and farther apart.
Case in point: two opinions that put you on roughly the same line of anti-pundit knowingness would be “the first round of games was actually great” and “Switzerland weren’t that exciting yesterday; Spain were just terrible.” Maybe you really feel those things, or have numbers to back them up. But in most cases, I’d guess that the attraction of these stances has a lot to do with the fact that they put some space between you and the thousand-mile pandemonium of cliches blasting out of the TV studios and the pages of your favorite newspaper. It’s not only that they make you sound like you know what you’re talking about, although there’s no discounting the lure of savvy disaffectedness. They also just turn down the volume.
That sort of contrarianism for the sake of saying something new is a constant temptation for anyone tasked with writing something people will find interesting. Sometimes it's right. Sometimes it's David Berri running a regression and declaring Dennis Rodman more valuable than Michael Jordan or that NBA coaches don't understand who their best players are. If you're trying to combat the conventional wisdom, you should regard it a tricky, wily foe that requires something more than a blunt-force blow.
Etc.: Citi dumps its Rose Bowl sponsorship.