a terrible blight on our fine country
warning: internet/sports journalism/meta post. it's six on friday so no bitching.
via press coverage
Way back in the mists of time when I'd just been fired from my engineering job for not doing much actual engineering I was wondering whether or not I actually wanted another one when Jamie Mottram emailed me. He asked if I'd be interested in being a "lead" for the college football section of this Fanhouse thing he'd convinced AOL to start*. I said yes and my career as a pants-optional blogger started.
A couple years later, Mottram was at Yahoo and I was on the phone with a guy who seemed to put "-ize" at the end of every verb trying to convince him that Adam Jacobi was a key asset even if he kept posting conversations with Joe Paterno in which he decried DIRTY IRISHMEN. This was the middle of the end, and a couple months later I was out, too.
By that point I didn't much care. I'd stopped posting much because headlines like "God Not A Big Fan Of Sam Maresh, Says Sam Maresh" were getting converted into things like "Sam Maresh Has Further Health Problems." The thing I owned was making sufficient money that I didn't have to put up with aggravation for ten bucks a post.
When I latched on with Sporting News a couple months later it was mostly so I could tell people I wrote for Company You've Heard Of X when that was convenient or lent credibility, and when that got shipped over to SB Nation I cut my workload there down to a couple things I do weekly. The business story of the blog is gradually in-sourcing all of the writing I do, even if it's about the World Cup.
"We're Not Bleacher Report"
Elsewhere, not so much. When AOL decided to blow Fanhouse up and give the Sporting News the brand for five million a year, I wasn't surprised. Ben Koo made a case that it was a stupid move, but we are talking about a company that's had a half-dozen people run Fanhouse in under five years, let Mottram walk out the door, immediately undermined his replacement with HAWT TITS, reversed course on that after 90 seconds, and then did another 180 to hire Jay Mariotti. It's not a surprise AOL has changed course wildly, hoping that doing the exact opposite of their last stupid idea will be the opposite of stupid.
What is something of a surprise is the naiveté shown by some of the outgoing. Dave Kindred interviewed a few of them for IU's National Sports Journalism Center and it's like they've never been part of an aging relic with a declining legacy business before:
"In December," Lisa Olson said, "we were told how great we were doing." Once a columnist at the New York Daily News, Olson remembered The National strutting on stage in 1990, a national sports newspaper hiring good people from everywhere. She thought of FanHouse that way, a gathering of veterans on a journalistic adventure. "We were all experienced and qualified, not some 25-year-old bloggers," she said. "The motto was, ‘Go, go, go. Grow, grow, grow.' And we did. Then, this. It's devastating."
This one in particular even referenced "The National," which lasted all of 18 months. Another complains "we had no idea this was coming," etc. More than one takes shots at bloggers. There's the one above, and then there's the EIC who ended up axing me** stating that when they arrived Fanhouse was nothing more than "a quirky blog."
The theme running through the piece all the way up to Kindred, who titles it "Waiting for the day readers march in and demand an end to the dreck," is journalists bemoaning the fact that their quality isn't recognized as they die by the thousands and Bleacher Report is getting eight-digit funding rounds. Kindred uses the recent press conference in which Jim Boeheim slammed the reporter who asked a question about point-shaving because the internet's been talking about it as a leaping-off point. You'd think they'd know by now.
You Are Bleacher Report
So… the column and those quoted in it are rife with misconceptions that speak to why AOL abandoned ship and why newspapers will slowly bleed readership until internet natives are at the helm in 20 years, at which point they'll just be another voice in the clamor.
Believing Bleacher Report is in the content business. Bleacher Report is not a content company any more than Demand or Associated Media. It is an SEO/marketing company that runs garbage through filters until it comes out with google/newsletter gold. The way they do this is clever, but their success—likely overstated anyway—has nothing to do with the success or failure of people who write for a living.
Believing Fanhouse content was functionally different than Bleacher Report's content. I only subscribed to the college football bit in my RSS reader, but it was a progression of boring AP-style articles, Clay Travis columns, the leftover guys who got in the door under Mottram who were cheap and non-controversial, and Brett McMurphy breaking stories about USF. Meanwhile the larger site had Marriotti.
You know what Mariotti and Travis are? They're trolls. They write controversial things they don't believe for attention. How much of the vaunted 50% non-AOL traffic—the same figure we were told, BTW—was either SEO or people stopping by to tell the various trolls why their stupid arguments were stupid? Mariotti is just a Bleacher Report writer with an editor, and he's the star attraction. This is not hyperbole.
A personal example from my time there: slideshows were pushed ever harder until people started editing posts to stick in random slideshows, hopefully vaguely sexy slideshows, whenever your post could be tangentially connected to one. Slideshows, man.
Fanhouse journalists complaining about how their quality is not appreciated aren't quite right. Anyone who reads above a third grade level can tell there's a vast gulf between it and BR, but when that gulf spans the gap between "offensive to the English language" and "newspaper stuff mostly about things I don't care about" it doesn't matter. Instead of widely loathed you're ignored unless you're breaking news, which is ephemeral.
It's no secret that I hate Deadspin. At least, I hate its bottom 20% and don't care about its middle 70%. But even though I don't read it much I still remember a dozen things—great things—it's published in the past year. If there's anyone who understands making it in internet media it's Nick Denton, and he's decided on lots of dongs and lots of outstanding, smart, highbrow content that people will post on their Facebook wall. Minus the dongs, I try to do the same thing for my niche. That's quality that separates you from BR, not spelling "lose" correctly.
Believing a site that gathers metrics similar to Bleacher Report is long for this world. You can't out-troll Anonymous.
I'd love to know what Fanhouse's direct hit numbers were. Nobody went to Fanhouse from a bookmark. Fifty percent of this site's hits have no referrer; Fanhouse was probably under 10%. Again, that's Bleacher Report except BR has a legion of halfwits voting and commenting on each other's posts to get more RadPoints*** . And if you're like Bleacher Report except you're paying people—giving people benefits—you lose. How many BR halfwits can you vaguely curate for one Jay Mariotti salary? Thousands, and their content is no different except for the platform. Once that platform enjoys content-sharing deals with, oh, say, the Washington Post, the guy with the benefits is screwed.
Bleacher Report's secret is that it's awesome at being terrible. It hammers that dong demographic. Here I try to be really specifically awesome for a niche. Deadspin has it both ways. Fanhouse was just okay at the dong demo, okay at the boring stuff, and there wasn't one thing in the history of that site anyone would remember two days after they read it. That's the same mistake they always make.
When Mottram left for Yahoo he corrected the mistake he made with Fanhouse by creating a suite of independent single-source blogs that are run by a guy. You can tell because each of them comes with a picture.
Not all posts are by these guys, but they own the blog in a way no one owned Fanhouse. Each is "quirky" to some extent. The soccer one has regular posts in which an obscure Polish goalkeeper rants about corn and his neighbor and the week's events. Doctor Saturday annually embarks on a defense of the recruiting-industrial complex. Each one is a central part of its sports blogosphere, written extraordinarily well by people who may have worked in newspapers but didn't live them. Most of the contributors are just people who write well. They haven't been blown up, and Mottram ascended the ladder at Yahoo to do the same across the company.
I don't know what to do about the fading ability of people to pay responsible news-reporting types. Fanhouse was run by incompetents and destined to implode anyway. But I might miss it if it wasn't so goddamn boring.
*[I imagine him crashing through the window of a conference room holding dozens of high-level executives on a chandelier, sword in hand, rose in teeth.]
**[Not that he should have kept me and my two posts a week output.]
***[mwa ha ha. Seriously, though, points here are for troll control and have only incidentally grown into an e-peen contest.]
[Note: iPhone app is currently broken; that is the #1 priority in terms of fixes. Hope to have it up by Monday.]
This has nothing to do with Michigan football but the least I can do to help the greater health of the internet is to offer some measure of advice for people who find themselves hacked in the face.
I'm not an expert. Please read the comments for people disagreeing with me, as they may/are better at this than I am. But I just went through this and if you're in the same boat here's what happened with me and what I took from it.
Boatmurdered. BURN. ALL BURN.
"Last known good" may not be as good as you think. We have a backup. That backup overwrites itself on a nightly basis. Correction: that backup overwrote itself on a nightly basis. Going forward we wanted to be able to roll back up to a week.
However, we found out that would not have helped us here. Some of our infected files were last modified in early January. A "last known good" configuration from last weekend would have still featured multiple scripts with backdoors that Eastern European hackers could jump in.
We're still going to change our backup system so that it has more snapshots—an injection attack would be more susceptible to a DB rollback, I think—and we are going to have a billion and two backups of the actual code so that if, God forbid, something like this happens again we can have a reference point to pull forward stuff we customized and don't want to lose.
BURN. ALL BURN. I'm not pulling anything forward except select bits and pieces I can hand-inspect. The rest of it dies in a fire. I thought we were destroyed until my brother asked "how long would it take to recreate it from scratch?" This was the moment in the movie when the camera zooms out and the city becomes transparent. It would take… um… maybe a couple hours. The defining feature of a CMS is that everything is in the database. So if you're confident the database isn't the issue you can pick that out, raze the world, download and install all your crap, and not have to worry about finding every last piece of corrupted code. You're going to break a few things when the new versions of your modules don't work exactly as expected but it's way better than the alternative.
Then change your FTP password over SSH. And then, if you're paranoid (ie: us now), turn FTP off entirely for a while. We had to use plain FTP, which is not very secure, because for some reason enabling encryption turned directory listing into a cripplingly slow process. A reader had related an experience in which a corrupted local computer had been giving away FTP passwords, giving hackers direct access to the server. We're not taking any chances despite my incessant scanning.
Burn, all burn exception: we pulled the "files" folder forward despite it being too massive to check because it's all data and those folders are locked down by server permissions so they can't execute anything. Everything else was pored over.
Why we thought it wasn't the database. Well, one, we found plenty of stuff indicating the server had taken a direct hit in the form of scripts that included helpful comments like "webshell by oRb." We brought those shells up and didn't find any database functionality.
Also, injection attacks usually don't affect the entire site—they're more likely to be hostile code submitted by users (something Drupal is good about) that affect only the pages they're submitted on. The malware was being delivered via the CSS and JS files, which are amongst the few bits of the page you're reading that don't come from the DB. While the server corruption could have in turn hit the DB, we didn't see obvious avenues for that and all of the problems were segregated from said DB.
We're now watching it closely just in case, but the evidence pointed to something other than an SQL injection.
What to search for. This article is fairly comprehensive but I'd also suggest looking for "unescape" or the string "%3C%69%66%72%61%6D%65." If you run that through the unescape function you get "<iframe". What are the chances that's helpful code? Not so good.
Don't waste your time with "StopBadware." This is the site you get funneled to if you click the I'm-so-screwed button on the Google warning page. Their extremely awesome advice is to look for the bad things and remove them. They list scripts, redirects, and iframes as the main ways you transmit the bad things—okay, probably helpful—and then offer this up:
There exist several free and paid website scanning services on the Internet that can help you zero in on specific badware on your site. There are also tools that you can use on your web server and/or on a downloaded copy of the files from your website to search for specific text.
Awesome! Where are they? Which are the best ones?
StopBadware does not list or recommend such services, but the volunteers in our online community will be glad to point you to their favorites.
Fu. The "online community" at "badwarebusters" mostly consists of people screaming about erroneous hits. About four threads pop up per day and they can go days without a response. If you're looking to do something quickly it's useless.
That's annoying. This is the worst advice possible:
Once you have located the code that is causing the badware behavior, removing it is often as simple as deleting the offending code from all files in which it appears. Sometimes, it is easier, if you have a clean backup of your site’s contents, to re-upload all of the site’s files, though be careful about overwriting files that may have changed since your last backup.
They've just glossed over the difference between the offending iframe and the code that generated it. Backdoors are not mentioned. This section needs to be replaced with:
BURN. ALL BURN.
Whoever wrote it should be horsewhipped. The next section is about "preventing future infection" when the previous section has essentially advised a n00b who needs to be informed that scripts and iframes are bad, mmmmkay, that "removing the offending code" "often" solves the problem. False. Burn. All burn.
If you aren't already, sign up with Google's Webmaster tools. We first found out the aggregated JS file was an issue from them, and they periodically updated their findings to let us know we still hadn't killed the problems. Tip: if you're aggregating js and css you may want to stop for more precise identification of the end destinations.
These are not the sources. You have to find those, or just burn everything to the ground.
Don't get notifications other than security notifications. This site is now running dozens of Drupal modules, some of which actually have release changelists that read, in their entirety, "fixed typo X." After a while you stop checking just to see that some random module has done some stuff you don't care about, and then you don't know when certain modules are out of date. We're still not sure what the attack vector was but one of the main candidates was known, patched holes in Drupal. I went from weekly updates about everything to daily updates about security. Drupal shouldn't have other options.
Status. We're not entirely out of the woods yet but it's looking promising, and we have installed various alarms in the system to blare at us whenever anything unexpected (a file getting updated outside of the areas that's supposed to happen) goes down. Hopefully if there is another breach we will catch it long before anything starts getting delivered.