Lessons From The Great Malware Disaster Of 2011

Submitted by Brian on January 27th, 2011 at 12:38 PM

[Note: iPhone app is currently broken; that is the #1 priority in terms of fixes. Hope to have it up by Monday.]

This has nothing to do with Michigan football but the least I can do to help the greater health of the internet is to offer some measure of advice for people who find themselves hacked in the face.

I'm not an expert. Please read the comments for people disagreeing with me, as they may/are better at this than I am. But I just went through this and if you're in the same boat here's what happened with me and what I took from it.

all-bird

Boatmurdered. BURN. ALL BURN.

"Last known good" may not be as good as you think. We have a backup. That backup overwrites itself on a nightly basis. Correction: that backup overwrote itself on a nightly basis. Going forward we wanted to be able to roll back up to a week.

However, we found out that would not have helped us here. Some of our infected files were last modified in early January. A "last known good" configuration from last weekend would have still featured multiple scripts with backdoors that Eastern European hackers could jump in.

We're still going to change our backup system so that it has more snapshots—an injection attack would be more susceptible to a DB rollback, I think—and we are going to have a billion and two backups of the actual code so that if, God forbid, something like this happens again we can have a reference point to pull forward stuff we customized and don't want to lose.

But…

BURN. ALL BURN. I'm not pulling anything forward except select bits and pieces I can hand-inspect. The rest of it dies in a fire. I thought we were destroyed until my brother asked "how long would it take to recreate it from scratch?" This was the moment in the movie when the camera zooms out and the city becomes transparent. It would take… um… maybe a couple hours. The defining feature of a CMS is that everything is in the database. So if you're confident the database isn't the issue you can pick that out, raze the world, download and install all your crap, and not have to worry about finding every last piece of corrupted code. You're going to break a few things when the new versions of your modules don't work exactly as expected but it's way better than the alternative.

Then change your FTP password over SSH. And then, if you're paranoid (ie: us now), turn FTP off entirely for a while. We had to use plain FTP, which is not very secure, because for some reason enabling encryption turned directory listing into a cripplingly slow process. A reader had related an experience in which a corrupted local computer had been giving away FTP passwords, giving hackers direct access to the server. We're not taking any chances despite my incessant scanning.

Burn, all burn exception: we pulled the "files" folder forward despite it being too massive to check because it's all data and those folders are locked down by server permissions so they can't execute anything. Everything else was pored over.

Why we thought it wasn't the database. Well, one, we found plenty of stuff indicating the server had taken a direct hit in the form of scripts that included helpful comments like "webshell by oRb." We brought those shells up and didn't find any database functionality.

Also, injection attacks usually don't affect the entire site—they're more likely to be hostile code submitted by users (something Drupal is good about) that affect only the pages they're submitted on. The malware was being delivered via the CSS and JS files, which are amongst the few bits of the page you're reading that don't come from the DB. While the server corruption could have in turn hit the DB, we didn't see obvious avenues for that and all of the problems were segregated from said DB.

We're now watching it closely just in case, but the evidence pointed to something other than an SQL injection.

What to search for. This article is fairly comprehensive but I'd also suggest looking for "unescape" or the string "%3C%69%66%72%61%6D%65." If you run that through the unescape function you get "<iframe". What are the chances that's helpful code? Not so good.

Don't waste your time with "StopBadware." This is the site you get funneled to if you click the I'm-so-screwed button on the Google warning page. Their extremely awesome advice is to look for the bad things and remove them. They list scripts, redirects, and iframes as the main ways you transmit the bad things—okay, probably helpful—and then offer this up:

There exist several free and paid website scanning services on the Internet that can help you zero in on specific badware on your site. There are also tools that you can use on your web server and/or on a downloaded copy of the files from your website to search for specific text.

Awesome! Where are they? Which are the best ones?

StopBadware does not list or recommend such services, but the volunteers in our online community will be glad to point you to their favorites.

Fu. The "online community" at "badwarebusters" mostly consists of people screaming about erroneous hits. About four threads pop up per day and they can go days without a response. If you're looking to do something quickly it's useless.

That's annoying. This is the worst advice possible:

Once you have located the code that is causing the badware behavior, removing it is often as simple as deleting the offending code from all files in which it appears. Sometimes, it is easier, if you have a clean backup of your site’s contents, to re-upload all of the site’s files, though be careful about overwriting files that may have changed since your last backup.

They've just glossed over the difference between the offending iframe and the code that generated it. Backdoors are not mentioned. This section needs to be replaced with:

BURN. ALL BURN.

Whoever wrote it should be horsewhipped. The next section is about "preventing future infection" when the previous section has essentially advised a n00b who needs to be informed that scripts and iframes are bad, mmmmkay, that "removing the offending code" "often" solves the problem. False. Burn. All burn. 

If you aren't already, sign up with Google's Webmaster tools. We first found out the aggregated JS file was an issue from them, and they periodically updated their findings to let us know we still hadn't killed the problems. Tip: if you're aggregating js and css you may want to stop for more precise identification of the end destinations.

These are not the sources. You have to find those, or just burn everything to the ground.

Don't get notifications other than security notifications. This site is now running dozens of Drupal modules, some of which actually have release changelists that read, in their entirety, "fixed typo X." After a while you stop checking just to see that some random module has done some stuff you don't care about, and then you don't know when certain modules are out of date. We're still not sure what the attack vector was but one of the main candidates was known, patched holes in Drupal. I went from weekly updates about everything to daily updates about security. Drupal shouldn't have other options.

Status. We're not entirely out of the woods yet but it's looking promising, and we have installed various alarms in the system to blare at us whenever anything unexpected (a file getting updated outside of the areas that's supposed to happen) goes down. Hopefully if there is another breach we will catch it long before anything starts getting delivered.

Comments

jhackney

January 27th, 2011 at 12:47 PM ^

I feel like I am taking those first steps right out of the nuclear bunker to navigate the damage right now. Did we get our missles off at those Reds from tsio?

Eyebrowse

January 27th, 2011 at 2:03 PM ^

It's like I don't even know where I am anymore.  

On a more serious note, this whole malware disaster has really shown me the depths of my own addiction.  I can say I am both proud and saddened by the state of my inner self since then.  

Let's also not talk about how I haven't shaved or showered in days (okay weeks, but that's nothing out of the ordinary).  

jhackney

January 27th, 2011 at 2:11 PM ^

Addicted?! Who is addicted?! I'm not addicted!

 

All I want to know is if that malware has any offers? TomVH? I think MGoMalware was great breakaway speed and punishing hits over the middle. He has to at least have 4 stars.

dennisblundon

January 27th, 2011 at 1:02 PM ^

Good luck Brian, hope you get this back up and running again soon. For those of you eagerly awaiting my ability to post again I have a couple of gems ready on deck. One is a projected depth chart for 2011, complete with win/loss predictions for next years schedule. The other is why aren't we recruiting Cardale Jones with more tenacity. It will be glorious.

ottomatic

January 27th, 2011 at 4:34 PM ^

The goal of most modern malware is to operate quietly. Get in, stay in, and exfiltrate data, or gain control of additional systems. The malware that infected the MGOBLOG site attempted (for some not all) to download  a payload of a known spyware, there's a good chance it would have redirected to additional malware as well. The ultimate aim being that criminal syndicate want to steal your shit and control your computer.

Worldwide there are close to 7 million computers still infected with conficker. Those infected systems can be thought of as a zombie army ready to take commands and participate in distributed denial of service attacks (DDOS). The criminal syndicates that control these assets will even rent them out. In some of the shadier corners of the internet you can bid-on or buy a zombie contingent to launch your attack.

Sort of like paying Scam Newton to win the BCS.

a non emu

January 27th, 2011 at 1:06 PM ^

I am still not taking any chances on my work computer. I am running firefox with AdBlock, and NoScript set to everything, including iFrames, disabled. One promising thing is that the site no longer shows up as an attack site on Firefox, so looks like you are more or less out of the woods. But, good luck! My MGoWithdrawal symptoms have been terrible recently.

Need. Moar. Mgo. Less. Malware.

Hugh Jass

January 27th, 2011 at 1:06 PM ^

to this kind of technology psycho babble.  You were talking a foreign language to me.  I have faith that a Blog about Michigan football will rise from the ashes and be bigger and better than ever!  Also thanks for saving all my mgopoints that I have wasted several hours of potentially productive time to earn -  You Rock!

Don

January 27th, 2011 at 1:07 PM ^

FWIW.

We need to get a complete record of phone calls and emails going back and forth from Eastern Europe/Russia to:

• West Lafayette

• Columbus

• East Lansing

• Rivals

CRex

January 27th, 2011 at 1:14 PM ^

For anyone infected and running a version of Windows I suggest: Windows Defender.

Yes it is a Microsoft product, but it is free and good at what it does. I started using it back when Vundo hit the scene and was blowing by things like McAfee and Norton. I'd have machines come in with McAfee flashing "OMG VUNDO DETECTED! DOOM! DOOM! DOOM!" but unable to clean it. However installed Windows Defender resulted in Vundo being blown away in under an hour. Basically it looks like MS is tired of being a bad joke in the security arena and is out for blood. It updates daily and does a major Rev at least 2 or 3 times a year.

Should you be a Michigan student and using McAfee 8.0/8.1/8.5/9.0 or whatever version is currently up on itd.umich.edu/bluedisc get rid of it and get something better. Sometime around 8.0 McAfee lost its mojo in terms of virus detect and their spyware detection is a freaking joke. Plus they do things like decide system32.exe is a virus and delete it. System32.exe is a key Windows system file and deleting it basically bricks your computer. Get rid of it and avoid Virus Busters (the UMich group) like the plague. They have no clue what they're doing (assuming they do anything, given their page hasn't been updated since 2004).

As a side note, don't run multiple virus scanners at one. They end up trying to scan the same chunk of memory from time to time and bad things happen. So if you get a new one, be sure to remove the old one.

Bronco648

January 27th, 2011 at 1:35 PM ^

If MS Security Center is part of Windows Defender (or vice-versa), it didn't do it's job on my Dad's computer.  It was infected and he had MS Security Center installed.  It didn't even bat an eye at the malware propogated thru this site.  My brother-in-law does workstation support for a Fortune 100 company and recommends the current version of Norton.  Say what you will about Norton (or any of the other anti-virus software products) but I have it installed and was not infected.

My Dad decided to go with something called Viper that's supposed to work well on Windows 7 (all of the PCs I use are Win XP).  I'm not saying CRex doesn't know what he's talking about, I'm just trying to add more information since most of the comments here indicate this subject is beyond a lot of folks comprehension (no offense to anyone).

Unfortunately, there are people out there that want to ruin other's internet experience.  A lot of you use your computers for work, home finance, personal archives, etc.  Please don't let something like this ruin all of your hard work.  Get some sort of anti-virus protection.  If you decide to go the freebie route, do some reasearch first to see what it can and cannot do.  Don't be cheap just for the sake of being cheap.

CRex

January 27th, 2011 at 1:56 PM ^

What on earth is MS Security Center beyond a generic webpage that provides updates and links to various tools.  It's not some kind of program you can install, just a links of removal tools and links to updates, post infection removal tools and scanners.  Thus I have no idea what parts of that system your dad had installed or didn't have installed.  

Windows Defender (spyware only) and Microsoft Security Essentials (spyware + virus) both do realtime and benchmark in 95+% for detection rating.  Security Essentials actually benchmarked hirer than AVG (the most popular free antivirus) during the 2010 tests by SpywareTools.  

As it stands right now I think SE is pushing about a 95% in latest tests and normally is in the 95-98% range for detection (above average in rootkit detection and repair, below average in zero threat handling*).  I like it better because its idiot proof and updates via normal Windows mechanisms which I feel is better for less savvy folks.     Security Essentials is known for its fast boots and good realtime protection, with actual scanners taking a bit longer (but you can do those at 2 am when you're asleep).  

I'll avoid any kind of flamewar over who likes what, no tool is perfect.  Rather just be an intelligent consumer and Google likes like "Microsoft Security Essentials detection rating" and "Microsoft Security Essentials review".  Don't rely on single accounts or "he said/she said" stuff.  av-tests.org, icsalabs.com/ and virusbtn.com/ are all good places to go for reviews beside just general Googling.  

*For non techies, zero day means the exploit hits without any warning.  Normally you see hackers talking on their forums about exploits and people have time to patch or prepare.  A zero day is a sucker punch out of nowhere.

Go Blue Eyes

January 27th, 2011 at 1:23 PM ^

"Their extremely awesome advice is to look for the bad things and remove them."

I guess that's like calling the fire department and they say, "Put water only on the spots that are on fire."

SanFrancisco_W…

January 27th, 2011 at 1:29 PM ^

I'm not trying to get ahead of you Brian, I am really just curious.  How long do you think it will take before we get points and voting back up and running as well as the ability to view our account and recent posts?  This change in formatting is making me muy nerviouso.

ottomatic

January 27th, 2011 at 3:44 PM ^

My first Mac was the Mac IIFX circa. 1990. 9000.00 USD without RAM or Video card. By 1997 my kids were using it as key-banger toddler toy.

As for infections on a Mac. Last year I clicked on a link on this site (I'm not blaming) to a bit torrent of a UM game. The destination had some  additional links regarding sound problems and the need to download an additional file, yada, yada, yada ... that additional link tried to download a  remote control app that would have executed on the Mac. So yeah, I love the security of Mac but I still run ClamXAV as well as other precautions.

Sgt. Wolverine

January 27th, 2011 at 1:52 PM ^

has made me realize how much I miss the boxes around each individual comment.  These particular boxes are sort of unwieldy and ugly, but I think it would be worth reconsidering the decision to remove the boxes from the comments.  I find the boxed comments easier to follow.