Home | Gallery | Songs | Dates | Merchandise | News | Bio | Lists | Discussion | Thanks

Autopsy Report

On Friday, November 2, 2001, at about 3:30 am, the recently-launched Second Set website was completely deleted by unknown intruders. What follows is an account of events, based on what I found after the fact. I write this a) as punishment for myself, like writing "I will not write crappy software" 1000 times on the board; and b) so that anyone reading might learn from my mistakes. Note that you might have to be a computer dork to get some of this.

First, some background. The Second Set website runs on a server in my house. The network it's on is connected to the outside world via a fairly thin 128kbps DSL line because that's the best I can get out here in the 'sticks. It serves http(s), ftp, and ssh. I'm usually pretty paranoid, so the other machines on the inside only serve ssh. This was why the intruder was unable to get to any other inside machines.

Now all the code for the website was my handiwork. (For details about the setup, peep the old technical details document.) The codebase grew haphazardly as I tested and grafted new ideas onto what was there. Because Cocoon is such a good architecture, the spaghetti factor was lower than expected; still, the code was basically a prototype in production. This was mistake number one.

The expoited bug was in the SQL escaping code. When someone typed a discussion message, this code was supposed to enclose the text in single-ticks (') and double up all the single-ticks in the message, which is the standard sql escape. So the message What's up? became 'What''s up?' The bug was that because of the way I scanned the string, an even number of single-ticks got turned into an odd number of single-ticks like this: I'm a cracker''; system(evil code); turned into 'I''m a cracker'''; system(evil code);' and the evil code would execute as the database user.

(I won't call that mistake number two. It's a bug in a program I wrote. Bugs are mistakes in a sense, but they're basically unavoidable, so I don't hold them against the programmer. Not preparing for bugs properly is an avoidable mistake…)

Mistake number two was not keeping proper backups. The nightly backups were on disk, right next to the originals. Only the actual database was backed up to tape. Mistake number three was lack of consistent paranoia: for my convenience, both the originals and the backup were writable by the database user.

So if it isn't obvious yet, the intruder's evil code was your basic rm -rf * after a few discovery commands. They tried to set up easier access to the machine, but their attempts were blocked by the firewall. They tried a few trivial expoits of the other inside machines but those machines only serve ssh. Finally, perhaps frustrated by the inconvenient form of access they had, they ran find / -exec rm -rf {} \; and deleted everything they could, then ran away.

This isn't the first time someone's broken into a machine or a network in my charge. I used to run a university network, so I know all about that. Hence my healthy paranoia, which limited the damage and allowed me to piece together what happened from all the silly logs.

To the intruder, I want to say this. You did two things: a) you found a flaw in a black-box piece of software, and b) you exploited it destructively. It took some skill to do (a), and I would have appreciated it immensely if you would have dropped me a note about it. But doing (b) took no skill at all. It cost you a possible friend and ally. Next time, try helping instead of hurting -- you'll find that it's even more exciting.