Gmail doesn’t use RAID

This guy arranged for thousands of email messages to be sent to his Gmail account at once, and surprise, surprise! It fell over. What a jackass. One way to know for sure he’s a jackass is the theory about compression and RAID.

First, Gmail doesn’t use RAID. It uses the Google Filesystem. It is like RAID, but implemented in user space, which lets Google play games with filesystem semantics you can’t do when you are restricted to one of the Unix filesystems on top of a hardware RAID box. It also leverages the CPU power you get for free when you buy thousands of PCI busses and IDE interfaces. The mainframe guys figured out long ago the key to throughput is lots of I/O paths. The Google Filesystem lets them use lots of CHEAP I/O paths, instead of the insanely expensive I/O paths that Netapp and Sun sell.

As an aside: It is hilarious to watch the other email providers use their giga-buck Netapp filers to try to compete with Google on email account size. What little profit they might have been making from ads is going to go right out the window when the ops team has to buy 20 new NetApps to meet the demand caused by marketing’s little stunt of trying to keep up with Google.

Second, the compression stuff is bogus. Disk is cheap. Buying, powering, and fixing the millions of CPUs it would take to do all that compression would dwarf the savings from the compression. Not to mention adding latency to the UI, which is something Google prides itself on avoiding.

To Google’s credit, this jackass only managed to make his account unusable. Oh, and the accounts of the few thousand other people who have parts of their mail spool files stored on the same disk servers as him (and even those people were relatively less effected, since it’s highly unlikely they all depended on precisely the same set of disk servers that Kevin’s account is on). At least he didn’t manage to destroy the inbound mail servers. This is typically a case where you’d see cascading and amplifying failure. Typically, what you see in a test like that is an overloaded disk server, which would have caused the inbound mail server to queue, then fail. When that inbound mail server was taken out of service, another one would have fallen, and so on. So instead of only having one set of disk servers overloaded, you have that and zero remaining inbound capacity. Cascading and amplifying. Two words sysadmins don’t like very much.

Of course, as of today, two other words Gmail sysadmins probably don’t like very much are “Kevin” and “Rose”.

Firefox 0.9.1

First the good news: Firefox 0.9.1 for Win32 fixes my crashing problems. (So all you Mozilla zealots that found this page last time I said something about Firefox, please don’t spam me.)

The bad news is that Firefox 0.9 was a crashy mess, and it stomped my icons such that my HTML files that I saved out of Excel no longer had the “Excel generated HTML” icon, and no longer had the “Edit” option on the context menu.

So I upgraded to Firefox 0.9.1 and found that the UI has new icons. Again.

Now, a word to to the Firefox developers: please focus on making a 1.0 release that doesn’t crash, and who’s user interface does not change every time I upgrade a point release. All I want from my browser is a rock solid foundation which I can ignore while I go about my daily life online. It is NOT a UI scripting and experimentation platform, it is NOT a playground for graphical designers, is it NOT for anyone’s vanity. It should be like the footing of a bridge; resolute and steadfast.

Thank you for your astounding work on the Mozilla project. Please keep up the good work.

Fahrenheit 9/11

Karl and I went to see Fahrenheit 9/11 tonight. There are fewer classic Moore scenes in it than his other movies, which is a huge improvement. His message is way too important to be lost in the noise of his wacky hijinks. In some ways, I think the criticism of his previous movies have taught him what Moore-moments are defensible, and which ones he needs to just leave out of movies. (The bank-rifle scene in Bowling for Columbine scene is a case where he was fairly excoriated by the critics, I think. To my knowledge, there are none of those in Fahrenheit 9/11.)

One thing I really liked about it was the pace. He does a very good job of balancing serious stuff that brings a tear to your eye and requires thought to absorb, and zany vintage-Moore scenes that make you laugh.

However, one of the transitions was very uncomfortable; he cut from footage of Bush on the aircraft carrier declaring the end of major combat to footage of a mortar landing near some American troops. The shock of the mortar going off and the troops disappearing into smoke was like the punchline in a joke. But I immediately found myself thinking, “please God, let that just be a bang, not a deadly attack”. Alas, the troops were injured by the attack, and you see them being carried off in stretchers. It made me unhappy I’d perceived the boom as a punchline, and it made me mad at Moore for putting me in that situation.

Also, there’s been a bit of a kerfuffle over the graphic scenes in the movie. There are some, and they are appropriate to the message of the parts of the movie they are in. The MPAA made the right call on the R rating, I think. I understand Moore’s argument about the rating, but there was no compelling reason to bend the rules for this movie, and the rules clearly put the kind of images Moore chose to put into his movie into the “R” rating. Perhaps Moore’s argument about the rating was just a publicity stunt. I wouldn’t put it past him.

All in all, this movie tells Moore’s viewpoint, and I’m all for that. One of the few things from the political world that has recently warmed my heart is that the right wing zealots have given up attacking this film on the grounds that Michael Moore shouldn’t be allowed to say these things. It could simply be for the pragmatic reason they are trying to keep from giving him press. But I prefer to believe it is because even the right wing understands that once we’ve lost the freedom of speech, we’re all in real trouble.

Except the Fox News Channel. Fox News long ago lost the freedom to voice the words “Fair and Balanced” without having us all titter…