Flutterby™! : Wayback Machine

Last night at the Weblogger Interest Group shindig, Ev said that How the Wayback Machine Works was a good read, a look at building a 100 terabyte database, mostly with basic commodity hardware.

So if a book is a megabyte, which is about what it is, and the Library of Congress has 20 million books, that's 20 terabytes. This is 100 terabytes. At that size, this is the largest database ever built. It's larger than Walmart's, American Express', the IRS. It's the largest database ever built. And it's receiving queries -- because every page request when people are surfing around is a query to this database -- at the rate of 200 queries per second.