Thursday, September 27, 2007

Why the Kosmos open-source file system matters

Congratulations to Sriram, Anand, and our other friends over at Kosmix on the release of the Kosmos distributed file system.

This is important stuff. Why? Google. Google has built up a platform advantage for doing web-scale computing that puts them in a different league than everyone else. Their hundreds of thousands of servers are old news, but what's regularly ignored is the superb software engineering layer that makes it possible to run large analytic processes -- like search crawling, search relevance, and ad targeting -- seamlessly across those many boxes. GFS, Google's distributed file system, is part of what makes that possible, and this Kosmix open source release is the next big step, after the Hadoop project, for this critical technology to be available to the Rest of Us.

In the same way that widely available technology like Linux and MySQL made what we now call Web 2.0 possible, layers like KFS will enable major companies (hello, Facebook and the social graph) to scale and compete better today, and allow the next generation of startups to flourish.

I'll be writing more about this area -- a lot more -- as Zvents is at play in the same sandbox. In the meantime, Rich Skrenta is on the story, too.

No comments: