[ Content | View menu ]

A Google of One

Mark Mzyk | February 20, 2008

Yahoo has announced they are now using Hadoop, an open source implementation of MapReduce, for processing search data, and they’ve also announced impressive performance gains because of it.

While investigating this story I also came across mention of IBM using Hadoop for it’s cloud computing initiative.

I think these related stories point towards the future of the web.  As open source continues to grow, software building blocks are becoming easier to obtain and use.  Cloud computing is now promising the same ease of use building blocks for hardware.

You currently need to be a company the size of Yahoo, Google, IBM, or Amazon to have the resources to run operations like they do.

But with the likes of open source and cloud computing, I think the playing field is leveling and individuals will gain the power to compete against the largest web companies that currently exit.

What is to stop me from writing a piece of software that deploys Hadoop across Amazon’s computing cloud and then offering a service off of that?  So long as I can afford the initial entry servers, I can use Amazon’s expertise to run my back end while keeping my maintenance cost down by using open source solutions.  I now have a service, that if implemented smartly, can scale and run quickly, at a fraction of the cost Google paid to develop their service while potentially competing against Google in some area.

Examine that scenario.  Does it sound plausible to you?  It certainly does to me.  The resources that used to be precious are now freely available.  So then what becomes the bottleneck?  What becomes the skill that is valued?

Two things:

  • Implementation.  The ability to leverage all the pieces together effectively.
  • Business Acumen.  By this, I mean having a smart business plan.  If you are going to pull this off, you are going to need to cover the cost of using the cloud computing and the initial servers.  Yet if you can come up with a way to make enough money off of each individual using the service, it could potentially be self sustaining.  Google and Yahoo could even help you compete against them by supplying you with revenue from ads.

The pieces are all there.  Now all we need is the first enterprising individual to prove it can be done.