This idea has been bouncing around in my head for a while, but it’s only just now that I’ve managed to get it down onto paper in more than just a series of bullet points.

Management and Twitter face the same scaling problems.

This thought is directly related to agile practices.  We’ve all known the traditional management style, the command hierarchy, where everything flows from the top down, with lateral communication only happening at equal levels on the tree.

This style fits with waterfall development nicely.  However, given the rise of agile, the communication structure has changed.

Instead of one way communication, communication now flows along a continuous loop.  Each person, instead of reporting to one node in the system, now reports to multiple nodes.  Each person may still only report to one node explicitly, but implicitly, for the system to work, each person must report to each other.

The total system depends on the free flow of information and the flexibility of each node.

Think of this in terms of Twitter.  If you’ve followed the news about (or used) Twitter, you know there was a time when Twitter seemed to fail almost every other day.  Eran Hammer-Lahav wrote an excellent series of posts on the challenges facing Twitter.  Eran points out that what Twitter is attempting to do isn’t a problem that has been solved before: how do you give updates to one person about multiple people that all operate independently?

Let’s make that a little less abstract.  I want to know what my friends are up to, so I log onto Twitter.  Now Twitter has to find all the most recent updates on my friends.  But how does it do this?  It has to query each of the people I follow, finding all the recent updates for each person, and then determine which of those are the most recent overall and should be displayed.  It ends up being a very expensive query.  Now factor in database sharding on the back end and everything goes to hell in a hand basket pretty quickly.

As Eran notes:

“Current database and caching solutions are simply unable to handle a complex network of multiple relationship [sic] between objects.”

So it goes with management.

Agile is a system with multiple independent nodes that are expected to report information to each other while fluidly adapting to the situation at hand.  It’s an expensive human query, the same type faced by Twitter on a technical level.

The solution to scaling agile beyond relatively small teams hasn’t been found, especially when multiple managers are involved.  This is why so many companies claim to be agile but then fail to implement truly agile processes.  The cost of implementing agile processes in a large organization are expensive and scaling agile goes against what the current management paradigm has been optimized for.  Until the solution to scaling agile is found and disseminated throughout the industry, management is going to continue to give lip service to agile while experiencing its own fail whale.

That’s how it always is with something new.

I’d like to thank John for creating the images used in this post.