Off-Topic: The Agile Manifesto, or How to Become Google

#agile#management#software-engineering#off-topic
October 7, 2008 · 💬 Join the Discussion
If you're lazy, click here for the TL;DR

I’m nowhere near one of the best students of Agile methodologies out there. So I’ll allow myself the luxury of a perhaps “naïve” view of what I personally see in this subject. And I know, I know, the word “Google” in the title is there to grab attention. I’ll explain at the end ;-)

First of all, I want to separate two things: methodology and philosophy. The relevant part is always the philosophy. If a company or professional hasn’t absorbed the Agile philosophy, they’ll rarely be truly Agile, no matter how much methodology, meaning a series of procedures, they implement.

You can read recipes for French dishes all day, but until you understand how a real chef thinks, until you absorb French culture, you’ll rarely cook decent French food. You’ll produce low-quality mechanical copies.

What matters isn’t the “how” but the “why.” The Agile Manifesto says so right in its first value. Let’s recall the four Agile values:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The first value already says it all: individuals over processes. In this article I want to show why the vast majority of companies are not effectively Agile, even when they implement Agile “methodologies.”

Principles Behind the Agile Manifesto

To recap, it’s worth citing the 12 Principles behind the Manifesto. Plenty of people have read all these items, but “reading” and “understanding” are two completely different things. For the purposes of my explanation, I’ll put some keywords in bold so I can refer back to them later.

We follow these principles:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

  • Business people and developers must work together daily throughout the project.

  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

  • Working software is the primary measure of progress.

  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

  • Continuous attention to technical excellence and good design enhances agility.

  • Simplicity–the art of maximizing the amount of work not done–is essential.

  • The best architectures, requirements, and designs emerge from self-organizing teams.

  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

The Fifth Element

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

I don’t know how the Manifesto’s founders arrived at these principles, but based on this point alone I imagine they’re very experienced people. This element, for me, is the densest of all the Principles.

Corollaries to the Fifth Element are the 11th and 12th principles. I’ll explain why.

Scaling Agility

I recently cited a chapter from the book Scaling Lean and Agile Development, by Craig Larman and Bas Vodde. I believe many people didn’t have the patience to read it, so I’ll summarize the part that interests me.

The PDF discusses the differences between Feature Teams and Component Teams. Simplified, an Agile team is necessarily a Feature Team: a team as independent as possible, one that owns a whole product or a complete feature of a product, from start to finish, from requirements to customer contact.

A Component Team is the traditional, departmental style. Each team answers for only a slice of several products: interface team, infrastructure team, architecture team, visual components team, database team, quality team, and so on.

Feature Teams are cross-functional, usually made of generalists. Component Teams are narrow, usually made of specialists. A Scrum Team is, by definition, a Feature Team, capable of carrying out all the work on a Product Backlog item.

As I said before, forget the “how” for now.

Conway’s Law

Melvin Conway, in April 1968, wrote a paper in which one passage would enter the annals of computing history:

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.”

As explained on his site, the famous Frederick Brooks cited this idea in the classic The Mythical Man Month (which every tech professional should read), naming it Conway’s Law. Brooks recognized that the law had important corollaries in management theory. Here’s one statement of his:

“Because the design that occurs first is almost never the best possible, the prevailing system concept may need to change. Therefore, flexibility of organization is important to effective design.”

As I already said in Killing the Average (in Portuguese), to this day we still apply Gaussian methodologies and processes. I haven’t managed to write a more complete piece about it yet, so watch the video I recorded and read the reference material I point to there. For now it’s enough to understand that most of the “uncontested truths” of Organization Theory no longer apply.

These outdated theories, with their rigid chains of command, “feudal” departments, and cultures of title and political power, were made to give control to section heads in 19th-century factories, where the most anyone expected from a worker was tightening screws.

As Conway’s Law says, teams create structures that mirror the structure of the organization. The corollary: a company that rewards mediocre work gets mediocre teams. Remember the Fifth Element: "…give them the environment and support they need…"

The Larman and Vodde PDF mentions Brad Silverberg, former senior VP of Windows and Office at Microsoft, who emphasized:

“The software tends to mirror the structure of the organization that built it. If you have a big, slow organization, you tend to build big, slow software.”

An ironic statement, but a true one.

Generalists vs Specialists

Traditional companies that fool themselves into thinking they’ll get total control over their teams only manage to create unproductive employees with no drive, complacent, limited, who will never learn anything new.

That’s because the Total Control style prefers factory-floor division of labor, where each team does one part of the whole. In this management style, the incentive for Specialists reigns.

A specialist is the employee who started a certain piece of the system and is the only one who knows how that piece works. He’s not comfortable sharing that knowledge, and even less comfortable accepting outside changes. He usually gets treated as a hero: when there’s an emergency, nobody else has time to learn that piece, and only he can fix the problem.

If you have heroes of this kind in your company, they should be the first ones fired.

There’s a reason Scrum teams are, by definition, Feature Teams. Otherwise, the first effect that surfaces is this one:

Recognize it? Each Component Team runs its Sprint, its iteration, and the next team’s iteration only starts when the previous one ends. That’s because no team owns the complete feature and they all depend on pieces from the others.

There’s a name for this: welcome back to Waterfall! It doesn’t matter that people call it mini-waterfall: a cascade is a cascade, whatever its size.

There’s another serious side effect: since nobody effectively owns the product, no team feels responsible for the whole, only for its part. Naturally the behavior emerges: “I did my part, it’s the other team’s fault.”

And there’s more. Component Teams deal with a single type of problem, which forces their members into becoming specialists. That professional narrows his own knowledge and gets absolutely no motivation to learn new things.

Remember the Fifth Element? “Build projects around motivated individuals…” Where are the motivated individuals in an organization that pushes everyone toward mediocrity? Of course nobody needs to know everything about everything: each professional will always have disciplines where they fit best. But everyone should be encouraged to know a bit of the rest.

That’s why Feature Teams matter: you have several specialists, and each one knows a little of what the colleague next to them knows how to do. With that, Pair Programming gains new meaning.

And Test Driven Development starts making much more sense when the team is truly responsible for a product or feature that adds value to the end customer. With no dependency between teams, it becomes clear how to run complete tests. But there’s another question about testing that I’ll explain further below.

Self-Organization

This is the crucial point of the new organizational theories. Don Tapscott and Anthony D. Williams would call it Wikinomics, the economics of collaboration.

Traditional organizations are horrified by chaos and pursue absolute control in a pathological way, to the point of harm.

And they should indeed be horrified by chaos. What they need to understand is that there’s a phenomenon of order emerging out of chaos.

We’re used to thinking in isolated events, in results that are the sum of independent events. Add one spoonful of sugar and the tea gets sweet. Add two and the tea gets twice as sweet.

Dynamic systems don’t work like that. Certain systems are very sensitive to initial conditions and produce non-linear results. Natural phenomena like social relationships, food chains, and economic events are all non-linear systems.

This goes back to my talk about Power Law Distributions, or Pareto Distributions. Recapping: a Platonic, linear world can be modeled according to Gauss. That kind of distribution is comfortable for analysts because it has a defined mean and a stable standard deviation. Power Laws are the opposite: characterized by the absence of a mean and by a standard deviation that tends to infinity.

The most obvious point: bell curves like the Normal require independent, isolated events, like rolling dice or flipping a fair coin. Human behavior can be many things, but independent is not one of them: by definition, humans relate to each other, so we form highly dependent systems.

And here’s the counterintuitive part. Dynamic networks don’t form random connections, whose distribution would be Normal. They exhibit a Pareto distribution. Albert-László Barabási explains in detail how scale-free networks form.

If we think of nodes (people, animals, neurons, viruses) and connections (friendship, transmission, synapse), the natural thing is to imagine nodes connecting in a random, chaotic way. Barabási’s studies and detailed observation of natural phenomena showed the opposite: these networks tend to be scale-free, with a few nodes concentrating the vast majority of connections and many nodes splitting the few that remain, forming something like this:

For those of us in tech, this looks like a map of the Internet, where the nodes are websites and the connections are, literally, the links between them. And that’s exactly right: the Internet follows a Pareto distribution.

For those with socialist tendencies, forget Marx, who was obviously wrong to try dragging all of society down to the average. The idea of “taking from the rich to give to the poor” is anti-natural. The natural thing is the opposite: a few people will always hold the vast majority of the world’s wealth, while the majority has less. The only way for the poor to get richer is to make the entire system richer, including the rich themselves. Nature favors meritocracy, never mediocrity.

Assuming everyone studies a bit more about Barabási, Poincaré, Mandelbrot, Zipf, Pareto, Bak, and subjects like scale-free networks, power laws, self-organized criticality, phase transition, chaos, and fractals, we can conclude quickly: order does tend to emerge from chaos. Networks form as Barabási described, through mechanisms like preferential attachment, and in the end we get scale-free, self-organized networks.

80/20

The famous 80/20 rule didn’t come out of nowhere. In a study in Italy, Vilfredo Pareto found that 80% of Italian land was in the hands of no more than 20% of the population. Hence “80/20.”

That split is exactly what a Pareto distribution shows:

As Chris Anderson explains in The Long Tail, think about Wikipedia. By the time the book came out, in 2006, Wikipedia already had about a million articles, an order of magnitude above the Encyclopædia Britannica.

Jimmy Wales’s idea was bold and controversial, despite the good intention: providing a rich, extensive, free encyclopedia to everyone in the world, including poor children in underdeveloped countries who might otherwise never have access to information.

Wales started with the Nupedia project in 2000, with few entries and an idea that had already caught on with Linus Torvalds and his Linux: a totally open platform where anyone could contribute, review, refine.

Looking from 2008, everyone would say Jimbo (as he’s known) is a genius. In 2000, he was considered crazy. In retrospect, it’s always easy to build a narrative that fits perfectly into events that already happened. As Nassim Nicholas Taleb would say: after a Black Swan happens, explaining it is easy; before it happens, predicting it is impossible.

Following Torvalds’s example, Jimbo created an environment suited for collaborators. He knew how to motivate people and, most importantly, how to trust them: unlike the traditional editorial system, there would be no editors and no filters, and everything anyone typed would be available immediately. The hope was that gross errors would be quickly corrected by the collaborators themselves, as Eric S. Raymond describes in the classic The Cathedral and the Bazaar:

“Given enough eyeballs, all bugs are shallow.”

Eric called this “Linus’s Law,” which can also be put this way: “Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.”

Wikipedia leveraged this same law. In a dynamic, open system, where collaborators trickle in and evolve into a self-organized scale-free network, errors will happen, but most will be corrected fast. The benefit of gathering information from thousands of people around the world is orders of magnitude greater than the small defects that show up now and then.

In the traditional control model, people think like Britannica: few entries, all of them very accurate. Which is worth more: a smaller encyclopedia with a near-zero error rate, or one ten times larger with a small error rate? Some retrograde critics still believe one small error on Wikipedia invalidates a million hits.

Software as Art

Pete McBreen wrote back in 2001 about Software Craftsmanship, and I’m a defender of that definition.

Many people think of software, of programming, as Engineering. Sorry to inform them: it isn’t. Software is music: developing software is very close to composing a song.

Software is also painting: developing is like painting a picture. No disrespect to engineering, which already brought us wonders like the Great Wall of China and the Egyptian pyramids, but in software’s case many people find it simpler to think of it as engineering than as art.

The reason is the same as always: the attempt at control. Engineering is predictable, controllable, measurable. Art is creative, rebellious, unpredictable, chaotic. I find it very interesting that great figures of the past like Pythagoras and Da Vinci were great generalists, artists with a body of work in science and mathematics. That’s exactly what a software developer needs to be: a Renaissance artist.

As I said before, a member of a Feature Team has his specialties, but keeps an absolutely open mind to try new things, learn new crafts, explore and create.

Art can’t be implemented by reading procedures. And that’s exactly what many of those who call themselves “developers” or “programmers” do: they learn one or a few ways of doing things and repeat forever what they were taught. Artists learn from mentors, train tirelessly in a long process of trial and error, draw inspiration from the work of other masters, understand their techniques and try to blend them into their own.

Without claiming this as absolute truth, I tend to think that developers who actively participate in open source projects, like Linux, are far more complete programmers than those who left college straight into Component Teams inside traditional organizations. Especially if they stayed too many years in the same organization.

A specialist programmer, member of a Component Team in a traditional Gaussian organization, is like a house painter: he only knows how to run the whitewash roller up and down, symmetrically, without a drop of creativity and with no talent whatsoever for learning and self-evolution.

80% of an employee’s professional profile is a direct reflection of the organization they work for. The other 20% is the employee’s own fault, for refusing to leave the comfort zone. Both carry 100% of the blame for only 29% of software projects being considered successes, and for the USD 55 billion spent on cancelled projects. (source: Standish Chaos Reports)

The Open Source World

I believe there’s no need to explain much more about open source projects. There’s no miracle: a project won’t succeed automatically just because it’s open code. Quite the contrary, hundreds of projects never see the light of day.

Again, we’re talking about Pareto: maybe only 20% of open source projects achieve great success. However, the other 80% are identified as failures much faster: some merge into bigger projects, some simply stop. The decision to stop is much faster and more effective than in traditional corporate projects, which have already invested resources, time, money, and the reputation of those involved.

With everything I explained above, it’s easy to understand that open source projects start with simple initial conditions: an idea, a small implementation, a few people. And it starts making sense how they evolve from chaos to scale-free networks through self-organization.

Rigid prior consensus doesn’t exist in this environment: these projects could only evolve into Feature Teams, where collaborators have different and complementary skills.

It’s also not hard to understand that, as on Wikipedia, the most important or best-known entries get filled in first, and the more obscure ones, over time. In the best Pareto style, 20% of the priorities happen first. In an environment of scarce resources, with no physical presence and volunteer collaborators, motivating people matters even more, and the priorities that deliver the most value to the group as a whole get implemented first.

Once you understand software as art, it’s simple to see that developers who participate in several open source projects are automatically exposed to many different expressions of art and, with that, to different ways of implementing software. A good developer starts incorporating these differences into his own style, greatly improving the quality of his work.

A developer alone, or in a familiar team, has little reason to write tests for his own code. But when he collaborates in a community where any stranger can see his code and assess his reputation, the motivation to write quality code, decently covered by tests, becomes obvious. That’s why I said earlier that Test Driven Development starts making even more sense and becomes a real necessity.

The project’s founder, the developer or group of programmers who started it, will be forced to manage it. In an open environment, with no titles, no salaries, no bosses and no direct clients, any attempt to use Gaussian techniques of traditional project management goes down the drain. Now we’re talking about real projects, without the comfort zone of the cubicle.

The project maintainer will find himself in a position where he’s forced to make decisions. He’ll quickly understand that unanimity is impossible and will put on the hat of benevolent dictator: if he’s too rigid and authoritarian, he’ll drive away all his collaborators, who have no obligation to follow him; if he’s too flexible, he’ll show insecurity, indecision, sluggishness, and may motivate a coup, a kind of state coup where the project gets forked and a more charismatic and effective maintainer takes his place. Or worse: the project may simply stall and cease to exist.

There’s no environment more hostile and at the same time more rewarding for a true project manager than open source. Take away a manager’s title and power, and only then can you assess whether he knows what “managing” means.

Agile Principles, Redux

With all that said, I think we can recap the 5th, 11th, and 12th principles:

  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

  • The best architectures, requirements, and designs emerge from self-organizing teams.

  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

The other principles are consequences.

Given a proper environment, with professionals raised above the average and motivated, we can truly trust their capacity for self-organization. The organic, non-hierarchical structure will lead its members to readjust their routines according to the problems they face. That generates quality code, with only the essential being produced, attention to refactoring, testing, and continuous integration, which naturally leads to the best architectures.

The system as a whole feeds itself in a continuous positive feedback loop, creating a sustainable, always productive environment, with professionals researching and implementing innovations that, from time to time, deliver leaps in quality and productivity for the whole company.

As a result, the customer receives products that add real value, and changing requirements can be accepted without much trouble, because the organization is flexible and each member feels responsible for the whole. In this virtuous cycle, professionals are in constant learning, growing their skills at an ever-increasing pace. The result is an innovative company, above average, that doesn’t depend on the past to try to predict the future: its professionals are prepared for whatever future arrives. Constant change doesn’t scare them anymore; on the contrary, they want change.

How to Get There?

It was no accident that I wrote about:

The first two articles focus on the professional: an attempt to wake up the clock-punching employees to the fact that the world isn’t static, the future isn’t stable, and the Gaussian world is an illusion.

The last two talk specifically about a tool: Git. It’s a hint for creating the environment a developer needs, as stated in the Fifth Principle. But a tool, like a methodology, is useless if company and employee don’t internalize the Agile Values and Principles.

Hence this article.

As I said at the start, I don’t consider myself any great scholar of this philosophy, but for some reason I identify with its foundations and I clearly observe its practical application in open source projects. It’s also clear that, in Pareto’s world, this model survived and bore impressive fruit, like Wikipedia and the fact that most of the world’s web servers run Apache.

Is software part of your company’s core business? Apply the “open source” model, or the “bazaar” model, per Eric Raymond. Without necessarily opening your code to the public on the internet, of course.

Create a simple repository, one all employees have unrestricted access to, with a low barrier to adoption. Encourage them to participate in projects outside their traditional departments. At first, badly written code, low quality, no tests, and every kind of defect will surface. But pointing fingers solves nothing: the goal is to break the vicious cycle that generates this kind of software.

“A bad programmer will write bad code, no matter what language or tool you give him.” So the goal is to create excellent programmers, not to swap tools. Impressively, a good programmer will write good code even in ASP or Perl (again, no offense to Perl, I’m only speaking to its reputation, created mainly by bad programmers).

Like a sculpture, it’s time to trim the edges, redo some pieces, reshape what doesn’t look right, and turn this work into a piece of art, collaboratively. It’s the best way to avoid waste: whoever has spare time on one team can help their busier colleagues on the other.

In the beginning there will be disorder and signs of chaos. There will be duplicated work, desynchronization, and communication problems, since nobody was used to actually communicating. Skill and knowledge gaps will become obvious. Every existing problem will come to the surface, and it will be ugly and uncomfortable.

However, with a bit of persistence and trust in people, the group as a whole will come out of the chaos. The true leaders will emerge as hubs in the scale-free network. The preferential attachment phenomenon will start outlining the order. Given enough time, people will self-organize organically.

From there, yes, we can start talking about sustainable growth, with a constant or growing pace of productivity.

A Day of Innovation

At Google there’s that old story that every employee gets one day a week to do whatever they want.

Put that way, the first image that comes to mind is a bunch of kids riding bicycles, playing video games, and sipping caipirinhas by the campus pool.

But assuming you read the whole article, picture Orkut Büyükkökten on one of those days. He has the right environment, the right culture, the right motivation, the right knowledge. He decides to start a personal project to experiment with social networking concepts.

He has a place to put his code, and he understands he must detach from it. He naturally understands how open source-style collaboration works and, because of that, he knows how to communicate.

He announces the project internally. Members of his team and others, all tuned into the culture of proactivity, innovation, acceptance of change, and collaboration, immediately understand the value of the idea. More than that: they know where to download the code and how to start collaborating.

I don’t know how Google really works, I never worked there, and it probably has as many problems as any normal company. Even so, I fantasize that many of their products started like this: in a permissive environment, oriented toward innovation. Hiring PhDs from MIT or Stanford is not enough if there’s no proper environment and culture to make them truly produce. Since Larry and Sergey started out in this permissive open source world, I keep wondering whether they created an organization that follows exactly this model, even if by instinct.

Many companies want to be the next Google. Worth remembering it takes much more than comfortable sofas, foosball tables, video game rooms, and a Japanese restaurant inside the company. That part is easy: just buy it.

The hard part is cultivating a culture. Many companies complain that good professionals resign and go elsewhere. Obvious: truly intelligent people don’t put up with a Gaussian culture for long. We don’t like sameness, retrograde thinking, and lack of attitude. True artists need environments that inspire creativity.

The average company’s cubicle is a terrible place to create.

Bibliography

Finally, once the philosophy is understood, we can go back to methodology. Now it makes sense to apply the tools that methodologies like XP or Scrum advocate: Pair Programming, Planning Game, Test Driven Development, Continuous Integration, Refactoring, Small Releases, Collective Code Ownership, Simple Design, Sustainable Pace, and so on.

Read my reading recommendations with fresh eyes:

Tip: most of these books have translations into several languages.

I know some of these books don’t relate directly to this subject (like Carl Sagan’s), but believe me: they make a huge difference in how we form our ideas.