[Off-Topic] You Don't Understand Anything About Scrum

“And neither do I. But I keep studying :-P”

My research style is quite Web-like, I’d say :-) For example, by some cosmic accident, only this week did I decide to read about the origin of Scrum, the Agile methodology that ended up becoming the most well-known today. I was very surprised by what I found.

I always defend that people need to constantly, obsessively research and never accept anything without argument. We should all already know that there are no silver bullets. Simply quoting someone to justify something is a fallacy I don’t accept, as I already said in my article Obedience to Authority.

That is, even to defend agile methodologies I need a better argument than saying “Kent Beck is awesome, Kent Beck invented XP, so I should use XP.” Or something like “Toyota is awesome, Toyota uses Lean, so I should use Lean.” That’s not an argument.

Fact: many companies today say they use Scrum, many people say they understand Scrum, but I’d bet a lot of money that the overwhelming majority don’t know more than Scrum-butt, or what we’d call, in a very Brazilian way, “half-baked Scrum.” Better than nothing, yes, but quite poor compared to the level of hyperproductivity that Scrum promises: 400% or more.

David Hume said:

A wise man, therefore, proportions his belief to the evidence.

Laplace’s Principle (Pierre Simon Laplace) says:

The weight of evidence for an extraordinary claim must be proportional to its strangeness.

Carl Sagan popularized:

Extraordinary claims require extraordinary evidence.

Given that I think this way, I need much more. I can even use something based on my intuition that it works, but it can’t become a comfort zone. This has led me to a reverse engineering and a research path that has lasted more than a year — represented dozens of books, dozens of technical papers, multidisciplinary research among the history of management, production engineering, psychology, sociology, physics, etc. It’s a search for the roots of what should constitute a true “agile organization.” For me, agile methodologies are the current stage in the evolution of project management, but they’re not the last! That is, implementing an agile methodology isn’t a goal, it’s a means.

And actually, the last few months have only been the most constant. Don’t forget that I certified as a PMP in 2005, and back then I was already studying PMBOK and OPM3. Two years before, my research line was RUP, SW-CMM (before CMMi), and Indian software factories. My first contact with agile methodologies was in 2003 with an introduction to Extreme Programming. In the 90s I followed the evolution of UML and RUP. Coad & Yourdon, Rumbaugh, Booch, Jacobson, etc.

This whole introduction is actually a mea culpa regarding recent talks I’ve given, especially Beyond Chaos: Random Thoughts on Agility. It’s the most recent and most daring version of a series of talks I gave throughout 2009 in several cities. It’s clear that many people didn’t have the background to understand all these concepts.

In that talk I dared to mention concepts like:

• Complexity Theory and Nonlinearity
• Chaos Theory (specifically the “Edge of Chaos”)
• Entropy Systems Theory
• Emergence and Self-Organization
• Complex Adaptive Systems
• Evolution and Punctuated Equilibrium
• Game Theory

I really did try to explain each of these topics, but I found out that talks of less than 1 hour are terrible for explaining such dense concepts at the same time. I’ll still find a better way.

Each of these topics represents an entire branch of scientific research. And I swear I’m not talking about pseudo-science — I’m talking about serious research. In fact, these branches are precisely the “answer” to the question: “how do agile methodologies — if properly implemented — effectively lead teams to a stage of sustainable hyperproductivity?”

The first time I mentioned “Self-Organization” at the beginning of 2009, I heard many people literally “making fun” of what I said. Some thought I had “invented” this “strange” concept. Unfortunately I’m not that smart :-) By chance, I reached the same conclusions as the creators of Scrum. Watch Jeff Sutherland, co-creator of Scrum, himself listing some of these concepts:

This excerpt is from the talk Self-Organization: The Secret Sauce for Improving Your Scrum Team, which he recently presented at Google Tech Talk.

I confess it was very strange to hear that reaction since I’m talking about people who — theoretically — should be agilists, since they claim to know how to implement Scrum. That’s why I published the article The Agile Manifesto, or How to Become Google where I started to interpret what everyone should have already memorized: the Agile Manifesto:

“The best architectures, requirements, and designs emerge from self-organizing teams.”

Yes, this is one of the 12 principles of the Manifesto. But if you don’t understand the scientific concept of “emergence” and “self-organization,” then you don’t understand the Manifesto and will have many problems trying to be Agile.

You know what else? Ken Schwaber was co-creator of Scrum in the mid-90s, together with Jeff Sutherland. He wrote a formal paper presented at OOPSLA-95. I can’t understand how anyone can say they “know Scrum” and have never read this paper. In it, Schwaber says:

“The closer a development team operates to the edge of chaos, while still maintaining order, the more competitive and useful the resulting system will be.”

Sutherland and Schwaber were some of the first Westerners to understand and publish about the enormous difference between Defined (predictable) and Empirical (unpredictable) processes. The Japanese had already understood this with Deming years earlier. Waterfall, RUP, and CMM methodologies are defined processes. Scrum, XP, and other Agile methodologies are mixed processes between defined and empirical. This is the first big difference that makes the two groups incomparable. Schwaber explains it very well in Controlling Chaos: Living on the Edge, presented at OOPSLA-96.

Also recently, I sometimes still hear someone wanting to start a discussion like “Scrum vs. XP” or “Scrum vs. Lean” or “Adding Scrum to Kanban” or “Adding Scrum to CMMi,” and it’s hard not to get frustrated with the futility of these discussions, where people only discuss the top of the iceberg and don’t come close to touching the base of the principles that precede each of these topics.

In the words of Scrum co-creator Jeff Sutherland, from November 2007:

After thinking about the early days of Scrum, I think the root of both Scrum and Lean are complex adaptive systems. When we created Scrum we didn’t talk about Lean, we talked about complex adaptive systems. I think Scrum and Lean are complementary implementations of ways of dealing with physical realities where things are usually non-linear, aren’t simple, and aren’t predictable.

The first Scrum implemented all the practices we know from today’s Scrum, plus everything that would later come to be called XP practices, plus a secret seasoning that creates hyperproductive teams.

(…) But more importantly, we captured the “punctuated equilibrium” effect, something I’m still yet to see any Scrum team achieve. It’s one of the secrets of hyperproductive teams. Before a developer picks a task in a Scrum meeting, the entire team discusses with them whether that task is the fastest path to a user-visible feature. That was a sophisticated implementation of self-organization and “test-first” development.

What’s funny is that, as I said before, I only read this description of Scrum’s history this week. I ended up arriving at the same topics of Complex Adaptive Systems, Punctuated Equilibrium, and the other topics I listed above, through different paths. I already knew the novelties of scale-free network theory, which gave rise to my 2008 talk “Killing the Average.” Finally, reading and rereading the Agile Manifesto I came across the word self-organization. The sum of these two things led to dozens of readings that culminated in Complex Adaptive Systems. It’s a research path that seems somewhat inevitable, and it seems Sutherland and Schwaber reached the same conclusion more than 15 years ago.

And more than that, Sutherland also says the following about the origin of Scrum:

The first Scrum team was created directly from a paper that is mandatory reading for any Scrum practitioner. It explains how to set up self-organizing teams and clearly outlines management’s role in the process.

Takeuchi and Nonaka. The New New Product Development Game. Harvard Business Review, 1986

A longer paper that goes deeper, from a book that’s out of print. Some may find it easier to read.

Ken-ichi Imai, Ikujiro Nonaka, and Hirotaka Takeuchi. Managing the New Product Development Process: How Japanese Companies Learn and Unlearn.

Hirotaka Takeuchi and Ikujiro Nonaka are Japanese professors also known for their research in Knowledge Management. The Harvard Business paper Sutherland mentions costs a mere USD 6.50. I just bought it, and it’s very interesting that the “Scrum” analogy from “Rugby” is explained and named in that paper. That is, the name “Scrum” originates from Takeuchi and Nonaka. More than that, they already explain in that paper what this concept of Self-Organization is and how it works. They also explain sequential style (Type A) and overlapping development phases (Types B and C). Sutherland commonly talks about hyperproductive teams as Type C Scrum.

What Takeuchi and Nonaka explain in detail in these papers is the following:

From interviews with members of organizations, from CEO to young engineers, we learned that leading companies demonstrate 6 characteristics in the management of new product processes:

• Internal Instability
• Self-organizing project teams
• Overlapping development phases
• “Multi-learning”
• Subtle control
• Organizational transfer of knowledge

Once again, each of these items is the subject of an entire branch of research, so I recommend not trying, superficially, to extract any meaning from just these few words. The goal here is just to give a “taste.” Buy the paper and read more about the two’s work, especially about Knowledge Management, which is a bit different from what most companies understand about knowledge (for a change).

Another thing some may not know:

“In the same year (1995), Sutherland supported the development of Extreme Programming by giving Kent Beck all the behind-the-scenes information on the creation of Scrum and the result of two years of product development with the Scrum process from 1993-95. XP’s engineering practices evolved alongside Scrum, and the two Agile development processes work well together. Scrum and XP are the most used Agile processes in the world, and their creators are signatories of the Agile Manifesto.”

This is in the Scrum Papers, which obviously, if you’re an agilist, you should read. The point is: Scrum is founded on the Japanese principles that gave rise to Lean, while also serving as inspiration for the creation of XP. That is, there is no conflict between Lean, Scrum, XP. They have similar roots! That’s why it’s important to study principles more than just practices.

Be careful: I’m not citing all of this as a way of “proving” my point. I’m citing studies that happened to coincide with what I started doing on my own. It’s all research material that shouldn’t be used as dogma, but as a reference for research.

Finally, to close, I like to mention Mary Poppendieck. Of the agilists, she and Tom decided to go on the most purist line and wrote Lean Software Development. With this explanation, it should be clear that what Mary/Tom and Sutherland/Schwaber say isn’t contradictory or competitive — they are only complementary and totally compatible, each with their own experience background. And from Mary, I think this article on ‘Blame’ is important to read:

The purpose of what has become today’s organizational structure was clear: assigning responsibility would allow “immediate detection of work negligence… and pointing out the offender.”

(…) There’s such a thing as standard work, but standards should change constantly. Instead, if you think of the standard as the best you can do, it’s all over. Standard work is just a baseline for doing more kaizen. It’s kai-aku [change for the worse] if things get worse than now, and it’s kai-zen [change for the better] if things improve. Standards are dictated arbitrarily by humans, so why shouldn’t they change?

That last paragraph is, again, for those who still think silver bullets exist, that there’s some body of unshakable practices (CMMI, ITIL, RUP, etc.) where simply “following the recipe to the letter” will magically make everything work. After all, they’re “industry standards.”

Bullshit! Do your homework: keep researching!

PS: Read all my articles from the Off-Topic section for more related topics.