[Off-Topic] You Don't Understand Anything About Scrum
“And neither do I. But I keep studying :-P”
My research style is quite Web-like, I’d say :-) For example, by some cosmic accident, only this week did I decide to read about the origin of Scrum, the Agile methodology that ended up becoming the most well-known today. I was very surprised by what I found.
I always defend that people need to constantly, obsessively research and never accept anything without argument. We should all already know that there are no silver bullets. Simply quoting someone to justify something is a fallacy I don’t accept, as I already said in my article Obedience to Authority.
That is, even to defend agile methodologies I need a better argument than saying “Kent Beck is awesome, Kent Beck invented XP, so I should use XP.” Or something like “Toyota is awesome, Toyota uses Lean, so I should use Lean.” That’s not an argument.
Fact: many companies today say they use Scrum, many people say they understand Scrum, but I’d bet a lot of money that the overwhelming majority don’t know more than Scrum-butt, or what we’d call, in a very Brazilian way, “half-baked Scrum.” Better than nothing, yes, but quite poor compared to the level of hyperproductivity that Scrum promises: 400% or more.
David Hume said:
A wise man, therefore, proportions his belief to the evidence.
Laplace’s Principle (Pierre Simon Laplace) says:
The weight of evidence for an extraordinary claim must be proportional to its strangeness.
Carl Sagan popularized:
Extraordinary claims require extraordinary evidence.
Given that I think this way, I need much more. I can use something based on my hunch that it works, but it can’t turn into a comfort zone. This led me into reverse engineering and a research path that has already lasted more than a year: dozens of books, dozens of technical papers, multidisciplinary research spanning the history of management, production engineering, psychology, sociology, and physics.
It’s a search for the roots of what should make a real “agile organization.” For me, agile methodologies are the current stage in the evolution of project management, and still far from the last one. Implementing an agile methodology is a means to get somewhere bigger, never the end goal.
In reality, the last few months have only been the most constant stretch. Worth remembering that I certified as a PMP in 2005, and back then I was already studying PMBOK and OPM3. Two years before, my research line was RUP, SW-CMM (before CMMi), and Indian software factories. My first contact with agile methodologies was in 2003, with an introduction to Extreme Programming, and in the 90s I followed the evolution of UML and RUP: Coad & Yourdon, Rumbaugh, Booch, Jacobson.
This whole introduction is a mea culpa about recent talks I’ve given, especially “Beyond Chaos: Random Thoughts on Agility.” It was the most recent and most daring version of a series of talks I gave throughout 2009 in several cities. It became clear that many people didn’t have the background to follow all these concepts.
In that talk I dared to mention concepts like:
- Complexity Theory and Nonlinearity
- Chaos Theory (specifically the “Edge of Chaos”)
- Entropy Systems Theory
- Emergence and Self-Organization
- Complex Adaptive Systems
- Evolution and Punctuated Equilibrium
- Game Theory
I really did try to explain each of these topics, but I found out that talks of less than 1 hour are terrible for explaining such dense concepts at the same time. I’ll still find a better way.
Each of these topics represents an entire branch of scientific research. And I swear I’m not talking about pseudo-science, this is serious research. In fact, these branches are precisely the “answer” to the question: “how do agile methodologies, if properly implemented, lead teams to a stage of sustainable hyperproductivity?”
The first time I mentioned “Self-Organization” at the beginning of 2009, I heard many people literally “making fun” of what I said. Some thought I had “invented” this “strange” concept. Unfortunately I’m not that smart :-) By chance, I reached the same conclusions as the creators of Scrum. Watch Jeff Sutherland, co-creator of Scrum, himself listing some of these concepts:
This excerpt is from the talk Self-Organization: The Secret Sauce for Improving Your Scrum Team, which he recently presented at Google Tech Talk.
I confess it was very strange to hear that reaction, since I’m talking about people who, in theory, should be agilists, given that they claim to know how to implement Scrum. That’s why I published the article The Agile Manifesto, or How to Become Google, where I started to interpret what everyone should have already memorized: the Agile Manifesto:
“The best architectures, requirements, and designs emerge from self-organizing teams.”
Yes, this is one of the 12 principles of the Manifesto. But if you don’t understand the scientific concept of “emergence” and “self-organization,” then you don’t understand the Manifesto and will have many problems trying to be Agile.
You know what else? Ken Schwaber was co-creator of Scrum in the mid-90s, together with Jeff Sutherland. He wrote a formal paper presented at OOPSLA-95. I can’t understand how anyone can say they “know Scrum” and have never read this paper. In it, Schwaber says:
“The closer the development team operates to the edge of chaos, while still maintaining order, the more competitive and useful the resulting system will be.”
Sutherland and Schwaber were some of the first Westerners to understand and publish about the enormous difference between Defined (predictable) and Empirical (unpredictable) processes. The Japanese had already gotten there with Deming years earlier.
Waterfall, RUP, and CMM are defined processes. Scrum, XP, and other Agile methodologies are mixed, somewhere between defined and empirical. This is the first big difference that makes the two groups incomparable, and Schwaber explains it very well in Controlling Chaos: Living on the Edge, presented at OOPSLA-96.
Even today I hear someone wanting to start a discussion like “Scrum vs. XP,” “Scrum vs. Lean,” “Adding Scrum to Kanban,” or “Adding Scrum to CMMi.” It’s hard not to get frustrated with the futility of these conversations, where people only argue about the top of the iceberg and never come close to touching the base of the principles that precede each of these topics.
In the words of Scrum co-creator Jeff Sutherland, from November 2007:
After thinking about the early days of Scrum I think the root of both Scrum and Lean is complex adaptive systems theory. When we created Scrum we did not talk about Lean, we talked about complex adaptive systems. I think Scrum and Lean are complementary implementations of ways to deal with physical reality where things are often not linear, not simple, and not predictable.
The first Scrum implemented all the practices we know as Scrum today plus all of what later became known as XP practices plus some secret sauce that creates hyperproductive teams.
(…) But most important, we captured the “punctuated equilibrium” effect, something I have yet to see another Scrum team do. It is one of the secrets of hyperproductive teams. Before a developer chooses a task in the Scrum meeting, the entire team discusses with the developer whether that task will be the fastest path to an observable end-use feature. This was a sophisticated implementation of “self-organization” and “test first” development.
What’s funny is that, as I said before, I only read this account of Scrum’s history this week. I arrived at the same topics of Complex Adaptive Systems, Punctuated Equilibrium, and the others I listed above through different paths.
I already knew the novelties of scale-free network theory, which gave rise to my 2008 talk “Killing the Average.” And, reading and rereading the Agile Manifesto, I came across the word self-organization. The sum of these two things led to dozens of readings that culminated in Complex Adaptive Systems. It’s a research path that seems somewhat inevitable, and it seems Sutherland and Schwaber reached the same conclusion more than 15 years ago.
And more than that, Sutherland also says the following about the origin of Scrum:
The first Scrum team was created directly from a paper which is required reading for any Scrum practitioner. It explains how to set up self-organizing teams and clearly outlines management’s role in the process.
Takeuchi and Nonaka. The New New Product Development Game. Harvard Business Review, 1986
A longer paper from a book which is out of print goes into more depth. Some may find this easier reading.
Ken-ichi Imai, Ikujiro Nonaka, and Hirotaka Takeuchi. Managing the New Product Development Process: How Japanese Companies Learn and Unlearn.
Hirotaka Takeuchi and Ikujiro Nonaka are Japanese professors also known for their research in Knowledge Management. The Harvard Business paper Sutherland mentions costs a mere USD 6.50. I just bought it, and it’s telling that the “Scrum” analogy from “Rugby” is already explained and named right there. That is, the name “Scrum” comes from Takeuchi and Nonaka.
More than that, they already explain in that paper what the concept of Self-Organization is and how it works. They also explain sequential style (Type A) and overlapping development phases (Types B and C). Sutherland usually talks about hyperproductive teams as Type C Scrum.
What Takeuchi and Nonaka explain in detail in these papers is the following:
From interviews with organization members from the CEO to young engineers, we learned that leading companies show six characteristics in managing their new product development processes:
- Built-in instability
- Self-organizing project teams
- Overlapping development phases
- “Multilearning”
- Subtle control
- Organizational transfer of learning
Once again, each of these items is the subject of an entire branch of research, so don’t try to extract any meaning from just these few words. The goal here is only to give a “taste.” Buy the paper and read more about the two’s work, especially about Knowledge Management, which is a bit different from what most companies understand by knowledge (for a change).
Another thing some may not know:
“In the same year, Sutherland provided support for development of eXtreme Programming by giving Kent Beck all background information on the creation of Scrum and the results of two years of product development with the Scrum process from 1993-95. XP engineering practices evolved along with Scrum and the two leading Agile development processes work well together. Scrum and XP are the most widely used Agile processes worldwide and their creators are signatories of the Agile Manifesto.”
This is in the Scrum Papers, which obviously every agilist should read. The point is: Scrum is founded on the Japanese principles that gave rise to Lean, while also serving as inspiration for the creation of XP. There is no conflict between Lean, Scrum, and XP. They have similar roots! That’s why it’s important to study principles more than just practices.
Be careful: I’m not citing all of this as a way of “proving” my point. I’m citing studies that happened to coincide with what I started doing on my own. It’s all research material that shouldn’t be used as dogma, but as a reference for research.
To close, I like to mention Mary Poppendieck. Among the agilists, she and Tom decided to take the most purist line and wrote Lean Software Development. With this explanation, it should be clear that what Mary/Tom and Sutherland/Schwaber say is complementary and totally compatible, each with their own experience background. And from Mary, I think this article on ‘Blame’ is important to read:
The purpose of what became today’s organization chart was clear: The assignment of responsibility would enable “prompt detection of derelictions of duty… and point out the delinquent.”
(…) There is something called standard work, but standards should be changed constantly. Instead, if you think of the standard as the best you can do, it’s all over. The standard work is only a baseline for doing further kaizen. It is kai-aku [change for the worse] if things get worse than now, and it is kaizen [change for the better] if things get better than now. Standards are set arbitrarily by humans, so how can they not change?
That last paragraph is, again, for those who still think silver bullets exist, that there’s some body of unshakable practices (CMMI, ITIL, RUP, etc.) where simply “following the recipe to the letter” will magically make everything work. After all, they’re “industry standards.”
Bullshit! Do your homework: keep researching!
PS: Read all my articles from the Off-Topic section for more related topics.