[Off-Topic] Obedience to Authority

Update: 11/20 I updated the videos to the Portuguese-subtitled versions on Vimeo.

Update: 11/10 I added a final section on a recent topic from Fred Brooks.

In my recent talks on organizations, I show a video of the Asch experiment. It demonstrates how people go along with a group even when they know the group may be wrong. For all sorts of reasons, they conform anyway.

For us agilists, think of a team doing Planning Poker or a Retrospective. When most of the team answers one way, the minority tends to go with the group. Some hold their ground more firmly, but the default is conformity, and that has to be taken into account. Watch to understand:

Asch Conformity Experiment from Fabio Akita on Vimeo.

On top of that, we still run on organizations with outdated structures. Hierarchies, positions, bosses, procedures, policies. All of that, as I say in the talks, doesn’t work anymore and only limits the organization as a whole. Does it bring short-term results? Sure, that’s why everyone still uses it. In the long run, it builds a sick organization.

One of the factors is authority. In this kind of organization, when there’s a “boss,” the effect is simple: responsibility for the executed task shifts from the subordinate to whoever gave the order. It’s what everyone has heard before: “I was just following orders,” or “I just work here.”

This holds even for atrocities (Nazis torturing Jews, for example). One theory says the person may even understand that what they’re doing is wrong, but since the decision came from an authority, they don’t feel the same remorse, don’t think about alternatives, and don’t even consider just stopping.

The one who demonstrated this was Stanley Milgram, back in the 60s. The BBC redid the same experiment, and you can watch my subtitled summary:

These are all real dangers that happen in our day-to-day:

• People conform very easily
• People don’t like looking wrong, so they don’t take risks
• People obey authorities, whether bosses or people with higher status in some way

That’s one of the reasons so many agility projects fail. It’s no use saying “the team decides the best path” if there’s still someone with veto power. The veto only has to be used once for no one to take anything seriously again.

Current organizations limit people, their potential, any chance of motivation, any individuality, and in the end they turn people into robots. The more an individual conforms to this structure, the more amorphous they become, almost a walking piece of meat.

So the only way out is to eliminate power hierarchies. Watch my introduction to this idea in the talk I gave at Rails Summit 2009 (video credits to @agaelebe, thanks for filming):

Organizations, as classical, closed systems, tend to stagnate and collapse. It can take ten years, but that’s what happens, naturally. Complex adaptive systems, on the other hand, keep evolving, learn from small failures, and refine their own processes.

That requires heterarchies in place of hierarchies, spontaneously formed groups, with decentralized control. Hard to swallow? I agree, and that’s exactly why the people currently in positions of authority should be a bit more literate before they go off executing. Which brings me to another point:

• People today deal with knowledge superficially: they’d rather “do” than “waste time” learning.

Dan Pink gave a TED talk on motivation. To recap the starting point: no one can motivate another person, so don’t even try. People motivate themselves or they don’t, and the most a company or a manager can do is demotivate them. So the main thing is to stop getting in the way.

What Dan shows is the obvious: companies keep using archaic techniques based more on folklore than on science. The system of rewards and punishments doesn’t work. Take a look:

He says people need three things to motivate themselves: Autonomy, the desire to run their own lives; Mastery, the desire to get better and better at what they do; and Purpose, being part of something bigger than themselves. What he’s describing, deep down, are dynamic, interactive Agents in a Complex Adaptive System. These agents interact with each other, cooperate and compete, and evolve around “Strange Attractors,” which are the purpose or identity of the group or company.

Agility is a natural evolution of the old management processes. Democratic Organizations are the longest and most far-reaching step for an organization.

This article is just an introduction. Reread my off-topic articles from the last few months to get more of the picture.

Fred Brooks

I had the pleasure of watching Fred Brooks at Latinoware this year. I was a little frustrated by the thin crowd: his talk should have been packed and had barely anyone. A sign that most people don’t even know who he is or don’t care about the subject, which is pretty sad.

Fred talked about a couple of chapters from the book he’s writing. Let me note that he didn’t present the full content, so a lot of what I speculate here may be explained in the other chapters.

The theme was the importance of “someone” being the Master Guide of a project. He says projects, especially the large ones, need a Chief Architect or something similar to keep the final product coherent. He gave examples like building an airplane with separate teams, with people talking to all of them all the time to make sure the whole thing stayed integrated and consistent.

I agree with the explanation. It’s satisfactory, and complex system architectures really don’t just “appear” out of nowhere from several people or teams working separately.

There is, however, an enormous caveat here. Depending on how you read this explanation, if you hold the title of “Architect” or are certified in “Systems Architecture,” you might feel very justified by a celebrity like Fred saying architects are vital to a project. Well, you’re wrong.

An architect is, in the end, a leader, a condition earned by merit. Position, certification, or someone’s orders don’t enter into it. Take the Linux kernel. No one appointed Linus Torvalds to the post of “benevolent dictator” of the project. He emerged from a need, people accepted his leadership, and if they’d been very unhappy, all it took was a fork for a new leader to step in.

Real leaders emerge on their own. As a project like this grows, other leaders come up naturally in each part of it. The ones Linus calls his “lieutenants” show up, like Andrew Morton once was.

This point matters: leadership is acquired from the bottom up, and it isn’t a post of authority dictated from the top down. Leaders evangelize their point of view, convince people, and to a degree can even impose certain positions. But when there’s abuse of that position, the team around them can and should reassert their values. And if the fight ends with no winner, the way out is to leave the project, instead of conforming and carrying on, as the Asch and Milgram experiments show.

This works really well in the open source world because it’s a distributed environment, with no fixed hierarchies or positions. People move between the various parts of a project, there’s discussion, opinions count, and new ideas get tested in practice. It’s the polar opposite of how software gets made at a traditional company. A denser take is in the classic The Cathedral and the Bazaar, by Eric Raymond.

Someone who thinks they can solve a given problem starts taking the reins. The people around them see value in it and start collaborating. The initial architecture goes through revisions and evolves. That “leader,” usually the author, keeps the initial vision alive and folds in the proposed changes, even when they’re better than his own ideas, instead of falling into the backward stance of “I didn’t make it, so I don’t approve.”

Understanding the open source world is the key to understanding agility and democratic organizations. They were born this way because of the environment’s constraints: people spread around the world, volunteers, no physical contact, the need to keep the code coherent.

Leading is different from bossing. A leader has no right to just bark orders. A real leader is a servant leader: he collaborates and helps the team instead of ordering people around and telling them how to do it. He influences, evangelizes, surfaces other points of view, and seeks consensus.

Sometimes he has to make the harder calls, like going against the majority to preserve the project’s integrity. But do that the wrong way too often and the team should pull him from the position. Here, Emotional Intelligence counts as much as, or more than, raw IQ.

So yes, projects need a vision that ties everything together. That doesn’t justify anointing someone autocratically as the “architect,” though. In the best projects, that position emerges on its own. If a team can’t let a leader emerge and someone has to be appointed, it’s already fundamentally broken. Good luck with that project.