[Off-Topic] Agility, Chaos, Self-Organization
This article pulls together several topics that make up part of the knowledge you need to understand Self-Organization. And here it pays to look at the concept from its physical and mathematical foundation too.
The motivation comes from the 11th principle of the Agile Manifesto:
“The best architectures, requirements, and designs emerge from self-organizing teams.” – Principles behind the Agile Manifesto
Agility requires self-organization. But this concept is alien to most people and organizations. The 12th principle further adds:
“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” – Principles behind the Agile Manifesto
For this you need to understand the concept of a Learning Organization. Or better: understand the dynamic moment of learning and continuous improvement called the Edge of Chaos. Implementing Agility, in physics terms, means the following:
Removing the organization, which is a complex system, from its state of dynamic equilibrium, increasing the overall entropy of the system, forcing it toward chaos. At the ideal point, at the edge of chaos, self-organization happens and the organization becomes a learning organization.
Remember: “learning” also includes exploring the unknown, incorporating new information. Doing what you’ve always done means you’re not learning anything, by definition. You need to do things differently, make mistakes, and improve in order to learn.
Let’s get into the concepts now.
Translation: Chaos Theory in Organizational Development
from Wikipedia
Self-organization, as opposed to natural or social selection, is a dynamic change within the organization where system changes are made by recalculating, re-inventing and modifying its structure in order to adapt, survive, grow, and develop.
Self-organization is the result of re-invention and creative adaptation due to the introduction of, or being in a constant state of, perturbed equilibrium. One example of an organization which exists in a constant state of perturbation is that of the learning organization, which is “one that allows self-organization, rather than attempting to control the bifurcation through planned change.” (Dooley, 1995)
Being “off-balance” lends itself to regrouping and re-evaluating the system’s present state in order to make needed adjustments and regain control and equilibrium. By understanding and introducing the element of punctuated equilibrium (chaos) while facilitating networks for growth, an organization can change gears from “cruise” to “turbo” in regard to speed and intensity of organizational change. While maintaining an equilibrial state seems to be an intuitively rational method for enabling an organization to gain a sense of consistency and solidarity, existing on the edge of a chaotic state remains the most beneficial environment for systems to flourish, develop and grow.
For instance, two competing organizations that differ in regard to their levels of homeostasis will not be in competition for long. Generally speaking, the organization with the less-stable structure will come out ahead while the constant stability of the latter will eventually lead to its own demise. Although quite similar, small differences in homeostasis levels are enough to make a tremendous difference in future outcomes for each organization. The notion of similarity in origin vs. dissimilar results comes to fruition with the emergence of bifurcation.
The concept of bifurcation cannot be explained without discussion of the term frequently labeled “sensitivity to initial conditions.” Sensitivity to initial conditions refers to the high level of importance of primary conditions from which the future path and direction of a system stems. This sensitivity to initial conditions is commonly referred to as the Butterfly Effect, in which a butterfly flaps its tiny wings in one end of the world which results in a typhoon or hurricane somewhere else on the globe. While this is an entertaining notion, sensitivity to initial conditions remains in reality a very abstract concept without the presence of bifurcation, which is mathematically labeled as the actual splitting point of two near-identical entities which, due to the sensitivity of initial conditions, tend to take two very distinct paths and result in two totally different geographically or even evolutionary places.
The primary goal of an organizational development (OD) consultant is to initiate, facilitate, and support successful change in an organization. Using chaos theory as the sole model for change may be far too risky for any stakeholder buy-in. The concept of uncertainty on which chaos theory relies is not an appealing motive for change compared to many alternative “safer” models of organizational change which entail less risk. By careful planning and management of disorder a successful intervention is possible, but only with a truly dedicated arsenal of talented and creative resources. By permitting or actively forcing an organization to enter a chaotic state, change becomes inevitable and bifurcation imminent; but the question remains, “Will the new direction be the one intended?” In order to account for the direction of the new thrust, most planning attention should be focused on attractors instead of the initiation of disorder.
Although chaos eventually gives way to self-organization, how can we control the duration, intensity, and shape of its outcome? It seems that punctuating equilibrium and instilling disorder in an organization is risky business. Throwing an organization off balance could possibly send it in a downward spiral towards dissemination by ultimately compromising the structural integrity (i.e. identity) of the system to the point of no return. The only way to reap the benefits of chaos theory in OD while maintaining a sense of security is to adjust the organization towards a state of existence which lies on the edge of chaos.
By existing on the edge of chaos, organizations are forced to find new, creative ways to compete and stay ahead. Good examples of such learning organizations are found throughout the field of technology as well as the airline industry, namely organizations such as Southwest Airlines, which used re-invention not just for survival, but also to prosper in an otherwise dismal market. In contrast, there are organizations which, due to extended periods of equilibrium, find themselves struggling for survival. Telephone companies, for instance, were once solid and static entities that dominated the communication market. While the rest of the world was developing new communication technology, telephone companies did not creatively grow at the same rate. The result is an organization that is battling to stay alive unless they embrace the element of chaos due to crisis, and allow creative adaptability to function freely so that self-organization and re-invention can occur.
While organizations existing on the edge of chaos are known to be the most creative and adaptive of organizations, how do their members feel about constant evolution and re-invention? Is it possible to identify with, and stay loyal to, an organization that constantly changes shape? The short answer is yes. As long as the organization does not change its core essence, its identifiable, shared purpose, its members will still experience the organization as a developing system that changes shape but retains the same familiar face.
Perhaps the safest way to use chaos theory in OD is not in the instigation of organizational change, but in the use of its principles in dealing with issues that arise within the organization. By embracing organizational phenomena previously seen as dysfunctional, such as interpersonal conflict, and using it as a source for transformational change by applying principles found in chaos theory (Shelton, 2003), an organization can make “lemonade out of lemons” and become more responsive to change agents while continuously moving ahead and growing from the inside out without the fear of complete chaos.
How Business Is a Lot Like Life: The 4 Principles
Fast Company Magazine, March 2001. How Business is a Lot Like Life
Equilibrium is a precursor to death. When a living system is in a state of equilibrium, it is less responsive to the changes that are occurring around it. It is most at risk when it feels most secure.
When threatened or when galvanized by a compelling opportunity, living things move toward the edge of chaos. This condition evokes higher levels of mutation and experimentation and is more likely to result in fresh new solutions.
As living things move closer to the edge of chaos, they have a tendency to self-organize, and new forms emerge from the turmoil. This property of life, called “self-organization and emergence,” is a major source of innovation, creativity, and evolution.
Living systems cannot be directed along a linear path. Unforeseen consequences are inevitable. The challenge is to learn how to disturb them in a manner that approximates the desired outcome and then to correct the course as the outcome unfolds.
Definitions
from Wikipedia
Reductionism can either mean (a) an approach to understanding the nature of complex things by reducing them to the interactions of their parts, or to simpler or more fundamental things or (b) a philosophical position that a complex system is nothing but the sum of its parts.
In physics and systems theory, the superposition principle, also known as superposition property, states that, for all linear systems, the net response at a given place and time caused by two or more stimuli is the sum of the responses which would have been caused by each stimulus individually. So that if input A produces response X and input B produces response Y, then input (A + B) produces response (X + Y).
In mathematics, a nonlinear system is a system (…) that does not satisfy the superposition principle, or whose output is not proportional to its input. Less technically, a nonlinear system is any problem where the variable(s) to be solved for cannot be written as a linear combination of independent components. (…) Nonlinear problems are of interest to physicists and mathematicians because most physical systems are inherently nonlinear in nature. Nonlinear equations are difficult to solve and give rise to interesting phenomena such as chaos. The weather is famously nonlinear, where simple changes in one part of the system produce complex effects throughout.
Emergent Behavior: Thriving on the Edge of Chaos
by Chris Rollins, January 2009.
The cathedral termite, found in parts of Australia, is capable of creating mounds for the colony well over 10 feet high. Individual cathedral termites are just standard-looking bugs: head, thorax, abdomen, legs, and so on, with a tiny little primitive brain. But when combined with others of its species, the cathedral termite is capable of constructing a huge, complex hive to house the colony. Unlike human building projects, however, there is no foreman, no plan, and it’s unlikely that any termite even knows what it is helping to create.
How is this possible?
The answer lies in the fact that sometimes, a system can provide more complexity than the sum of its parts, leading to what scientists call “emergent behavior.”
Emergent behavior, or the spontaneous creation of order, is present all around us. Insects are a good example because they are familiar to us and manage to undertake massive building projects which we can appreciate. Other parts of the animal kingdom also display autonomous order: fish organize themselves into schools that move in concert; birds and pack animals flock or herd in a similar manner. And nonliving examples also abound: natural magnets align themselves into a common North-South orientation and crystals can form from liquids, showing a spontaneous increase in order despite the lack of a more “intelligent” force.
There’s a major thermodynamic problem with all of this, of course: entropy, a measure of disorder, is supposed to continually increase. The universe tends toward chaos.
How, then, can spontaneous order arise, especially in a purely physical system?
Entropy still must increase, even when crystals form or birds flock, but the important distinction is in where the entropy increases or decreases. It turns out that nature will allow entropy to decrease in certain areas provided that it increases elsewhere to compensate. For instance, when a sugar solution begins forming crystals those crystals have a net lower energy level than the free-floating molecules of sugar in the solution. When the sugar enters the structure, it loses energy that is transferred to the water in the form of heat: the most disorganized form of energy.
Therefore, the crystalline portion of the solution has now decreased in entropy while the total system, including the water and the crystal, has had a net increase in entropy.
Why Gaussian Statistics Are Almost All Wrong for Organizational Strategies
Power Law phenomena exhibit Pareto distribution rather than Gaussian (normal). The fundamental difference lies in the premise about the correlation of events. In Gaussian distribution, events are assumed to be independent. Independent events generate normal distributions, which are at the heart of modern statistics. When events are interdependent, normality in distributions is not the norm. Instead, Pareto distribution dominates because extreme events occur more frequently than the normal distribution, with its Gaussian bell curve, would lead us to expect. Physical, biological, ecological, social, and industrial systems exhibit a remarkable variety of fractal structure (Kaye, 1993). Many scholars now believe that power laws are the best analytical framework for describing the origin and form of most natural objects. Given the ubiquity of these findings and the nature of scale-free theory, we think they are equally ubiquitous phenomena in organizations, though unknown and underappreciated (Andriani, 2003).
Extremes vs. Averages
Linear thinking is encrusted in our mentality. Scientific and mathematical models are based on the concepts of equilibrium and linearity. Linearity means two things: (1) proportionality between cause and effect, and (2) that the dynamic of a system can be reconstructed by summing up the effects of single causes acting on single components (Nicolis and Prigogine, 1989), which allows efficient causality to operate, equations to be solved, and predictive modeling. Economics, for instance, is almost theistic in its (scarcely verified) assumption that economic phenomena trend toward “general equilibrium” (Mirowski, 1989). However, this assumption allows linear equations and analytical simplicity.
By focusing on systems in equilibrium, statisticians implicitly accept that the number of possible states a system can reach is limited (and computable) and that the search time following the onset of instability (i.e., an exogenous shock) is short compared to the equilibrium time. For this to be true, the many elements making up a system need to be independent data points by assumption; otherwise we could have interdependence, possible mutual causalities, and the occurrence of possible extreme events.
If we take 100 companies approximately of the same size belonging to the same sector and assume independence, and plot a variable, say profit, we expect most events to pack around the mean, exhibiting the classic bell curve. The bell shaped distribution is by far the most studied statistical distribution; it is assumed to correctly characterize much of our discoveries about the natural and social worlds. In real life, however, the crux of the point is whether all events are independent. In real life, for example, these companies could: benchmark against each other, imitate those perceived as successful, exchange information, organize cartels, pursue mergers and acquisitions, compete for limited resources, etc. In a word, they are most likely interdependent, not independent. The statistical distribution that governs interconnected agents does not yield a bell curve, but a power law: the Pareto distribution.
Gaussian and Paretian distributions differ radically. The Gaussian distribution is reliably characterized by its stable mean and finite variance (Greene, 2002). A Paretian distribution doesn’t show a well-behaved mean and variance. A power law, therefore, has no “average” that can be assumed to represent the typical features of the distribution and no finite variance upon which to base confidence intervals (Moss, 2002). There are two major implications.
The dream of social science, of building robust frameworks that allow prediction, is shattered by the absence of statistical regularities in phenomena dominated by persistent interconnectivity. Absent stable mean and finite variance, the probabilistic assessment of individual outcomes becomes much more difficult. This point reflects the more pervasive and structural issue of nonlinearity and emergence in complex systems. Linearity assumes the divisibility of systems into modules whose dynamics can be studied independently of context.
Paretian tails decay more slowly than those of normal distributions. These fat tails affect systems’ behaviors in significant ways. For instance, Buchanan (2004) reports that a 10% drop in the financial market in a single day should happen once every 500 years according to the normal distribution. Mandelbrot (Mandelbrot and Hudson, 2004) shows that, instead, such crashes happen roughly once every five years. Extreme events, that in a Gaussian world could be safely ignored, are not only more common than expected but also of vastly larger magnitude and consequence.