Variety and Information Theory

Welcome to the final lesson in our module on variety and regulation. In our previous discussion, we explored Ashby's concept of ultrastability and the Homeostat, uncovering the mechanism by which a system can adapt its very structure to maintain stability in the face of unforeseen challenges. This capacity for adaptation hinges on the system's ability to access a range of internal configurations.

This lesson builds a crucial bridge between that mechanical view of adaptation and the mathematical language of information theory we studied earlier. We will explore how Ashby's concept of variety can be formally related to Claude Shannon's concepts of entropy and information. By doing so, we will gain a more powerful, quantitative framework for understanding the fundamental principles of regulation and control. This synthesis of Ashby's and Shannon's work lies at the very heart of classical cybernetics.

1. From Counting States to Measuring Information

Let's begin by forging the primary link between variety and entropy. As you know, Ashby defined variety as the number of distinguishable states a system can exhibit. For a system with $N$ states, the variety is simply $V = N$ .

However, Ashby and other cyberneticians quickly realized the utility of a logarithmic measure of variety, particularly using base 2, which should be familiar from your computer science background:

V_{l o g} = lo g_{2} N

The logarithmic scale is useful for two main reasons. First, it compresses vast numbers of states into more manageable figures. Second, and more importantly for our purposes, it makes variety an additive quantity. If you have two independent systems with varieties $N_{1}$ and $N_{2}$ , their combined variety is $N_{1} \times N_{2}$ . In logarithms, this becomes $lo g_{2} (N_{1} \times N_{2}) = lo g_{2} (N_{1}) + lo g_{2} (N_{2})$ . This additive property is the key to connecting variety with information.

Now, recall Shannon's entropy, $H$ , which measures the average uncertainty or "surprise" associated with the outcomes of a random variable. It is calculated as:

H (X) = - i = 1 \sum N p_{i} lo g_{2} p_{i}

where $p_{i}$ is the probability of the system being in state $i$ .

What is the relationship between $V_{l o g}$ and $H$ ? The connection becomes clear when we consider the case of maximum uncertainty—that is, when we have no reason to believe any one state is more likely than another. In this situation, every state has an equal probability, $p_{i} = 1/ N$ .

Let's substitute this into the entropy formula:

H_{max} = - i = 1 \sum N \frac{1}{N} lo g_{2} (\frac{1}{N}) = - i = 1 \sum N \frac{1}{N} (- lo g_{2} N) = i = 1 \sum N \frac{1}{N} (lo g_{2} N)

Since $lo g_{2} N$ is a constant, we can pull it out of the sum:

H_{max} = (lo g_{2} N) i = 1 \sum N \frac{1}{N} = (lo g_{2} N) \cdot (N \cdot \frac{1}{N}) = lo g_{2} N

Thus, we arrive at a foundational identity: The logarithmic measure of a system's variety is equal to its maximum possible Shannon entropy.

This means that "variety" is the information-theoretic potential of a system, its capacity for uncertainty before any constraints are applied.

[PDF] Impact of the Cybernetic Law of Requisite Variety on a ... - SciSpace

This paper provides a clear and direct explanation of these foundational concepts. It was written with information science in mind, so its perspective should be quite accessible.

First, read the section on Variety and Constraint. Focus on the definitions of linear and logarithmic measures of variety. Then, jump to the subsection that explicitly connects this to information theory. Read the paragraphs from entropy expresses constraint. Pay close attention to the argument that logarithmic variety is equivalent to maximum entropy (also called information capacity).

This graph illustrates the binary entropy function (red curve) for a system with two states (e.g., a coin flip) with probability $p$ and $1-p$. Note that the entropy is maximized at $p=0.5$, where uncertainty is highest. At this point, $H = \log_2(2) = 1$ bit, which is exactly the logarithmic variety of a two-state system.

2. Regulation as a Battle of Information

Now we can reinterpret Ashby's Law of Requisite Variety in information-theoretic terms. The law states, "only variety can destroy variety." This means a regulator must be able to generate at least as many responses as there are disturbances to control.

Let's model this as a communication problem, as shown in the diagram below.

This diagram shows the basic components of a regulatory problem. The 'Disturbance' (D) perturbs the 'System being regulated' (S), threatening to push it away from the 'Desired Outcome'. The 'Regulator' (R) acts to counteract the disturbance and keep the outcome stable.

Disturbance (D): This is the "source" of messages or events that challenge the system. The variety of disturbances, $V_{D}$ , measured as $lo g_{2} (N_{D})$ , is equivalent to the entropy of the source, $H (D)$ . It quantifies the amount of uncertainty the environment is throwing at the system.
Regulator (R): This is the "channel" or "corrector." Its variety, $V_{R}$ or $H (R)$ , represents its capacity to take different actions. This is analogous to Shannon's channel capacity, $C$ .
Outcome (E): This is the final state of the system after regulation. The goal of regulation is to dramatically reduce the variety of outcomes, ideally to just one: the desired "good" outcome. That is, we want $H (E) \to 0$ .

Ashby formulated this relationship as an inequality of information:

H (E) \geq H (D) - H (R)

This equation states that the uncertainty in the outcome is, at best, the uncertainty from the disturbance minus the uncertainty absorbed or cancelled by the regulator.

If our goal is a single, predictable outcome, we want the variety of outcomes to be 1, which means its logarithmic variety (and thus its entropy, $H (E)$ ) must be 0. For this to be possible, the inequality requires:

0 \geq H (D) - H (R) ⟹ H (R) \geq H (D)

This is the Law of Requisite Variety expressed in the language of information theory. The regulator's capacity to generate information (its variety/entropy) must be greater than or equal to the amount of information generated by the disturbances. A regulator succeeds by being an effective information filter.

3. Constraint as Information

Another way to frame this is through the concept of constraint. As the paper you read mentioned, constraint is fundamental to cybernetics. A constraint is anything that reduces variety. If a six-sided die is constrained so it can never land on '1' or '2', its variety is reduced from 6 to 4.

In information theory, this reduction in uncertainty is measured by redundancy. Redundancy is the difference between the maximum possible entropy and the actual entropy of a system:

Redundancy = H_{max} - H_{actual}

A perfectly random source has zero redundancy. A highly structured or predictable source (like the English language, where 'q' is almost always followed by 'u') has high redundancy.

A regulator's job is to act as a source of constraint. It observes the variety from the disturbance and applies a corresponding action that constrains the outcome to a small, desirable set. In essence, regulation is the act of imposing redundancy on the outcome space. A perfectly regulated system is one where the outcome is 100% redundant—it is completely predictable, regardless of the disturbance. This is why Stafford Beer, whom we will study next, famously defined the purpose of a system as "what it does." What a regulated system does is produce predictable outcomes by actively constraining variety.

4. The Challenge of Circularity in Information Theory

Ashby's goal was to create a general theory for analyzing complex systems. Given your background in CS, you'll appreciate his desire to find an "algebra" for complexity. He and his students tried to use information theory to decompose the total amount of constraint (or transmission of information) in a multi-variable system.

They extended Shannon's mutual information $T (A : B)$ to higher-order interactions using so-called "Q-measures," like $Q (A : B : C)$ , which were intended to quantify the unique, irreducible interaction among three variables. This would allow an analyst to see if a complex system was truly a tangled web or just a collection of simpler, pairwise interactions.

However, they ran into a significant theoretical problem, one that strikes at the core of cybernetics.

[PDF] Ross Ashby's Information Theory - CORE

This paper by Klaus Krippendorff, one of Ashby's former students, details this fascinating chapter in the history of cybernetics. It's a more advanced text, but it reveals a deep insight into the limits of classical information theory.

First, read the section starting from Shannon's entropies. This part introduces the information-theoretic measures Ashby worked with, including the Q-measures for multi-variable interactions. Note the discussion of how Q could become negative, a confusing result. Next, read the following section, Information in circular systems. This is the crucial part. It explains why the Q-measures failed: they are incompatible with systems that contain circular relationships—feedback loops! It also introduces the modern solution: iterative algorithms that can correctly calculate information flows in such systems.

The key takeaway is profound: Shannon's original theory, grounded in a probability framework that assumes linear chains of events (a message is sent, then noise is added, then it's received), breaks down when applied to systems with the very feedback loops that define cybernetics. A message received can, in a cybernetic system, affect the next message sent.

This discovery doesn't invalidate the link between variety and information. Instead, it deepens it. It shows that to fully apply information theory to the complex, circular systems we care about, we need more sophisticated computational tools than the simple additive formulas of early information theory. The analysis of cybernetic systems requires a cybernetic approach to computation itself—one that embraces iteration and feedback.

5. A Philosophical Coda: The Ultimate Limits

Ashby's interest in information theory also led him to profound philosophical questions about the limits of observation and computation, which will likely resonate with your interests. He was fascinated by Bremermann's limit, a theoretical ceiling on the processing capacity of any material system.

[PDF] Ross Ashby's Information Theory - CORE

Let's return to the Krippendorff paper for one last thought-provoking section.

Read the section titled Material and informational numbers. This part contrasts the physical limits of computation with the boundless combinatorial explosion of possibilities that cyberneticians must grapple with.

Ashby's conclusion was that cyberneticians deal with "informational numbers" (the number of possible states, $2^{N}$ ) that are hyper-astronomically larger than the "material numbers" (the number of computations a physical device, even the size of the Earth, could ever perform). This staggering gap underscores why brute-force exploration of a system's possibilities is hopeless. It reinforces the absolute necessity of the principles we've been studying: feedback, constraint, and variety attenuation are not just clever engineering tricks; they are the only possible ways to make sense of and control a world whose complexity fundamentally outstrips our capacity to compute it.

Conclusion

In this lesson, you have connected the concrete, state-counting concept of variety with the powerful, probabilistic framework of information theory. This synthesis provides a rigorous foundation for analyzing regulation and control.

Here are the key takeaways:

Variety as Maximum Entropy: The logarithmic measure of a system's variety ( $lo g_{2} N$ ) is numerically equal to its maximum possible Shannon entropy. It represents the system's total information potential.
Requisite Variety as Channel Capacity: The law of requisite variety can be modeled as a communication channel problem, where the regulator's variety must be at least as great as the disturbance's variety to ensure a predictable outcome.
Regulation as Constraint: From an information-theoretic viewpoint, regulation is the act of imposing constraints on a system's outcomes, thereby creating redundancy and reducing uncertainty.
The Challenge of Circularity: Classical information theory struggles with the feedback loops central to cybernetics, requiring more advanced iterative methods to properly analyze information flows in such systems.

You have now completed our foundational modules on systems thinking, cybernetic principles, feedback, and information. You are equipped with the core theoretical toolkit of a cybernetician. In the next module, we will begin to apply this toolkit to the domain that sparked your interest: the management of complex organizations. We will start by exploring Stafford Beer's pioneering work in management cybernetics, which sets the stage for his masterwork, the Viable System Model.

Recommended resources