Hello! In our last lesson, we explored how feedback loops determine a system's stability, introducing the crucial trade-off between a system's responsiveness and its tendency to remain stable. We saw that structure dictates dynamic behavior. Now, we shift our focus to the very "currency" of control and communication: information. Without the flow of information, feedback loops cannot operate.
This lesson begins our journey into Module 4, which is dedicated to the quantitative side of communication. Our objective is to define information in Shannon's mathematical sense and distinguish it from its everyday usage. This distinction is fundamental not just to cybernetics but to the entire field of computer science and communication engineering. You will learn to separate the colloquial idea of information—as meaning or knowledge—from the precise, powerful, and purely mathematical concept developed by Claude Shannon.
1. Information About What? Separating Meaning from Measure
In everyday language, "information" is inseparable from its meaning. When we say a news report is informative, we mean it tells us something new and significant about the world. This is what philosophers call semantic information: it has content, it is about something, and it can be true or false.
However, this is not what Claude Shannon was concerned with when he founded the mathematical theory of communication. He was an engineer at Bell Labs, facing a very practical problem: how to send messages over noisy telephone wires as efficiently and reliably as possible. For this engineering task, the meaning of the message was irrelevant. It didn't matter if you were transmitting a stock market order, a line of poetry, or complete gibberish. The challenge was to transmit the sequence of symbols accurately.
The following article from the Stanford Encyclopedia of Philosophy introduces this critical distinction. It highlights a tripartite analysis by Warren Weaver, Shannon's collaborator, which neatly categorizes the different problems related to information.
Semantic Conceptions of Information
This article clearly separates the different ways we can think about information and explains why Shannon's theory is not about 'meaning' in the ordinary sense.
First, read the introduction to understand the "polysemantic" nature of information and Weaver's distinction between technical, semantic, and influential problems. Shannon's theory deals exclusively with the first. Then, jump to the section "Conceptual implications of the mathematical theory of communication." Read the first two paragraphs under that heading, starting from "For the mathematical theory" and ending at "producing a lot of information". This part uses powerful examples to show how technical information is divorced from meaning.
As you've just read, from a Shannon-esque perspective, a perfectly random sequence of characters contains a very high amount of information, while a highly predictable message (like a text consisting of the same letter repeated a thousand times) contains very little. This is profoundly counter-intuitive if we are thinking about meaning, but it is the essential first step toward a mathematical definition. Shannon himself stated it unequivocally in his 1948 paper:
"Frequently the messages have meaning... These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages."
This insight is the key: information, in the technical sense, is a measure of the freedom of choice one has when selecting a message. It's about quantifying possibilities, not interpreting content.
2. A Mathematical Definition: Information as Surprisal
If information isn't meaning, then what is it? Shannon's great leap was to define information in terms of probability. He equated information with the reduction of uncertainty. The more "surprising" an event is, the more information its occurrence provides.
Think of it this way:
- If someone tells you the sun will rise tomorrow, you are not surprised. The probability is nearly 1. You have gained virtually zero information.
- If someone tells you that you have just won a multi-million dollar lottery, you are extremely surprised. The probability was incredibly low. You have gained a great deal of information.
The following video from the channel Computerphile provides an excellent and intuitive explanation of this idea of "surprisal". Given your background in computer science, you'll recognize the direct connection it makes between coin flips and the "bit".
Why Information Theory is Important - Computerphile
This video introduces the core idea of quantifying information by measuring how 'surprising' an event is, and formalizes it with the key equation.
Please watch the first part of the video, which covers the concept of surprisal and its formula. Pay close attention to: The initial coin flip analogy to introduce the idea of being "surprised" by an outcome and its connection to 'bits'. The derivation of the formula for surprisal, I = log(1/P(E)), and the justification for using a logarithm to capture this idea.
As the video explains, we can formalize this relationship. The amount of information (or surprisal), , associated with an event that occurs with probability is given by the formula:
Let's break this down:
- Inverse Probability: We use because as the probability gets smaller, the information gets larger. This matches our intuition about surprise.
- Logarithm: We use a logarithm for two main reasons. First, it makes information additive. The information from two independent coin flips (probability ) is the sum of their individual information values (). Second, as the video noted, it handles vast ranges of probabilities gracefully.
- Base 2 and the 'Bit': The choice of base 2 for the logarithm is a convention, but a powerful one. It means the unit of information is the bit. A single bit is the amount of information required to resolve the uncertainty between two equally likely outcomes (like a fair coin toss, where ). Plugging this into the formula gives bit.
This can also be understood as the number of yes/no questions needed to identify an outcome, an idea explored in the next video.
Intro to Information Theory | Digital Communication | Information Technology
This video from Up and Atom provides another clear analogy for quantifying information using the idea of yes/no questions.
Watch the segment from "a really useful way to think". It explains how the 'least amount of yes/or-no questions' you need to ask to determine the state of a system is a measure of its information content.
This "erotetic" (question-based) analysis solidifies the link between resolving uncertainty and the binary digits that form the foundation of all digital computing.
3. A Formal, Abstract Concept
By defining information in terms of probability, Shannon lifted it out of the domain of semantics and into the realm of pure mathematics. Information theory, at its core, is a branch of probability theory. It deals with properties of probability distributions, not with the meaning of the events those distributions describe.
The following reading elaborates on this abstract nature of Shannon's concept, a perspective that should align well with your interest in mathematical formalism and philosophy.
[PDF] What is Shannon information? - PhilSci-Archive
This paper from the philosophy of science archive discusses the purely quantitative nature of Shannon's theory and its modern interpretation as a formal mathematical tool.
First, read the brief opening of Section 3, "A quantitative theory without semantic dimension," for Shannon's classic statement on the irrelevance of meaning. Next, read the first two paragraphs of Section 10, "A formal view of Shannon information." This part explains how modern treatments of the subject present it as an abstract mathematical framework built on random variables and probability distributions, whose applications (like communication) are interpretations of the formalism, not the formalism itself.
This formal view is what makes information theory so powerful and widely applicable. By detaching "information" from any particular meaning or physical carrier, it can be used to analyze everything from data compression (how many bits do you really need to store a file?) to thermodynamics, and, most importantly for us, the flow of communication and control in any cybernetic system.
Conclusion
In this lesson, we have drawn a sharp and crucial distinction between the everyday, semantic meaning of information and the precise, mathematical definition pioneered by Claude Shannon. This is a foundational concept for our ongoing study of cybernetics.
Here are the key takeaways:
- Information has multiple meanings. Shannon's theory deals with the technical problem of transmitting data, not the semantic problem of what that data means.
- Shannon information is quantitative and as-emantic. It is a measure based on the statistical properties of a message source, not its content or meaning.
- Information is defined as a measure of "surprisal" or reduction in uncertainty. The less probable an event, the more information is conveyed by its occurrence.
- The mathematical formula is . The unit of information is the bit, which represents the information gained when choosing between two equally likely outcomes.
So far, we have only defined the information associated with a single event. However, systems like language or a sensor feed produce continuous streams of data. In our next lesson, we will build on today's definition to understand entropy, which measures the average information produced by a source over time. This will allow us to answer the question posed in the video: which contains more information, five coin flips or a five-letter word?