Text to Numbers: Why Neural Networks Need Numerical Data

1. The Numerical Nature of Neural Networks

At its heart, any neural network, regardless of its complexity, is a mathematical function. It takes numerical inputs, performs a series of calculations like matrix multiplications and non-linear transformations, and produces numerical outputs.

To solidify this core concept, the following short video from 3Blue1Brown offers a superb visualization.

As the video illustrates, the entire process is numerical. A single neuron or "unit" forms the basic building block. It computes a weighted sum of its inputs and adds a bias term. In mathematical terms, for an input vector $\mathbf{x}$, the unit calculates $z=\mathbf{w}\cdot \mathbf{x}+b$. This value $z$ is then passed through a non-linear activation function, such as a sigmoid or ReLU, to produce the unit's output. Every operation here—dot products, additions, function applications—requires numbers.

For a more formal walkthrough of this process, you can refer to the "Neural Networks" chapter by Jurafsky and Martin, which you may find a useful reference throughout this course. It details how a single unit processes real-valued inputs into an output. This confirms that at the most granular level, a neural network is not equipped to handle symbolic data like text strings. You cannot, for example, compute the dot product of the word "language" and a weight vector.

2. The Representational Gap: From Symbols to Vectors

We've established that neural networks demand numerical input. Human language, however, consists of discrete symbols: characters, which form words, which in turn form sentences. This creates a fundamental representational gap. Our task is to find a principled way to convert these symbols into numbers that a network can process.

The overall process of preparing text for a model involves several stages, which are neatly summarized in the following diagram.

A standard pipeline for converting raw text into a numerical format suitable for a neural network. This lesson focuses on the necessity of the final step: vector encoding.

As shown, raw text is first standardized (e.g., converted to lowercase) and then broken into smaller units called tokens (tokenization). Each unique token is assigned an integer index from a predefined vocabulary. The final, crucial step is to convert these integer indices into a vector format.

3. A First Attempt: One-Hot Encoding

The simplest and most direct way to represent a token numerically is through one-hot encoding. Imagine we have a vocabulary of all possible unique words in our dataset (let's say its size is $V$). A one-hot vector is a vector of length $V$ that is filled with zeros, except for a single '1' at the index that corresponds to the specific word.

For example, if our vocabulary is ["a", "cat", "mat", "on", "sat", "the"] ($V=6$), the word "cat" (at index 1) would be represented as:

And the word "sat" (at index 4) would be:

This method provides a unique and unambiguous numerical representation for each word. The following video segment from a Stanford NLP course provides an excellent overview of this concept and its significant limitations.

4. The Pitfalls of Discrete Representation

As the video points out, while one-hot encoding successfully converts words into numbers, it creates a representation that is deeply flawed for machine learning. The core issue lies in what these vectors fail to capture: any notion of similarity or relationship between words.

Let's break down the key problems, which are also detailed in the following reading.

The reading and video highlight two main drawbacks:

1. Orthogonality and Lack of Similarity: In the vector space created by one-hot encodings, every word vector is orthogonal to every other word vector. The dot product of the vectors for "hotel" and "motel" is zero, just as it is for "hotel" and "car". This mathematical independence means the representation itself gives the model no clue that "hotel" and "motel" are semantically related, while "hotel" and "car" are not. Consequently, anything the model learns about "hotel" from training data provides zero benefit when it later encounters "motel". This severely limits the model's ability to generalize.

2. High Dimensionality and Sparsity: The length of a one-hot vector is equal to the size of the vocabulary. A typical vocabulary for a large language model can easily contain 50,000 to 100,000 tokens or more. This means that every single word is represented by a very high-dimensional vector that is almost entirely composed of zeros. Processing such sparse, high-dimensional data is computationally inefficient and demands enormous amounts of training data for the model to learn meaningful patterns.

These problems demonstrate that simply converting text to any numerical format is not enough. We need a representation that is not only numerical but also meaningful.

Conclusion

In this lesson, we have established the fundamental reason for converting text to numbers: neural networks are mathematical functions that operate exclusively on numerical data. We explored the most basic method for this conversion, one-hot encoding, and uncovered its critical flaws. The resulting vectors are high-dimensional, sparse, and, most importantly, they fail to capture any semantic relationships between words, forcing the model to learn about every word in isolation.

This foundational problem motivates the need for a more sophisticated approach. The ideal representation would be a dense, lower-dimensional vector where similar words have similar vectors. This is the concept of embeddings.

In our next lesson, we will take a step back and look at tokenization: the process of breaking down raw text into the basic units (words, parts of words, or characters) that we will ultimately represent. Following that, we will see how embedding layers solve the problems of one-hot encoding by learning to map these discrete tokens into the dense, meaningful vectors that power modern large language models.