Encoding and Decoding Text with ASCII and UTF-8

Hello! In the previous lesson, you interpreted the same bytes as different integers depending on byte order. Now the question changes: when bytes represent text, how do you know which characters they mean?

You’ll use ASCII to read simple text directly from hex bytes, then use UTF-8 to encode and decode characters that ASCII cannot represent. This is useful whenever a program reads a file, receives a request, or sends a response: the sender and receiver must agree on a text encoding, not just on where the bytes begin and end.

ASCII: a shared mapping from numbers to characters

A computer stores bytes, not letters. A text encoding supplies rules for translating between characters and bytes. ASCII assigns codes from 0 through 127 to English letters, digits, punctuation, and control characters. It is a 7-bit code, although an ASCII character is commonly stored in an 8-bit byte with its highest bit set to zero.

Watch these portions of How Computers Store Text by NoBS Code. Focus on the distinction between assigning a number to a character and storing that number as bytes.

The Linux ascii(7) manual is a useful lookup table rather than something to memorize. Read its opening explanation and main character table, using the Hex column for the examples below.

For example, the ASCII bytes 4F 4B 0A decode to O, K, and a newline. Conversely, encoding the text OK produces 4F 4B. Each pair of hex digits describes one byte; the spaces between pairs are just a way of writing the sequence clearly.

ASCII’s limit is equally important: it has no code for é. Seeing a byte such as E9 does not make it an ASCII character. To represent a wider range of text reliably, we need both a larger set of assigned numbers and rules for placing them into bytes.

Unicode numbers are not UTF-8 bytes

Unicode assigns a numerical code point to a text element. We write a code point as U+ followed by hexadecimal digits. For example, A is U+0041, while é is U+00E9.

A code point answers which value has been assigned? An encoding such as UTF-8 answers which bytes store that value? The notation U+00E9 is therefore not itself a claim that the text contains a byte E9.

UTF-8 has a particularly useful relationship with ASCII: every ASCII code uses exactly the same single byte in UTF-8. Thus A is byte 41 under either encoding. UTF-8 represents code points outside ASCII using sequences of two, three, or four bytes.

This also clarifies a distinction from the previous lesson. Endianness tells you how to assign numerical weights to the bytes of a multi-byte integer. UTF-8 instead defines an ordered sequence of bytes with particular roles. To decode é, you do not reverse its UTF-8 bytes according to your computer’s integer byte order.

How UTF-8 builds a byte sequence

A UTF-8 byte carries some prefix bits that identify its role. The remaining bits carry pieces of the code point:

Here, each x is a bit taken from the code point. The first byte tells you the sequence length; every additional byte begins with 10 and is called a continuation byte. UTF-8 uses the shortest valid sequence for a code point. The table describes byte shapes and ranges; some values within those ranges are reserved rather than valid Unicode text.

Read the Encoding subsection of the Linux utf-8(7) manual to see the underlying templates. That manual also displays historical five- and six-byte templates: skip those. Modern Unicode UTF-8 uses at most four bytes, with U+10FFFF as its upper limit.

Let’s encode é. Its code point, U+00E9, is 11101001 in binary. It is above U+007F but fits in the two-byte UTF-8 pattern. Fill that pattern’s eleven x positions from right to left with the code-point bits, adding leading zeros as needed:

Code point:               00011101001
Split into payloads:      00011  101001
UTF-8 pattern:            110xxxxx 10xxxxxx
Filled bytes:             11000011 10101001
Hex bytes:                C3       A9

The image below shows that same encoding. Its differently colored bits distinguish the fixed prefixes, the code-point bits, and the leading zeros needed to fill the available payload positions.

To decode C3 A9, recognize 110 in the first byte and 10 in the second. Remove those prefixes and join the remaining bits, 00011 101001. That gives 00011101001, or U+00E9: é. Notice that A9 by itself is not a complete UTF-8 character; its 10 prefix marks it as a continuation byte.

The same method works for a three-byte character. The UTF-8 bytes for € are E2 82 AC:

Bytes in binary:          11100010 10000010 10101100
Without UTF-8 prefixes:       0010   000010   101100
Joined code-point bits:   0010000010101100
Code point:               U+20AC (€)

A four-byte sequence follows the same rule with one leading byte and three continuation bytes. For instance, F0 9F 98 80 represents U+1F600, 😀. The number of bytes is not generally the number of characters.

Reading text as a byte stream

Consider a short message containing café followed by a newline:

Characters:       c    a    f    é       newline
UTF-8 bytes:     63   61   66   C3 A9       0A

ASCII-compatible characters retain their familiar single-byte values. é takes two bytes. The whole message contains five code points but six bytes. That difference matters when a backend program declares a message’s byte length or receives only part of a text stream. If a read ends immediately after C3, the UTF-8 sequence is incomplete until A9 arrives.

The bytes alone also need an agreed encoding. For example, a lone E9 is not valid UTF-8 for é; the UTF-8 representation is C3 A9. When interpreting an input as UTF-8, a decoder should not quietly pretend that any arbitrary byte sequence is valid text.

Python makes the boundary between text and bytes visible:

text = "café\n"
raw = text.encode("utf-8")

print(raw.hex(" "))            # 63 61 66 c3 a9 0a
print(len(text))               # 5 code points in this example
print(len(raw))                # 6 bytes
print(raw.decode("utf-8"))     # café, followed by a newline

print("A".encode("ascii").hex(" "))  # 41
print("A".encode("utf-8").hex(" "))  # 41

Calling .encode("utf-8") produces bytes from text; calling .decode("utf-8") interprets bytes as text. Trying "é".encode("ascii") fails because ASCII has no value for it. Similarly, bytes.fromhex("c3").decode("utf-8") fails because the leading byte promises a continuation byte that is missing. These errors are useful: they expose a broken or incomplete encoding assumption rather than hiding it.

Key takeaways

ASCII maps 128 codes to characters and controls; its codes occupy one byte each in UTF-8. Unicode assigns code points, while UTF-8 determines their byte representation. To decode UTF-8, read the leading byte to find the sequence length, check the 10 continuation bytes, then join the payload bits. The UTF-8 byte sequence has its own defined order—your machine’s integer endianness does not reverse it.

Next, you’ll turn from text to floating-point numbers and see why representing a value with a finite number of bits can introduce rounding.