Building and Using a Character-Level Vocabulary with Padding and Masks

Hello! Last time, you traced how a clean sequence becomes partly masked. To put that idea into code, each character and each special symbol needs a stable integer ID. Today you’ll build a small character-level vocabulary and use it to encode and decode text in Colab. This is the representation the masking code will operate on in the next lessons.

Allow about 15–20 minutes for the reading and code.

Decide what counts as a token

The previous examples used tokens such as cat and naps. For our small model, one character will be one ordinary token. That includes spaces and punctuation: cat naps. has nine characters, so it will have nine token IDs.

We also need two symbols that are not characters from the training text:

• [MASK] means “a character was hidden by the corruption process.” The denoiser will later try to predict it.
• [PAD] fills unused positions when sequences of different lengths share a fixed-length batch. It is not a missing character for the denoiser to guess.

The PyTorch Chatbot Tutorial illustrates the underlying vocabulary idea with words rather than characters. Read only its short explanation of why text needs an index mapping; we’ll write a smaller version suited to this course.

In the diagram below, the example tokenizer divides text into larger pieces. Our division will be finer, but the distinction between text pieces and integer IDs is the same. The token embeddings shown above the IDs are model parameters, not something our encoder has to calculate today.

Build the vocabulary and encoder

Run this cell in Colab. We reserve IDs 0 and 1 for [PAD] and [MASK], then sort the characters found in two tiny training examples. Sorting makes the assignments reproducible.

corpus = ["cat naps.", "dog naps."]

special_tokens = ["[PAD]", "[MASK]"]
characters = sorted({char for text in corpus for char in text})
vocab = special_tokens + characters

char_to_id = {token: index for index, token in enumerate(vocab)}
id_to_char = {index: token for index, token in enumerate(vocab)}

PAD_ID = char_to_id["[PAD]"]
MASK_ID = char_to_id["[MASK]"]

def encode(text):
    unknown = [char for char in text if char not in char_to_id]
    if unknown:
        raise ValueError(f"Character not in vocabulary: {unknown[0]!r}")
    return [char_to_id[char] for char in text]

print(vocab)
print(encode("cat naps."))

The encoded sentence should be:

[5, 4, 12, 2, 8, 4, 10, 11, 3]

For example, 5 identifies c, while 2 identifies the space. These numbers are labels, not measurements: 12 does not mean that t is “greater than” c. The only requirement is that the same vocabulary assigns the same ID every time.

This encoder deliberately does no lowercasing or other normalization. "Cat" will raise an error because uppercase C was not in the corpus. That is preferable to silently changing an input; when we use a larger corpus, we can choose its preprocessing rules explicitly.

Decode clean and corrupted sequences

Decoding looks up each ID in the reverse mapping. We’ll normally omit [PAD] from displayed text, but keep [MASK] visible so we can inspect corrupted examples.

def decode(ids, skip_pad=True):
    pieces = []
    for token_id in ids:
        if skip_pad and token_id == PAD_ID:
            continue
        pieces.append(id_to_char[token_id])
    return "".join(pieces)

clean_ids = encode("cat naps.")
print(decode(clean_ids))

padded_ids = clean_ids + [PAD_ID, PAD_ID]
print(decode(padded_ids))

corrupted_ids = clean_ids.copy()
corrupted_ids[0] = MASK_ID   # Hide 'c'
corrupted_ids[4] = MASK_ID   # Hide 'n'
print(decode(corrupted_ids))

The first two prints both show cat naps.. Padding changes the list of IDs, but skipping padding restores the displayed clean text. The last print shows [MASK]at [MASK]aps.. Decoding cannot recover the hidden c and n: those original IDs have been replaced. Recovering them will be the denoiser’s job.

There is one useful boundary to keep in mind: the literal string "[MASK]" is not accepted as input text by this character encoder. [MASK] is a reserved vocabulary item inserted as an ID by our code, not a character-level spelling that the encoder discovers in a sentence.

Takeaway

You now have a reversible mapping between supported clean text and character IDs, plus distinct IDs for masking and padding. Clean text can be encoded and decoded exactly; corrupted text can be displayed, but its hidden characters cannot be reconstructed by decoding alone.

Next, you’ll put several encoded sentences into fixed-length PyTorch batches and mark which positions are real characters rather than padding.