Foundations of Text Representation

Explain why raw text must be converted to numerical representations for neural network processing.

Compare character-level, word-level, and subword-level tokenization in terms of vocabulary size, handling of rare words, and sequence length.

Describe how Byte Pair Encoding (BPE) iteratively builds a subword vocabulary from a text corpus.

Define a vocabulary and explain its relationship to the tokenization process and the model's output layer.

Explain how token embedding layers map discrete token IDs to dense, continuous vectors.

Derive sinusoidal positional encodings and explain why position information must be explicitly injected in non-recurrent architectures like the Transformer.