How Does Tokenization Work for NLP Models: Inside the TinyTorch Implementation

Tokenization converts raw text into integer IDs that NLP models can process, using either character-level mapping or Byte-Pair Encoding (BPE) subword algorithms as implemented in the Harvard Edge TinyTorch educational library.

Tokenization serves as the critical first step in any natural language processing pipeline, transforming human-readable text into numerical representations that neural networks can understand. In the harvard-edge/cs249r_book repository, the TinyTorch library provides a pedagogical implementation of tokenization strategies used by modern large language models. This article examines how tokenization works for NLP models by analyzing the source code in tinytorch/src/10_tokenization/10_tokenization.py, covering both simple character-level approaches and sophisticated subword algorithms.

The Tokenizer Base Interface

Every tokenizer in the TinyTorch ecosystem inherits from the abstract Tokenizer base class defined in tinytorch/src/10_tokenization/10_tokenization.py#L308-L342. This design establishes a uniform contract across all tokenization strategies.

The base class defines two essential methods:

• encode(text: str) -> List[int]: Converts raw text into a sequence of integer token IDs
• decode(tokens: List[int]) -> str: Reconstructs the original text from token IDs

This abstraction allows downstream components—such as embedding layers and transformer blocks—to remain agnostic to the specific tokenization strategy employed. Whether using character-level or BPE tokenization, the interface remains consistent.

Character-Level Tokenization with CharTokenizer

The CharTokenizer class, implemented in tinytorch/src/10_tokenization/10_tokenization.py#L380-L460, provides the simplest form of tokenization for NLP models. It maps each Unicode character directly to a unique integer identifier.

Building the Vocabulary

The tokenizer constructs a bidirectional mapping between characters and IDs using char_to_id and id_to_char dictionaries. The vocabulary includes a special <UNK> token (assigned ID 0) to handle unknown characters that may appear during encoding.

Encoding and Decoding Process

During encoding, the tokenizer iterates through each character in the input text and looks up its corresponding ID. If a character is not present in the vocabulary, it falls back to the <UNK> token ID.

During decoding, the process reverses: token IDs map back to characters via id_to_char, with unknown IDs rendering as <UNK>.

This approach guarantees zero out-of-vocabulary errors since every Unicode code point can be represented, though it produces longer token sequences (one token per character).

from tinytorch.core.tokenization import CharTokenizer

vocab = ['h', 'e', 'l', 'o', ' ', 'w', 'r', 'd']
char_tok = CharTokenizer(vocab)

text = "hello world!"
ids = char_tok.encode(text)          # → [1, 2, 3, 3, 4, 5, 0, 6, 4, 7, 3, 5, 0]

print("IDs:", ids)

recovered = char_tok.decode(ids)       # → "hello world"

print("Recovered:", recovered)

Byte-Pair Encoding (BPE) Tokenization

Modern NLP models predominantly use Byte-Pair Encoding (BPE), a subword tokenization algorithm that balances vocabulary size against sequence length. The BPETokenizer class in tinytorch/src/10_tokenization/10_tokenization.py#L466-L740 implements this greedy merge strategy.

The BPE Training Algorithm

The training process follows these steps:

1. Initialization: Each word splits into characters plus an end-of-word marker </w> via _get_word_tokens.
2. Pair Counting: The _count_byte_pairs method (lines 530-566) tallies frequency-weighted adjacent character pairs across the corpus.
3. Merge Selection: Identify the most frequent pair for merging.
4. Merge Execution: The _merge_pair method (lines 598-637) replaces every occurrence of the selected pair with a concatenated token across all word token lists.
5. Iteration: Repeat steps 2-4 until reaching the target vocab_size or exhausting merge candidates.
6. Mapping Construction: Build token_to_id and id_to_token dictionaries via _build_mappings.

Encoding New Text

After training, encoding new sentences involves:

• Splitting text into words and converting to character tokens with _get_word_tokens
• Applying the learned merge sequence via _apply_merges
• Converting final tokens to integer IDs using token_to_id

Decoding reverses this process, concatenating tokens, stripping </w> markers, and normalizing whitespace.

from tinytorch.core.tokenization import BPETokenizer

corpus = ["hello", "world", "hello", "hell"]
bpe = BPETokenizer(vocab_size=20)
bpe.train(corpus)

# Encode a new sentence

sentence = "hello world"
token_ids = bpe.encode(sentence)
print("BPE token IDs:", token_ids)

# Decode back to text

print("Decoded:", bpe.decode(token_ids))

Integrating Tokenization into the NLP Pipeline

In the TinyTorch architecture, the tokenization module sits at the head of the data flow:

Raw Text ──► Tokenizer (CharTokenizer / BPETokenizer) ──► Token IDs ──► Embedding Layer

The embedding layer expects integer token IDs and produces dense vector representations for each token. Downstream transformer and attention mechanisms operate on these embedded sequences.

Because all tokenizers implement the standard encode/decode interface, switching between character-level and BPE strategies requires only a single line change. This modularity enables rapid experimentation with different tokenization approaches without modifying model architecture.

Summary

• Tokenization transforms text into integer IDs, serving as the essential bridge between human language and neural network computation.
• The Tokenizer base class in tinytorch/src/10_tokenization/10_tokenization.py defines a consistent encode/decode interface used by all implementations.
• CharTokenizer provides character-level tokenization that never produces out-of-vocabulary tokens but creates longer sequences.
• BPETokenizer implements the Byte-Pair Encoding algorithm through methods like _count_byte_pairs and _merge_pair, learning subword units that balance vocabulary efficiency and expressiveness.
• The modular design allows seamless integration with embedding layers and facilitates educational exploration of tokenization internals.

Frequently Asked Questions

What is the difference between character-level and BPE tokenization?

Character-level tokenization assigns a unique ID to every individual character, resulting in small vocabularies but long sequences. BPE tokenization learns to merge frequently co-occurring character pairs into subword units, creating compact vocabularies that handle rare words through decomposition while keeping sequences shorter.

How does BPE handle out-of-vocabulary words?

BPE handles unknown words by breaking them into known subword units learned during training. For example, if "tokenization" was not in the training data but the model learned "token" and "ization" as separate tokens, it can represent the new word as a combination of existing subwords, eliminating true out-of-vocabulary errors.

Why is tokenization necessary for NLP models?

Neural networks operate on numerical tensors, not raw text. Tokenization converts variable-length strings into fixed-vocabulary integer sequences that can be embedded into dense vectors and processed by mathematical operations in transformer or recurrent architectures.

How do you choose the vocabulary size for BPE?

The vocabulary size represents a trade-off between sequence length and model capacity. Smaller vocabularies require longer token sequences but have fewer embedding parameters, while larger vocabularies shorten sequences but increase memory requirements. The BPETokenizer accepts vocab_size as a constructor parameter, typically set between 10,000 and 50,000 tokens for general-purpose models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}