# How Does Tokenization Work for NLP Models: Inside the TinyTorch Implementation

> Discover how tokenization works for NLP models. Explore character-level mapping and BPE subword algorithms with TinyTorch implementation.

- Repository: [Harvard Edge Computing/cs249r_book](https://github.com/harvard-edge/cs249r_book)
- Tags: deep-dive
- Published: 2026-02-19

---

**Tokenization converts raw text into integer IDs that NLP models can process, using either character-level mapping or Byte-Pair Encoding (BPE) subword algorithms as implemented in the Harvard Edge TinyTorch educational library.**

Tokenization serves as the critical first step in any natural language processing pipeline, transforming human-readable text into numerical representations that neural networks can understand. In the `harvard-edge/cs249r_book` repository, the TinyTorch library provides a pedagogical implementation of tokenization strategies used by modern large language models. This article examines how tokenization works for NLP models by analyzing the source code in [`tinytorch/src/10_tokenization/10_tokenization.py`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/10_tokenization/10_tokenization.py), covering both simple character-level approaches and sophisticated subword algorithms.

## The Tokenizer Base Interface

Every tokenizer in the TinyTorch ecosystem inherits from the abstract `Tokenizer` base class defined in `tinytorch/src/10_tokenization/10_tokenization.py#L308-L342`. This design establishes a uniform contract across all tokenization strategies.

The base class defines two essential methods:

- `encode(text: str) -> List[int]`: Converts raw text into a sequence of integer token IDs
- `decode(tokens: List[int]) -> str`: Reconstructs the original text from token IDs

This abstraction allows downstream components—such as embedding layers and transformer blocks—to remain agnostic to the specific tokenization strategy employed. Whether using character-level or BPE tokenization, the interface remains consistent.

## Character-Level Tokenization with CharTokenizer

The `CharTokenizer` class, implemented in `tinytorch/src/10_tokenization/10_tokenization.py#L380-L460`, provides the simplest form of tokenization for NLP models. It maps each Unicode character directly to a unique integer identifier.

### Building the Vocabulary

The tokenizer constructs a bidirectional mapping between characters and IDs using `char_to_id` and `id_to_char` dictionaries. The vocabulary includes a special `<UNK>` token (assigned ID 0) to handle unknown characters that may appear during encoding.

### Encoding and Decoding Process

During **encoding**, the tokenizer iterates through each character in the input text and looks up its corresponding ID. If a character is not present in the vocabulary, it falls back to the `<UNK>` token ID.

During **decoding**, the process reverses: token IDs map back to characters via `id_to_char`, with unknown IDs rendering as `<UNK>`.

This approach guarantees zero out-of-vocabulary errors since every Unicode code point can be represented, though it produces longer token sequences (one token per character).

```python
from tinytorch.core.tokenization import CharTokenizer

vocab = ['h', 'e', 'l', 'o', ' ', 'w', 'r', 'd']
char_tok = CharTokenizer(vocab)

text = "hello world!"
ids = char_tok.encode(text)          # → [1, 2, 3, 3, 4, 5, 0, 6, 4, 7, 3, 5, 0]

print("IDs:", ids)

recovered = char_tok.decode(ids)       # → "hello world"

print("Recovered:", recovered)

```

## Byte-Pair Encoding (BPE) Tokenization

Modern NLP models predominantly use **Byte-Pair Encoding (BPE)**, a subword tokenization algorithm that balances vocabulary size against sequence length. The `BPETokenizer` class in `tinytorch/src/10_tokenization/10_tokenization.py#L466-L740` implements this greedy merge strategy.

### The BPE Training Algorithm

The training process follows these steps:

1. **Initialization**: Each word splits into characters plus an end-of-word marker `</w>` via `_get_word_tokens`.
2. **Pair Counting**: The `_count_byte_pairs` method (lines 530-566) tallies frequency-weighted adjacent character pairs across the corpus.
3. **Merge Selection**: Identify the most frequent pair for merging.
4. **Merge Execution**: The `_merge_pair` method (lines 598-637) replaces every occurrence of the selected pair with a concatenated token across all word token lists.
5. **Iteration**: Repeat steps 2-4 until reaching the target `vocab_size` or exhausting merge candidates.
6. **Mapping Construction**: Build `token_to_id` and `id_to_token` dictionaries via `_build_mappings`.

### Encoding New Text

After training, encoding new sentences involves:

- Splitting text into words and converting to character tokens with `_get_word_tokens`
- Applying the learned merge sequence via `_apply_merges`
- Converting final tokens to integer IDs using `token_to_id`

Decoding reverses this process, concatenating tokens, stripping `</w>` markers, and normalizing whitespace.

```python
from tinytorch.core.tokenization import BPETokenizer

corpus = ["hello", "world", "hello", "hell"]
bpe = BPETokenizer(vocab_size=20)
bpe.train(corpus)

# Encode a new sentence

sentence = "hello world"
token_ids = bpe.encode(sentence)
print("BPE token IDs:", token_ids)

# Decode back to text

print("Decoded:", bpe.decode(token_ids))

```

## Integrating Tokenization into the NLP Pipeline

In the TinyTorch architecture, the tokenization module sits at the head of the data flow:

```

Raw Text ──► Tokenizer (CharTokenizer / BPETokenizer) ──► Token IDs ──► Embedding Layer

```

The embedding layer expects integer token IDs and produces dense vector representations for each token. Downstream transformer and attention mechanisms operate on these embedded sequences.

Because all tokenizers implement the standard `encode`/`decode` interface, switching between character-level and BPE strategies requires only a single line change. This modularity enables rapid experimentation with different tokenization approaches without modifying model architecture.

## Summary

- **Tokenization** transforms text into integer IDs, serving as the essential bridge between human language and neural network computation.
- The `Tokenizer` base class in [`tinytorch/src/10_tokenization/10_tokenization.py`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/10_tokenization/10_tokenization.py) defines a consistent `encode`/`decode` interface used by all implementations.
- **CharTokenizer** provides character-level tokenization that never produces out-of-vocabulary tokens but creates longer sequences.
- **BPETokenizer** implements the Byte-Pair Encoding algorithm through methods like `_count_byte_pairs` and `_merge_pair`, learning subword units that balance vocabulary efficiency and expressiveness.
- The modular design allows seamless integration with embedding layers and facilitates educational exploration of tokenization internals.

## Frequently Asked Questions

### What is the difference between character-level and BPE tokenization?

Character-level tokenization assigns a unique ID to every individual character, resulting in small vocabularies but long sequences. BPE tokenization learns to merge frequently co-occurring character pairs into subword units, creating compact vocabularies that handle rare words through decomposition while keeping sequences shorter.

### How does BPE handle out-of-vocabulary words?

BPE handles unknown words by breaking them into known subword units learned during training. For example, if "tokenization" was not in the training data but the model learned "token" and "ization" as separate tokens, it can represent the new word as a combination of existing subwords, eliminating true out-of-vocabulary errors.

### Why is tokenization necessary for NLP models?

Neural networks operate on numerical tensors, not raw text. Tokenization converts variable-length strings into fixed-vocabulary integer sequences that can be embedded into dense vectors and processed by mathematical operations in transformer or recurrent architectures.

### How do you choose the vocabulary size for BPE?

The vocabulary size represents a trade-off between sequence length and model capacity. Smaller vocabularies require longer token sequences but have fewer embedding parameters, while larger vocabularies shorten sequences but increase memory requirements. The `BPETokenizer` accepts `vocab_size` as a constructor parameter, typically set between 10,000 and 50,000 tokens for general-purpose models.