# How the Modular Model Conversion System Generates Modeling Files in Transformers

> Discover how the modular model conversion system in Hugging Face Transformers generates modeling files. Learn about parsing, merging, and dependency resolution for efficient code generation.

- Repository: [Hugging Face/transformers](https://github.com/huggingface/transformers)
- Tags: internals
- Published: 2026-02-22

---

**The modular model conversion system uses libcst to parse `modular_<name>.py` files, merges overrides with base modeling code, resolves dependencies, and outputs traditional single-file components like `modeling_<name>.py` and `configuration_<name>.py`.**

The **modular model conversion system** in the `huggingface/transformers` repository enables contributors to define new models by writing only the differences from existing architectures. Instead of manually editing multiple large files, developers create a single `modular_<name>.py` file containing class overrides and new definitions. The system automatically generates the standard library structure through a sophisticated Python-to-Python transformation pipeline.

## The 7-Step Conversion Pipeline

The conversion is orchestrated by **[`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py)**, which transforms a modular source file into complete, import-ready modeling components. The pipeline executes seven distinct phases to ensure accurate dependency resolution and code generation.

### 1. Load and Import the Modular Module

The converter begins by dynamically importing the modular file specified via command line argument. It locates the module using `importlib.util.find_spec`, then reads the raw source code into memory.

The function **`get_module_source_from_name`** (lines 53–61) handles this discovery phase, enabling the system to process any `modular_<name>.py` file within the `src/transformers/models/` directory structure.

### 2. Parse the AST with libcst

Once loaded, the source string is wrapped in a **`MetadataWrapper`** from the `libcst` library. This wrapper preserves parent-node relationships, scope information, and positional metadata while a **CST visitor** traverses the concrete syntax tree.

This approach, implemented in **`ModelFileMapper.visit_and_merge_dependencies`** (lines 558–562), allows the converter to analyze code structure while maintaining full fidelity to the original formatting and comments.

### 3. Build Dependency Maps

The **`ModuleMapper`** class (abstract base defined in lines 29–55) catalogs every definable object in the modular file. During the visitor pass, it populates:

- **`self.classes`** – Class definitions and inheritance hierarchies
- **`self.functions`** – Function and method definitions  
- **`self.assignments`** – Module-level constants and variables
- **`self.imports`** – External dependencies

Simultaneously, it constructs **immediate** and **recursive** dependency graphs through `self.object_dependency_mapping` and `self.object_recursive_dependency_mapping`, tracking which symbols reference which other symbols.

### 4. Merge Modular Definitions into Target Files

**`ModelFileMapper`** extends `ModuleMapper` to perform the actual code integration. Its **`merge_modular_dependencies`** method (lines 438–447) accepts the original modeling module (e.g., [`modeling_bert.py`](https://github.com/huggingface/transformers/blob/main/modeling_bert.py)) and the parsed modular objects.

The merger **overwrites** existing functions and assignments that are redefined in the modular file, **adds** entirely new classes, and updates the global node map to reflect the combined codebase.

### 5. Resolve Topologically Correct Ordering

After merging, **`compute_relative_order`** (lines 221–285) determines the final sequence of definitions. The algorithm uses a deterministic sort based on:

- Original line numbers from the modular file
- The dependency graph constructed in step 3

This guarantees that classes appear after all dependencies they inherit from or reference, preventing forward-reference errors in the generated output.

### 6. Unravel Super Calls and Rename Symbols

Two specialized transformers modify the merged CST before final output:

- **`ReplaceSuperCallTransformer.leave_Call`** (lines 124–135) replaces `super().method(...)` invocations with the **actual body** of the parent method, inlining inherited logic directly into the subclass.
- **`ReplaceNameTransformer._replace_name`** (lines 34–39) performs case-preserving rewrites throughout the file, converting identifiers like `Bert` to `MyBert` while maintaining capitalization variants (`my_model`, `MyModel`, `MY_MODEL`) in docstrings, comments, and type hints.

### 7. Generate the Final Modeling File

The transformed CST is rendered back to valid Python source. The script prepends the **`AUTO_GENERATED_MESSAGE`** constant (lines 44–50) as a header comment warning against manual edits, then writes the file to the appropriate location:

```python

# 🚨🚨🚨 This file was automatically generated from src/transformers/models/bert/modular_bert.py.

# Do NOT edit this file manually …

```

## Core Architecture Components

| Component | Location | Purpose |
|-----------|----------|---------|
| [`modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/modular_model_converter.py) | `utils/` | Entry point orchestrating the full pipeline |
| `ModuleMapper` | [`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py) (lines 29–55) | Abstract visitor extracting classes, functions, and imports |
| `ModelFileMapper` | [`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py) | Merges modular definitions with existing files and computes ordering |
| `ReplaceSuperCallTransformer` | [`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py) (lines 124–135) | Inlines parent method bodies for `super()` calls |
| `ReplaceNameTransformer` | [`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py) (lines 34–39) | Case-preserving identifier renaming |
| `find_all_dependencies` | [`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py) | BFS-style resolution of class and function dependencies |

## End-to-End Usage Example

Create a modular file defining only the architectural differences from the base model:

```python

# src/transformers/models/bert/modular_bert.py

from .modeling_bert import BertEmbeddings

class MyBertEmbeddings(BertEmbeddings):
    def forward(self, input_ids):
        # Custom embedding logic here

        return super().forward(input_ids)

```

Execute the converter from the repository root:

```bash
python utils/modular_model_converter.py bert

```

The system generates three complete files:

- [`src/transformers/models/bert/modeling_bert.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py) (merged and transformed)
- [`src/transformers/models/bert/configuration_bert.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/configuration_bert.py)  
- [`src/transformers/models/bert/tokenization_bert.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py)

The resulting modeling file contains all original classes plus the overridden `MyBertEmbeddings` with the `super().forward()` call expanded to the actual parent implementation and all "Bert" references converted to "MyBert".

## Summary

- **Modular files** (`modular_<name>.py`) contain only deltas from existing models, reducing boilerplate and maintenance burden.
- **[`utils/modular_model_converter.py`](https://github.com/huggingface/transformers/blob/main/utils/modular_model_converter.py)** parses these files using `libcst` and `MetadataWrapper` to preserve code structure and metadata.
- The **`ModuleMapper`** and **`ModelFileMapper`** classes build dependency graphs and merge modular overrides with base modeling files.
- **Topological sorting** ensures classes appear after their dependencies in the final output.
- **CST transformers** inline `super()` calls and perform case-preserving renaming across docstrings and identifiers.
- Generated files include an **`AUTO_GENERATED_MESSAGE`** header and are written directly to the standard Transformers model directory structure.

## Frequently Asked Questions

### What is the purpose of the modular model conversion system?

The system allows contributors to define new Transformer models by writing only the differences from existing architectures in a single `modular_<name>.py` file. This approach eliminates the need to manually maintain multiple large files (`modeling_*.py`, `configuration_*.py`, etc.) while ensuring the final library code remains compatible with standard Transformers import patterns.

### How does the converter handle inheritance and method overrides?

The converter uses **`ReplaceSuperCallTransformer`** to detect `super().method()` invocations and inline the actual parent method body directly into the subclass definition. This occurs during the CST transformation phase after merging, ensuring the generated code contains concrete implementations rather than dynamic super calls.

### Why does the system use libcst instead of the standard ast module?

The **`libcst`** library provides a **Concrete Syntax Tree** (CST) rather than a standard Abstract Syntax Tree. This preserves exact formatting, comments, and whitespace while enabling robust metadata tracking through **`MetadataWrapper`**. The standard `ast` module would discard comment and formatting information essential for generating clean, readable modeling files.

### Can I manually edit files generated by the modular model converter?

No. Generated files contain an **`AUTO_GENERATED_MESSAGE`** header explicitly warning against manual edits. Any changes should be made to the source `modular_<name>.py` file, after which running `python utils/modular_model_converter.py <name>` will regenerate the modeling files with the updated logic.