# How the modeld Process Runs Deep Learning Inference with TinyGrad for Autonomous Driving

> Discover how openpilot's modeld process executes deep learning inference using TinyGrad on Qualcomm or AMD hardware to generate real-time driving commands for autonomous vehicles.

- Repository: [comma.ai/openpilot](https://github.com/commaai/openpilot)
- Tags: internals
- Published: 2026-03-05

---

**The modeld process in openpilot performs real-time driving inference by loading two pre-compiled TinyGrad models—a vision encoder and a policy network—and executing them sequentially on Qualcomm or AMD hardware to generate steering and acceleration commands.**

The **modeld** process serves as the core inference engine within the commaai/openpilot repository, responsible for transforming raw camera frames into actionable driving controls. This article examines how modeld leverages TinyGrad, a lightweight Python autodiff framework, to run both perception and planning neural networks on embedded automotive hardware.

## Overview of the modeld Architecture

The modeld process runs two distinct **TinyGrad**-based neural networks in series to achieve end-to-end driving inference:

- **Vision model** (`driving_vision_tinygrad.pkl`): Encodes NV12 camera buffers into a hidden state while simultaneously detecting lane lines, lead vehicles, and other critical perception features
- **Policy model** (`driving_policy_tinygrad.pkl`): Consumes the vision model's hidden state alongside high-level inputs (desired driving behavior, traffic convention) to output longitudinal and lateral control plans

Before loading models, modeld forces the TinyGrad runtime to select the correct compute backend. In [`selfdrive/modeld/modeld.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/modeld.py) lines 1-9, the code explicitly sets the device to `QCOM` for TICI boards or `AMD` when USB-GPU acceleration is present, ensuring the compiled graphs target the available silicon.

## Loading Compiled TinyGrad Models

Rather than constructing computation graphs at runtime, modeld deserializes pre-compiled TinyGrad artifacts from the `models/` directory. During `ModelState.__init__` in [`selfdrive/modeld/modeld.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/modeld.py) (lines 81-84), the process loads pickled executable graphs using Python's pickle module:

```python
self.vision_run = pickle.loads(read_file_chunked(str(VISION_PKL_PATH)))
self.policy_run = pickle.loads(read_file_chunked(str(POLICY_PKL_PATH)))

```

These loaded objects become callable Python functions (`vision_run` and `policy_run`) that execute the complete forward passes of their respective networks. This approach eliminates compilation overhead during the critical real-time loop, allowing modeld to begin inference immediately after initialization.

## Preprocessing Camera Frames with TinyGrad JIT

Raw camera frames arrive as **NV12** buffers requiring undistortion, resizing, and plane rearrangement before inference. Modeld handles this preprocessing through a JIT-compiled TinyGrad function rather than traditional CPU-based image processing.

The warp implementation resides in [`selfdrive/modeld/compile_warp.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/compile_warp.py), where the `warp_perspective_tinygrad` function constructs a perspective transformation using TinyGrad operations. The script JIT-compiles this transformation using `TinyJit(update_both_imgs, prune=True)` and caches the result as `warp_{w}x{h}_tinygrad.pkl` (lines 28-36 and 70-84).

At runtime, modeld loads this cached JIT function during the first frame processing. In [`selfdrive/modeld/modeld.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/modeld.py) (lines 133-138), the `update_imgs` callable transforms both low-resolution and high-resolution camera streams into the 6-channel tensor format expected by the vision model:

```python
out = self.update_imgs(self.img_queues['img'], self.full_frames['img'],
                       self.transforms['img'],
                       self.img_queues['big_img'], self.full_frames['big_img'],
                       self.transforms['big_img'])

```

This JIT fusion eliminates Python-level loops and achieves real-time performance on low-power embedded CPUs and GPUs.

## Running the Vision Model

After preprocessing, modeld feeds the warped tensors to the vision network. The vision model accepts two inputs: a low-resolution `img` tensor and a high-resolution `big_img` tensor. In [`selfdrive/modeld/modeld.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/modeld.py) (line 221), modeld executes the vision graph and extracts the raw output buffer:

```python
self.vision_output = self.vision_run(**vision_inputs).contiguous().realize().uop.base.buffer.numpy().flatten()

```

The `.contiguous().realize()` chain forces TinyGrad to execute the computation graph and materialize results in memory. The `Parser` class then converts this flattened NumPy array into a structured dictionary containing the hidden state (`features_buffer`) and perception outputs like lane line coefficients and lead vehicle positions.

## Running the Policy Model

The policy network combines temporal information with the current vision features to generate driving commands. Modeld maintains circular buffers (`InputQueues`) to accumulate historical states, desire pulses, and traffic convention flags.

Before each policy inference cycle, modeld copies the newest temporal slice into NumPy arrays that back the TinyGrad inputs. In [`selfdrive/modeld/modeld.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/modeld.py) (lines 224-229), the process updates the input buffers:

```python
self.full_input_queues.enqueue({'features_buffer': vision_outputs_dict['hidden_state'],
                                'desire_pulse': new_desire})
for k in ['desire_pulse', 'features_buffer']:
    self.numpy_inputs[k][:] = self.full_input_queues.get(k)[k]
self.numpy_inputs['traffic_convention'][:] = inputs['traffic_convention']

```

The policy model executes similarly to the vision model (line 229), producing a flattened output array parsed into longitudinal and lateral control plans.

## Publishing Driving Commands

Following inference, modeld combines vision and policy outputs into standardized message formats. The `fill_model_msg` and `fill_pose_msg` functions serialize the parsed results into `modelV2` and `drivingModelData` capnp messages for the openpilot messaging system. Downstream controllers consume these messages to execute steering, acceleration, and braking commands on the vehicle's CAN bus.

## Summary

- **Dual-network architecture**: Modeld runs a TinyGrad vision encoder followed by a policy network to convert camera pixels into driving controls
- **Pre-compiled execution**: Both networks load as pickled TinyGrad graphs via `pickle.loads()`, eliminating runtime compilation overhead
- **JIT preprocessing**: Camera undistortion and resizing use TinyGrad's `TinyJit` compiler, cached as `warp_{w}x{h}_tinygrad.pkl` for hardware-accelerated image warping
- **Hardware abstraction**: The process automatically selects `QCOM` or `AMD` backends to target comma device silicon or external GPUs
- **Real-time constraints**: All operations use `.realize()` to force synchronous execution, ensuring deterministic latency for safety-critical driving inference

## Frequently Asked Questions

### How does modeld handle different hardware backends?

Modeld detects available hardware at startup and sets the TinyGrad `Device` accordingly. For TICI boards using Qualcomm Snapdragon SoCs, it configures `QCOM`; when a USB-connected AMD GPU is present, it switches to `AMD`. This hardware abstraction allows the same pickled model files to execute across different comma device generations without code modification.

### Why does openpilot use TinyGrad instead of TensorFlow or PyTorch?

TinyGrad provides a lightweight, pure-Python autodiff engine that compiles to efficient backend code for embedded Qualcomm and AMD hardware. According to the openpilot source, this reduces binary size and eliminates heavy framework dependencies while still achieving the real-time performance necessary for 20Hz driving inference on automotive-grade compute modules.

### What is the purpose of the compile_warp.py script?

The [`selfdrive/modeld/compile_warp.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/compile_warp.py) script generates and JIT-compiles the image preprocessing functions used by modeld. It builds TinyGrad computation graphs for perspective warping and tensor rearrangement, then serializes them as pickle files (`warp_{w}x{h}_tinygrad.pkl`). These pre-compiled warps run faster than CPU-based OpenCV operations and integrate seamlessly with the TinyGrad inference pipeline.

### How are the vision and policy model outputs combined?

The `Parser` class in [`selfdrive/modeld/parse_model_outputs.py`](https://github.com/commaai/openpilot/blob/main/selfdrive/modeld/parse_model_outputs.py) decodes raw TinyGrad buffer outputs into structured dictionaries. Modeld calls `parse_vision_outputs` to extract perception features and hidden states, then `parse_policy_outputs` to extract control plans. These dictionaries merge into a single state representation passed to `fill_model_msg`, which creates the final messages consumed by openpilot's lateral and longitudinal controllers.