How the modeld Process Runs Deep Learning Inference with TinyGrad for Autonomous Driving

The modeld process in openpilot performs real-time driving inference by loading two pre-compiled TinyGrad models—a vision encoder and a policy network—and executing them sequentially on Qualcomm or AMD hardware to generate steering and acceleration commands.

The modeld process serves as the core inference engine within the commaai/openpilot repository, responsible for transforming raw camera frames into actionable driving controls. This article examines how modeld leverages TinyGrad, a lightweight Python autodiff framework, to run both perception and planning neural networks on embedded automotive hardware.

Overview of the modeld Architecture

The modeld process runs two distinct TinyGrad-based neural networks in series to achieve end-to-end driving inference:

• Vision model (driving_vision_tinygrad.pkl): Encodes NV12 camera buffers into a hidden state while simultaneously detecting lane lines, lead vehicles, and other critical perception features
• Policy model (driving_policy_tinygrad.pkl): Consumes the vision model's hidden state alongside high-level inputs (desired driving behavior, traffic convention) to output longitudinal and lateral control plans

Before loading models, modeld forces the TinyGrad runtime to select the correct compute backend. In selfdrive/modeld/modeld.py lines 1-9, the code explicitly sets the device to QCOM for TICI boards or AMD when USB-GPU acceleration is present, ensuring the compiled graphs target the available silicon.

Loading Compiled TinyGrad Models

Rather than constructing computation graphs at runtime, modeld deserializes pre-compiled TinyGrad artifacts from the models/ directory. During ModelState.__init__ in selfdrive/modeld/modeld.py (lines 81-84), the process loads pickled executable graphs using Python's pickle module:

self.vision_run = pickle.loads(read_file_chunked(str(VISION_PKL_PATH)))
self.policy_run = pickle.loads(read_file_chunked(str(POLICY_PKL_PATH)))

These loaded objects become callable Python functions (vision_run and policy_run) that execute the complete forward passes of their respective networks. This approach eliminates compilation overhead during the critical real-time loop, allowing modeld to begin inference immediately after initialization.

Preprocessing Camera Frames with TinyGrad JIT

Raw camera frames arrive as NV12 buffers requiring undistortion, resizing, and plane rearrangement before inference. Modeld handles this preprocessing through a JIT-compiled TinyGrad function rather than traditional CPU-based image processing.

The warp implementation resides in selfdrive/modeld/compile_warp.py, where the warp_perspective_tinygrad function constructs a perspective transformation using TinyGrad operations. The script JIT-compiles this transformation using TinyJit(update_both_imgs, prune=True) and caches the result as warp_{w}x{h}_tinygrad.pkl (lines 28-36 and 70-84).

At runtime, modeld loads this cached JIT function during the first frame processing. In selfdrive/modeld/modeld.py (lines 133-138), the update_imgs callable transforms both low-resolution and high-resolution camera streams into the 6-channel tensor format expected by the vision model:

out = self.update_imgs(self.img_queues['img'], self.full_frames['img'],
                       self.transforms['img'],
                       self.img_queues['big_img'], self.full_frames['big_img'],
                       self.transforms['big_img'])

This JIT fusion eliminates Python-level loops and achieves real-time performance on low-power embedded CPUs and GPUs.

Running the Vision Model

After preprocessing, modeld feeds the warped tensors to the vision network. The vision model accepts two inputs: a low-resolution img tensor and a high-resolution big_img tensor. In selfdrive/modeld/modeld.py (line 221), modeld executes the vision graph and extracts the raw output buffer:

self.vision_output = self.vision_run(**vision_inputs).contiguous().realize().uop.base.buffer.numpy().flatten()

The .contiguous().realize() chain forces TinyGrad to execute the computation graph and materialize results in memory. The Parser class then converts this flattened NumPy array into a structured dictionary containing the hidden state (features_buffer) and perception outputs like lane line coefficients and lead vehicle positions.

Running the Policy Model

The policy network combines temporal information with the current vision features to generate driving commands. Modeld maintains circular buffers (InputQueues) to accumulate historical states, desire pulses, and traffic convention flags.

Before each policy inference cycle, modeld copies the newest temporal slice into NumPy arrays that back the TinyGrad inputs. In selfdrive/modeld/modeld.py (lines 224-229), the process updates the input buffers:

self.full_input_queues.enqueue({'features_buffer': vision_outputs_dict['hidden_state'],
                                'desire_pulse': new_desire})
for k in ['desire_pulse', 'features_buffer']:
    self.numpy_inputs[k][:] = self.full_input_queues.get(k)[k]
self.numpy_inputs['traffic_convention'][:] = inputs['traffic_convention']

The policy model executes similarly to the vision model (line 229), producing a flattened output array parsed into longitudinal and lateral control plans.

Publishing Driving Commands

Following inference, modeld combines vision and policy outputs into standardized message formats. The fill_model_msg and fill_pose_msg functions serialize the parsed results into modelV2 and drivingModelData capnp messages for the openpilot messaging system. Downstream controllers consume these messages to execute steering, acceleration, and braking commands on the vehicle's CAN bus.

Summary

• Dual-network architecture: Modeld runs a TinyGrad vision encoder followed by a policy network to convert camera pixels into driving controls
• Pre-compiled execution: Both networks load as pickled TinyGrad graphs via pickle.loads(), eliminating runtime compilation overhead
• JIT preprocessing: Camera undistortion and resizing use TinyGrad's TinyJit compiler, cached as warp_{w}x{h}_tinygrad.pkl for hardware-accelerated image warping
• Hardware abstraction: The process automatically selects QCOM or AMD backends to target comma device silicon or external GPUs
• Real-time constraints: All operations use .realize() to force synchronous execution, ensuring deterministic latency for safety-critical driving inference

Frequently Asked Questions

How does modeld handle different hardware backends?

Modeld detects available hardware at startup and sets the TinyGrad Device accordingly. For TICI boards using Qualcomm Snapdragon SoCs, it configures QCOM; when a USB-connected AMD GPU is present, it switches to AMD. This hardware abstraction allows the same pickled model files to execute across different comma device generations without code modification.

Why does openpilot use TinyGrad instead of TensorFlow or PyTorch?

TinyGrad provides a lightweight, pure-Python autodiff engine that compiles to efficient backend code for embedded Qualcomm and AMD hardware. According to the openpilot source, this reduces binary size and eliminates heavy framework dependencies while still achieving the real-time performance necessary for 20Hz driving inference on automotive-grade compute modules.

What is the purpose of the compile_warp.py script?

The selfdrive/modeld/compile_warp.py script generates and JIT-compiles the image preprocessing functions used by modeld. It builds TinyGrad computation graphs for perspective warping and tensor rearrangement, then serializes them as pickle files (warp_{w}x{h}_tinygrad.pkl). These pre-compiled warps run faster than CPU-based OpenCV operations and integrate seamlessly with the TinyGrad inference pipeline.

How are the vision and policy model outputs combined?

The Parser class in selfdrive/modeld/parse_model_outputs.py decodes raw TinyGrad buffer outputs into structured dictionaries. Modeld calls parse_vision_outputs to extract perception features and hidden states, then parse_policy_outputs to extract control plans. These dictionaries merge into a single state representation passed to fill_model_msg, which creates the final messages consumed by openpilot's lateral and longitudinal controllers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}