# How to Optimize RF-DETR Inference with torch.compile: A Complete Guide

> Optimize RF-DETR inference with torch.compile by setting compile=True. Achieve maximum GPU performance with this easy guide. Learn how to speed up your object detection models today.

- Repository: [Roboflow/rf-detr](https://github.com/roboflow/rf-detr)
- Tags: how-to-guide
- Published: 2026-09-08

---

**Enable torch.compile in RF-DETR by setting `compile=True` when instantiating the model, which automatically wraps the inference model with `mode="reduce-overhead"` for maximum GPU performance.**

Optimizing RF-DETR inference with torch.compile unlocks significant speedups by leveraging PyTorch 2.0's Inductor compiler. The roboflow/rf-detr repository ships this capability out-of-the-box, but requires explicit opt-in to activate compilation. This guide covers the implementation details, configuration options, and best practices for production inference workloads.

## How torch.compile Works in RF-DETR

RF-DETR implements torch.compile through a conditional gating system that checks hardware compatibility and applies different optimization modes for training versus inference.

### Configuration Flag

The **compile** flag resides in [`src/rfdetr/config.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/config.py) and controls whether the model attempts compilation. When instantiating any RF-DETR model variant, pass `compile=True` to enable the feature:

```python
from rfdetr import RFDETRSmall
model = RFDETRSmall(num_classes=80, compile=True)

```

### Device Gating Logic

Compilation only activates on CUDA devices because the Inductor backend delivers minimal benefits on CPU or MPS. In [`src/rfdetr/training/module_model.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/training/module_model.py), the code validates three conditions before compiling:

1. `model_config.compile` must be `True`
2. `DEVICE` constant equals `"cuda"`
3. PyTorch Lightning uses a CUDA accelerator

If any condition fails, RF-DETR logs an informational message and falls back to eager mode:

```python

# src/rfdetr/training/module_model.py (lines 14-50)

compile_enabled = model_config.compile and DEVICE == "cuda" and uses_cuda_accelerator
if not compile_enabled:
    logger.info(
        "Disabling torch.compile: RF-DETR enables it only on a CUDA device",
        DEVICE, accelerator,
    )
if compile_enabled:
    torch._dynamo.config.suppress_errors = True
    torch._dynamo.config.capture_scalar_outputs = True
    self.model = torch.compile(self.model, dynamic=True)

```

### Dynamic Graph Support

For training, RF-DETR uses `dynamic=True`, allowing the compiled graph to handle variable input resolutions without recompilation. This multi-scale support is critical for DETR-style models that process different image sizes during training.

## Inference-Specific Optimizations

When loading a model for inference via `rfdetr.inference` or `RFDETRBase.predict()`, RF-DETR applies a specialized compilation mode optimized for production latency.

### Reduce-Overhead Mode

In [`src/rfdetr/detr.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/detr.py) at line 1445, the inference pipeline wraps the model with `mode="reduce-overhead"`, which minimizes the fixed cost of entering compiled regions:

```python

# src/rfdetr/detr.py (line 1445)

if model_config.compile:
    inference_model = torch.compile(inference_model, mode="reduce-overhead")

```

This mode delivers the fastest inference speeds for batch size ≥ 1 on GPU hardware.

## Step-by-Step Optimization Workflow

Follow these steps to activate torch.compile for your inference pipeline:

1. **Verify PyTorch version** (requires 2.0 or newer):
   ```bash
   pip install "torch>=2.0,<3.0"
   ```

2. **Enable compilation** during model instantiation:
   ```python
   model = RFDETRSmall(num_classes=80, compile=True).eval().to("cuda")
   ```

3. **Warm up the model** with a dummy forward pass to trigger compilation before measuring performance.

4. **Run production inference** – subsequent calls reuse the compiled graph for minimal latency.

## Practical Code Examples

### Basic Inference with Compilation

This example demonstrates the standard workflow for running compiled inference on CUDA:

```python
import torch
from rfdetr import RFDETRSmall

# Enable torch.compile (requires CUDA)

model = RFDETRSmall(num_classes=80, compile=True).eval().to("cuda")

# Prepare dummy input (batch=1, 3×640×640)

x = torch.randn(1, 3, 640, 640, device="cuda")

with torch.no_grad():
    # First call triggers compilation (may be slower)

    preds = model(x)

print(preds.keys())

```

### Measuring Throughput

Benchmark your model to verify the compilation benefits:

```python
import time
import torch
from rfdetr import RFDETRSmall

model = RFDETRSmall(num_classes=80, compile=True).eval().to("cuda")
x = torch.randn(1, 3, 640, 640, device="cuda")

# Warm-up: triggers compilation

with torch.no_grad():
    _ = model(x)

# Benchmark loop

num_iters = 100
start = time.time()
with torch.no_grad():
    for _ in range(num_iters):
        _ = model(x)

avg_latency = (time.time() - start) / num_iters * 1000
print(f"Avg latency: {avg_latency:.2f} ms")

```

### Loading Compiled Checkpoints

RF-DETR handles state dict compatibility automatically when loading checkpoints saved from compiled training runs. The code in [`src/rfdetr/models/weights.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/models/weights.py) strips the `"_orig_mod."` prefix that torch.compile adds to parameter keys:

```python
from rfdetr import RFDETRSmall

# Load checkpoint saved from compiled training

model = RFDETRSmall(num_classes=80, compile=True).load_from_checkpoint("path/to/ckpt.pth")
model.eval().to("cuda")

```

## Performance Considerations and Limitations

Understanding when to use torch.compile prevents configuration errors and ensures optimal inference speeds.

### When to Enable Compilation

- **GPU inference with batch size ≥ 1**: Set `compile=True` for maximum throughput
- **Multi-scale inference**: The dynamic graph support efficiently handles variable resolutions
- **CPU or MPS inference**: Keep `compile=False` – Inductor offers minimal benefit on non-CUDA devices
- **Model export**: Disable compilation for ONNX or CoreML export pipelines

### Error Handling and Compatibility

RF-DETR implements safeguards to prevent compilation failures from crashing your inference:

- **Version checking**: The code guards against PyTorch < 2.0 by checking runtime availability
- **Error suppression**: `torch._dynamo.config.suppress_errors = True` ensures that if a subgraph fails to compile (e.g., custom CUDA kernels), execution falls back to eager mode automatically
- **Logging**: Check the logger output for messages like "Disabling torch.compile" if compilation conditions are not met

## Summary

- RF-DETR supports torch.compile through a simple `compile=True` flag in the model constructor
- Compilation only activates on CUDA devices with specific gating logic in [`src/rfdetr/training/module_model.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/training/module_model.py)
- Inference uses `mode="reduce-overhead"` in [`src/rfdetr/detr.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/detr.py) for production latency optimization
- The `dynamic=True` setting handles multi-scale inference without graph recompilation
- Checkpoints saved from compiled models load correctly thanks to prefix stripping in [`src/rfdetr/models/weights.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/models/weights.py)
- Enable error suppression and check logs to debug compilation issues without crashing the pipeline

## Frequently Asked Questions

### What PyTorch version is required for RF-DETR torch.compile support?

You need PyTorch 2.0 or newer. The code guards against older versions by checking runtime availability, but compilation only provides benefits with PyTorch 2.x's Inductor backend.

### Why does my model run slower on the first inference call?

The initial forward pass triggers graph compilation and tracing, which adds overhead. After the first call, subsequent inferences use the cached compiled graph and run significantly faster. Always perform a warm-up pass before benchmarking.

### Can I use torch.compile with CPU inference or Apple Silicon?

No. RF-DETR explicitly disables compilation unless `DEVICE == "cuda"` and PyTorch Lightning uses a CUDA accelerator. The Inductor compiler currently provides minimal speedups on CPU or MPS (Apple Silicon) hardware.

### How does RF-DETR handle checkpoints saved from compiled models?

When you save a state dict from a compiled model, PyTorch prefixes keys with `"_orig_mod."`. RF-DETR automatically strips this prefix during loading in [`src/rfdetr/models/weights.py`](https://github.com/roboflow/rf-detr/blob/main/src/rfdetr/models/weights.py), ensuring checkpoints work transparently regardless of whether they were saved with compilation enabled.