# How to Use CUDA Event-Based Timing for Debugging in Nanotron

> Learn how to use Nanotron's CUDA event-based timing for efficient GPU debugging. Measure execution precisely without blocking CPU threads using torch.cuda.Event.

- Repository: [Hugging Face/nanotron](https://github.com/huggingface/nanotron)
- Tags: how-to-guide
- Published: 2026-03-03

---

**Nanotron provides a lightweight CUDA event-based timing subsystem via the `nanotron_timer` singleton in [`src/nanotron/logging/timers.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/logging/timers.py), enabling precise GPU execution measurements without blocking CPU threads using `torch.cuda.Event` pairs.**

CUDA event-based timing in Nanotron allows you to profile training loops, data loading, and model operations with minimal overhead. Unlike CPU-based timers that require `torch.cuda.synchronize()`, the CUDA event approach records GPU timestamps asynchronously, giving you accurate kernel duration measurements without stalling the host. This article explains how to enable and use the three available patterns—decorators, context managers, and direct API calls—based on the actual implementation in the Hugging Face Nanotron repository.

## Enabling the Timer Subsystem

Before collecting measurements, you must activate the timing infrastructure. Nanotron checks the `ENABLE_TIMERS` environment variable at initialization to determine whether to record events.

Enable timers via environment variable:

```bash
export ENABLE_TIMERS=1

```

Or enable programmatically at runtime:

```python
from nanotron.logging.timers import nanotron_timer

nanotron_timer.enable()

```

When disabled, all timer calls become no-ops, ensuring zero overhead in production training runs.

## Usage Patterns for CUDA Event-Based Timing

The `nanotron_timer` singleton supports three distinct patterns for instrumenting your code. All patterns default to `TimerType.CUDA` but accept an optional `timer_type` parameter to switch to CPU timing when needed.

### Method 1: Decorator for Function-Level Profiling

The decorator pattern automatically wraps function execution with start and end events. By default, it uses the function name as the timer identifier.

```python
from nanotron.logging.timers import nanotron_timer
import torch

nanotron_timer.enable()

@nanotron_timer
def train_step():
    x = torch.randn(1024, 1024, device="cuda")
    y = torch.matmul(x, x)
    return y

train_step()
nanotron_timer.log_all(rank=None)

```

For custom timer names and explicit synchronization, pass additional parameters:

```python
from nanotron.logging.timers import TimerType

@nanotron_timer("forward_pass", timer_type=TimerType.CUDA, cuda_sync=True)
def forward():
    x = torch.randn(512, 512, device="cuda")
    return torch.matmul(x, x)

```

Setting `cuda_sync=True` inserts `torch.cuda.synchronize()` before recording events, eliminating measurement errors from asynchronous kernel execution at the cost of slight overhead.

### Method 2: Context Manager for Code Blocks

Use the context manager pattern to profile specific sections within a larger function, such as data loading or gradient synchronization.

```python
from nanotron.logging.timers import nanotron_timer

def load_batch():
    with nanotron_timer("dataloader_fetch"):
        batch = torch.randn(64, 1024, device="cuda")
        torch.cuda.synchronize()
    return batch

```

As implemented in [`src/nanotron/data/dataloader.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/data/dataloader.py), the timer starts when entering the `with` block and stops on exit, automatically handling CUDA event recording regardless of whether an exception occurs.

### Method 3: Direct API for Fine-Grained Control

For loops or conditional logic, manually trigger `start()` and `end()` methods on a named timer instance:

```python
def training_loop(num_iters: int):
    for i in range(num_iters):
        nanotron_timer("iteration").start()
        
        # Training logic here

        loss = model(batch)
        loss.backward()
        
        nanotron_timer("iteration").end()
    
    nanotron_timer.log("iteration")

```

This pattern appears in [`src/nanotron/trainer.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/trainer.py) where distinct training phases require separate timing buckets within a single iteration.

## Understanding CUDA Event Implementation

The timing accuracy relies on `torch.cuda.Event` objects created in [`src/nanotron/logging/timers.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/logging/timers.py). When `start()` is called on a CUDA timer, the implementation records:

```python
self._current_start_event = torch.cuda.Event(enable_timing=True)
self._current_start_event.record()

```

Upon calling `end()`, an end event is recorded and the pair is stored in `_cuda_events`. The elapsed time calculation uses PyTorch's `Event.elapsed_time()` method, which returns milliseconds measured on the GPU clock. The `TimerRecord` class tracks multiple event pairs per timer name, allowing aggregation across training steps.

Key methods on the timer object include:

- **`elapsed`** – Returns the duration of the most recent start/end pair.
- **`total_time`** – Synchronizes all stored event pairs and sums their durations.
- **`average_time`** – Computes `total_time / call_count` for mean latency analysis.

## Practical Examples for Distributed Training

In multi-GPU scenarios, you often need to time operations that span communication collectives. The following example demonstrates timing gradient synchronization with explicit CUDA synchronization for maximum accuracy:

```python
from nanotron.logging.timers import nanotron_timer

nanotron_timer("all_reduce_grads", cuda_sync=True).start()

# Gradient synchronization across ranks

torch.distributed.all_reduce(grads)

nanotron_timer("all_reduce_grads").end()

# Log results only on specific ranks

nanotron_timer.log("all_reduce_grads", rank=0)

```

Call `log_all(rank=None)` to print aggregated statistics across all timers on every process, which is useful for verifying load balancing in distributed configurations.

## Summary

- **Enable timers** using `export ENABLE_TIMERS=1` or `nanotron_timer.enable()` before training.
- **Use decorators** (`@nanotron_timer`) for automatic function wrapping with default CUDA event timing.
- **Apply context managers** (`with nanotron_timer("name"):`) for profiling specific code blocks without refactoring functions.
- **Call the direct API** (`start()`/`end()`) for loop-level granularity and conditional timing.
- **Set `cuda_sync=True`** when you need precise measurements that account for GPU kernel overlap, accepting the minor synchronization overhead.
- **Reference [`src/nanotron/logging/timers.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/logging/timers.py)** for the `TimerRecord` and `Timers` class implementations using `torch.cuda.Event`.

## Frequently Asked Questions

### What is the difference between CUDA and CPU timers in Nanotron?

CUDA timers use `torch.cuda.Event` pairs to measure GPU-side execution time without blocking the CPU host, while CPU timers fall back to `time.time()` for pure Python or host-bound operations. Specify `timer_type=TimerType.CPU` when profiling data preprocessing or I/O operations that do not involve GPU kernels.

### How do I synchronize CUDA events to get accurate timing?

By default, Nanotron records events asynchronously. To force synchronization before measuring elapsed time, pass `cuda_sync=True` when creating the timer. This triggers `torch.cuda.synchronize()` at each start and end point, ensuring all preceding kernels complete before the timestamp is recorded. Use this only when precise isolation is required, as it introduces CPU-GPU synchronization overhead.

### Can I use timers in distributed training across multiple GPUs?

Yes, the `nanotron_timer` singleton operates independently per process. Use `nanotron_timer.log_all(rank=None)` to output timing statistics from all ranks, or specify a specific rank (e.g., `rank=0`) to limit logging to the main process. Each rank maintains its own `TimerRecord` instances and CUDA event pools.

### What is the overhead of using CUDA event-based timing?

When timers are disabled via `ENABLE_TIMERS=0`, the overhead is zero due to no-op implementations. When enabled, CUDA events add minimal overhead—typically microseconds per invocation—because they record timestamps asynchronously without blocking. The `cuda_sync=True` mode increases overhead significantly by forcing device synchronization, so it should be used sparingly for bottleneck analysis rather than continuous monitoring.