How to Use CUDA Event-Based Timing for Debugging in Nanotron

Nanotron provides a lightweight CUDA event-based timing subsystem via the nanotron_timer singleton in src/nanotron/logging/timers.py, enabling precise GPU execution measurements without blocking CPU threads using torch.cuda.Event pairs.

CUDA event-based timing in Nanotron allows you to profile training loops, data loading, and model operations with minimal overhead. Unlike CPU-based timers that require torch.cuda.synchronize(), the CUDA event approach records GPU timestamps asynchronously, giving you accurate kernel duration measurements without stalling the host. This article explains how to enable and use the three available patterns—decorators, context managers, and direct API calls—based on the actual implementation in the Hugging Face Nanotron repository.

Enabling the Timer Subsystem

Before collecting measurements, you must activate the timing infrastructure. Nanotron checks the ENABLE_TIMERS environment variable at initialization to determine whether to record events.

Enable timers via environment variable:

export ENABLE_TIMERS=1

Or enable programmatically at runtime:

from nanotron.logging.timers import nanotron_timer

nanotron_timer.enable()

When disabled, all timer calls become no-ops, ensuring zero overhead in production training runs.

Usage Patterns for CUDA Event-Based Timing

The nanotron_timer singleton supports three distinct patterns for instrumenting your code. All patterns default to TimerType.CUDA but accept an optional timer_type parameter to switch to CPU timing when needed.

Method 1: Decorator for Function-Level Profiling

The decorator pattern automatically wraps function execution with start and end events. By default, it uses the function name as the timer identifier.

from nanotron.logging.timers import nanotron_timer
import torch

nanotron_timer.enable()

@nanotron_timer
def train_step():
    x = torch.randn(1024, 1024, device="cuda")
    y = torch.matmul(x, x)
    return y

train_step()
nanotron_timer.log_all(rank=None)

For custom timer names and explicit synchronization, pass additional parameters:

from nanotron.logging.timers import TimerType

@nanotron_timer("forward_pass", timer_type=TimerType.CUDA, cuda_sync=True)
def forward():
    x = torch.randn(512, 512, device="cuda")
    return torch.matmul(x, x)

Setting cuda_sync=True inserts torch.cuda.synchronize() before recording events, eliminating measurement errors from asynchronous kernel execution at the cost of slight overhead.

Method 2: Context Manager for Code Blocks

Use the context manager pattern to profile specific sections within a larger function, such as data loading or gradient synchronization.

from nanotron.logging.timers import nanotron_timer

def load_batch():
    with nanotron_timer("dataloader_fetch"):
        batch = torch.randn(64, 1024, device="cuda")
        torch.cuda.synchronize()
    return batch

As implemented in src/nanotron/data/dataloader.py, the timer starts when entering the with block and stops on exit, automatically handling CUDA event recording regardless of whether an exception occurs.

Method 3: Direct API for Fine-Grained Control

For loops or conditional logic, manually trigger start() and end() methods on a named timer instance:

def training_loop(num_iters: int):
    for i in range(num_iters):
        nanotron_timer("iteration").start()
        
        # Training logic here

        loss = model(batch)
        loss.backward()
        
        nanotron_timer("iteration").end()
    
    nanotron_timer.log("iteration")

This pattern appears in src/nanotron/trainer.py where distinct training phases require separate timing buckets within a single iteration.

Understanding CUDA Event Implementation

The timing accuracy relies on torch.cuda.Event objects created in src/nanotron/logging/timers.py. When start() is called on a CUDA timer, the implementation records:

self._current_start_event = torch.cuda.Event(enable_timing=True)
self._current_start_event.record()

Upon calling end(), an end event is recorded and the pair is stored in _cuda_events. The elapsed time calculation uses PyTorch's Event.elapsed_time() method, which returns milliseconds measured on the GPU clock. The TimerRecord class tracks multiple event pairs per timer name, allowing aggregation across training steps.

Key methods on the timer object include:

• elapsed – Returns the duration of the most recent start/end pair.
• total_time – Synchronizes all stored event pairs and sums their durations.
• average_time – Computes total_time / call_count for mean latency analysis.

Practical Examples for Distributed Training

In multi-GPU scenarios, you often need to time operations that span communication collectives. The following example demonstrates timing gradient synchronization with explicit CUDA synchronization for maximum accuracy:

from nanotron.logging.timers import nanotron_timer

nanotron_timer("all_reduce_grads", cuda_sync=True).start()

# Gradient synchronization across ranks

torch.distributed.all_reduce(grads)

nanotron_timer("all_reduce_grads").end()

# Log results only on specific ranks

nanotron_timer.log("all_reduce_grads", rank=0)

Call log_all(rank=None) to print aggregated statistics across all timers on every process, which is useful for verifying load balancing in distributed configurations.

Summary

• Enable timers using export ENABLE_TIMERS=1 or nanotron_timer.enable() before training.
• Use decorators (@nanotron_timer) for automatic function wrapping with default CUDA event timing.
• Apply context managers (with nanotron_timer("name"):) for profiling specific code blocks without refactoring functions.
• Call the direct API (start()/end()) for loop-level granularity and conditional timing.
• Set cuda_sync=True when you need precise measurements that account for GPU kernel overlap, accepting the minor synchronization overhead.
• Reference src/nanotron/logging/timers.py for the TimerRecord and Timers class implementations using torch.cuda.Event.

Frequently Asked Questions

What is the difference between CUDA and CPU timers in Nanotron?

CUDA timers use torch.cuda.Event pairs to measure GPU-side execution time without blocking the CPU host, while CPU timers fall back to time.time() for pure Python or host-bound operations. Specify timer_type=TimerType.CPU when profiling data preprocessing or I/O operations that do not involve GPU kernels.

How do I synchronize CUDA events to get accurate timing?

By default, Nanotron records events asynchronously. To force synchronization before measuring elapsed time, pass cuda_sync=True when creating the timer. This triggers torch.cuda.synchronize() at each start and end point, ensuring all preceding kernels complete before the timestamp is recorded. Use this only when precise isolation is required, as it introduces CPU-GPU synchronization overhead.

Can I use timers in distributed training across multiple GPUs?

Yes, the nanotron_timer singleton operates independently per process. Use nanotron_timer.log_all(rank=None) to output timing statistics from all ranks, or specify a specific rank (e.g., rank=0) to limit logging to the main process. Each rank maintains its own TimerRecord instances and CUDA event pools.

What is the overhead of using CUDA event-based timing?

When timers are disabled via ENABLE_TIMERS=0, the overhead is zero due to no-op implementations. When enabled, CUDA events add minimal overhead—typically microseconds per invocation—because they record timestamps asynchronously without blocking. The cuda_sync=True mode increases overhead significantly by forcing device synchronization, so it should be used sparingly for bottleneck analysis rather than continuous monitoring.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}