# How to Debug Lock Contention with LiteBox Lock Tracing

> Debug lock contention with LiteBox lock tracing. Analyze JSONL files of lock attempts, acquisitions, and releases to pinpoint contention hotspots.

- Repository: [Microsoft/litebox](https://github.com/microsoft/litebox)
- Tags: how-to-guide
- Published: 2026-02-16

---

**LiteBox provides a built-in lock-tracing subsystem that records every lock attempt, acquisition, and release to JSONL files, enabling precise identification of contention hotspots through timestamp analysis.**

Debugging lock contention in concurrent Rust applications requires deep visibility into synchronization primitive behavior. The `microsoft/litebox` repository ships with a sophisticated **lock-tracing** feature that instruments mutexes and read-write locks to capture detailed timing data with minimal overhead. This guide explains how to enable the subsystem, record contention events, and analyze the resulting traces to optimize your application's parallel performance.

## Enabling Lock Tracing in LiteBox

### Activating the Cargo Feature

To use the tracing infrastructure, enable the `lock_tracing` feature in your [`Cargo.toml`](https://github.com/microsoft/litebox/blob/main/Cargo.toml). This feature gates the entire subsystem to ensure zero runtime overhead when disabled.

```toml
[dependencies]
litebox = { git = "https://github.com/microsoft/litebox", features = ["lock_tracing"] }

```

### Initializing the Global LockTracker

The tracing system centers on a singleton **LockTracker** that maintains per-thread stacks of held locks. According to the source code in [`litebox/src/litebox.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/litebox.rs), the tracker initializes automatically during `LiteBox::new`:

```rust
// litebox/src/litebox.rs#L62-L66
pub fn new(platform: &dyn Platform) -> Self {
    #[cfg(feature = "lock_tracing")]
    lock_tracing::init();
    // ... remainder of initialization
}

```

## Recording Lock Events

### Starting and Stopping Recording Sessions

The public API exposed in [`litebox/src/sync/mod.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/sync/mod.rs) provides three core functions to control data capture:

```rust
use litebox::sync::{start_recording, stop_recording, flush_to_jsonl};

fn analyze_critical_section() {
    start_recording();
    
    // Your concurrent code here...
    
    stop_recording();
    
    // Write events to stdout or file
    for line in flush_to_jsonl() {
        println!("{}", line);
    }
}

```

### Automatic Recording in Linux Userland

The `litebox_runner_linux_userland` crate demonstrates a full-process recording strategy. As shown in [`litebox_runner_linux_userland/src/lib.rs`](https://github.com/microsoft/litebox/blob/main/litebox_runner_linux_userland/src/lib.rs), the runner wraps program execution with automatic trace collection:

```rust
// litebox_runner_linux_userland/src/lib.rs#L34-L48
pub fn run_with_tracing<F>(f: F) 
where 
    F: FnOnce() 
{
    start_recording();
    f();
    stop_recording();
    
    // Automatically writes to /tmp/locks.jsonl
    std::fs::write("/tmp/locks.jsonl", flush_to_jsonl().join("\n")).unwrap();
}

```

## Detecting Lock Contention

### Console Diagnostics for Contended Locks

The tracing subsystem provides compile-time configuration constants in [`litebox/src/sync/lock_tracing.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/sync/lock_tracing.rs) to control console output. When **CONFIG_PRINT_CONTENDED_LOCKS** is enabled, the system emits "Attempt ... CONTENDED" messages before blocking acquisitions.

The `debug_log_println!` macro handles these emissions around line 92 of the same file. Additionally, setting **CONFIG_PRINT_LOCKS_SLOWER_THAN** (default 10ms) triggers "LONG WAIT ..." messages for acquisitions exceeding the threshold.

### Identifying Slow Lock Acquisitions in JSONL Traces

For quantitative analysis, examine the JSON Lines output produced by `flush_to_jsonl()`. Each event contains nanosecond-precision timestamps:

```json
{"event_type":"attempt","timestamp_ns":123456789,"lock_addr":"0x7fffd1234abc","lock_type":"Mutex","file":"src/main.rs","line":42}
{"event_type":"acquired","timestamp_ns":123476789,"lock_addr":"0x7fffd1234abc","lock_type":"Mutex","file":"src/main.rs","line":42}

```

The delta between `attempt` and `acquired` timestamps reveals contention duration. In this example, the 20,000 nanosecond (20µs) gap indicates the thread waited for another holder to release the lock.

## Analyzing Lock Traces

### Understanding JSONL Event Format

The **EVENT_RECORDER** buffer stores structured events defined in [`litebox/src/sync/lock_tracing.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/sync/lock_tracing.rs). When flushed, each `RecordedEvent` serializes to JSON with these fields:

- **event_type**: Operation classification (`attempt`, `acquired`, `released`, `created`, `destroyed`)
- **timestamp_ns**: Monotonic nanoseconds since tracker initialization
- **lock_addr**: Memory address identifying the specific lock instance
- **lock_type**: Synchronization primitive type (`Mutex`, `RwLockRead`, `RwLockWrite`)
- **file** / **line**: Source location captured via `file!()` and `line!()` macros

### Calculating Contention from Timestamps

To programmatically identify hotspots, aggregate events by `lock_addr` and compute wait times:

```rust
use std::collections::HashMap;

fn analyze_contention(jsonl_lines: &[String]) {
    let mut attempts: HashMap<String, u64> = HashMap::new();
    
    for line in jsonl_lines {
        let event: serde_json::Value = serde_json::from_str(line).unwrap();
        let addr = event["lock_addr"].as_str().unwrap().to_string();
        let ts = event["timestamp_ns"].as_u64().unwrap();
        
        match event["event_type"].as_str().unwrap() {
            "attempt" => { attempts.insert(addr, ts); }
            "acquired" => {
                if let Some(start) = attempts.remove(&addr) {
                    let wait_ns = ts - start;
                    if wait_ns > 10_000_000 { // 10ms threshold
                        println!("High contention on {}: {}ms wait", addr, wait_ns / 1_000_000);
                    }
                }
            }
            _ => {}
        }
    }
}

```

## Configuring Trace Verbosity

### Compile-Time Configuration Constants

The tracing behavior is controlled by boolean constants defined in [`litebox/src/sync/lock_tracing.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/sync/lock_tracing.rs) (lines 47-69). These are evaluated at compile time to ensure zero-cost when disabled:

- **CONFIG_PRINT_LOCK_ATTEMPTS**: Emit console messages for every lock attempt
- **CONFIG_PRINT_CONTENDED_LOCKS**: Print "CONTENDED" warnings before blocking acquisitions
- **CONFIG_PRINT_LOCKS_SLOWER_THAN**: Threshold in milliseconds for "LONG WAIT" messages (default 10ms)
- **CONFIG_ENABLE_RECORDING**: Buffer events in `EVENT_RECORDER` for JSONL export
- **CONFIG_PANIC_ON_NON_BRACKETED_UNLOCK**: Enable strict lock/unlock pairing validation

To customize, copy [`lock_tracing.rs`](https://github.com/microsoft/litebox/blob/main/lock_tracing.rs) into your crate as a module, modify the constants, and ensure your version takes precedence in the module hierarchy.

## Minimal Example: Recording Contention in Rust

The following complete example demonstrates instrumenting a multi-threaded workload to capture contention data:

```rust
// Cargo.toml
[package]
name = "contention_demo"
version = "0.1.0"
edition = "2021"

[dependencies]
litebox = { git = "https://github.com/microsoft/litebox", features = ["lock_tracing"] }
serde_json = "1.0"

// src/main.rs
use litebox::{LiteBox, sync::{start_recording, stop_recording, flush_to_jsonl}};
use std::sync::Arc;
use std::thread;
use std::time::Duration;

fn main() {
    // Initialize platform and LiteBox (creates LockTracker)
    let platform = litebox_platform_linux_userland::LinuxPlatform::new();
    let _lb = LiteBox::new(&platform);
    
    // Start recording lock events
    start_recording();
    
    // Create a LiteBox Mutex (instrumented when lock_tracing is enabled)
    let mutex = Arc::new(litebox::sync::Mutex::new(0u64));
    let mut handles = vec![];
    
    // Spawn threads that contend for the lock
    for i in 0..4 {
        let mu = Arc::clone(&mutex);
        handles.push(thread::spawn(move || {
            for _ in 0..100 {
                let mut guard = mu.lock();
                *guard += 1;
                // Hold lock briefly to force contention
                thread::sleep(Duration::from_micros(10));
            }
        }));
    }
    
    for h in handles {
        h.join().unwrap();
    }
    
    // Stop recording and output JSONL
    stop_recording();
    println!("\nLock trace output:");
    for line in flush_to_jsonl() {
        println!("{}", line);
    }
}

```

When executed with the `lock_tracing` feature enabled, this program outputs JSON Lines showing every `attempt`, `acquired`, and `released` event. Analyzing the timestamp deltas between `attempt` and `acquired` events reveals which threads experienced contention and for precisely how long.

## Summary

- **LiteBox** provides a compile-time optional lock-tracing subsystem activated via the `lock_tracing` Cargo feature.
- The **LockTracker** singleton initializes during `LiteBox::new` and maintains per-thread lock stacks throughout the process lifetime.
- Use **start_recording** and **stop_recording** to delimit capture windows, then **flush_to_jsonl** to export structured event data.
- Contention manifests as gaps between `attempt` and `acquired` timestamps in the JSONL output, or as "CONTENDED" console messages when `CONFIG_PRINT_CONTENDED_LOCKS` is enabled.
- Configure verbosity via compile-time constants in [`litebox/src/sync/lock_tracing.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/sync/lock_tracing.rs) to tailor diagnostic detail against runtime overhead.

## Frequently Asked Questions

### What is the performance overhead of LiteBox lock tracing?

When the `lock_tracing` feature is disabled, the subsystem imposes **zero runtime cost** due to compile-time conditional compilation. When enabled, overhead depends on configuration: buffering events to `EVENT_RECORDER` adds memory allocation and atomic operations, while console printing via `CONFIG_PRINT_LOCK_ATTEMPTS` incurs synchronous I/O latency. For production diagnostics, use `CONFIG_ENABLE_RECORDING` with periodic `flush_to_jsonl` calls rather than continuous console output to minimize blocking.

### How do I enable lock tracing without modifying LiteBox source code?

Enable the feature in your dependent crate's [`Cargo.toml`](https://github.com/microsoft/litebox/blob/main/Cargo.toml) by specifying `features = ["lock_tracing"]` for the `litebox` dependency. To customize configuration constants without forking the repository, copy [`litebox/src/sync/lock_tracing.rs`](https://github.com/microsoft/litebox/blob/main/litebox/src/sync/lock_tracing.rs) into your project as a local module, modify the `CONFIG_*` boolean constants at the top of the file, and ensure your build configuration prefers your local version over the crate's internal module.

### Can I use lock tracing in production environments?

Yes, provided you configure it for reliability. Set `CONFIG_PANIC_ON_NON_BRACKETED_UNLOCK` to `false` to prevent instrumentation errors from crashing your application. Use `CONFIG_ENABLE_RECORDING` to buffer events in memory, then periodically call `flush_to_jsonl` to write to disk asynchronously. Avoid `CONFIG_PRINT_LOCK_ATTEMPTS` in high-throughput scenarios due to console I/O blocking. The Linux userland runner demonstrates this pattern by writing to `/tmp/locks.jsonl` only after `stop_recording`.

### Where does LiteBox write the lock trace files?

By default, the tracing subsystem returns trace data via `flush_to_jsonl()`, which yields a `Vec<String>` of JSON Lines rather than writing directly to disk. The `litebox_runner_linux_userland` crate demonstrates a typical pattern by explicitly writing to `/tmp/locks.jsonl` after stopping the recording, as shown in [`litebox_runner_linux_userland/src/lib.rs`](https://github.com/microsoft/litebox/blob/main/litebox_runner_linux_userland/src/lib.rs). You can direct output to any path by collecting the strings from `flush_to_jsonl()` and using `std::fs::write` or your preferred logging infrastructure.