How to Debug KV Cache Issues in Ling Bot-Map Using the LINGBOT_DEBUG_KV Environment Variable

TLDR: Set the LINGBOT_DEBUG_KV environment variable to an integer value to enable per-frame logging of KV cache statistics—including entries, hit rates, and memory usage—from the streaming transformer models in Ling Bot-Map.

Ling Bot-Map manages attention key/value (KV) caching inside its transformer-based streaming models to optimize inference over long sequences. When diagnosing unexpected GPU OOM errors or sudden performance degradations, the LINGBOT_DEBUG_KV environment variable provides real-time visibility into cache behavior without modifying any model logic.

How LINGBOT_DEBUG_KV Works Internally

The debug mechanism is implemented across the streaming model files, where the environment variable is read at import time and used to gate conditional logging inside the forward pass.

Reading the Variable at Import

In lingbot_map/models/gct_stream.py, the code reads the variable once when the module loads:

# lingbot_map/models/gct_stream.py (line 24-25)

_KV_DEBUG = os.environ.get("LINGBOT_DEBUG_KV", "")

A similar pattern exists in lingbot_map/models/gct_stream_window_v2.py, ensuring both the standard and windowed streaming variants support the same debugging interface.

Parsing the Interval

The raw string value is parsed into an integer interval N that controls logging frequency. If the variable is empty or unset, debugging is disabled. When set to 1, statistics print every frame; when set to 10, output appears every 10th frame.

Emitting Statistics During Forward Pass

Inside the model's forward pass, a conditional block checks _KV_DEBUG and prints a formatted statistics line. According to the source comments in gct_stream_window_v2.py, the output includes:

• Frame index
• Number of KV entries allocated
• Cache hit-rate percentage
• Memory used (MiB)

The actual low-level cache data originates from lingbot_map/layers/flashinfer_cache.py, which manages the underlying KV cache buffers.

When to Enable KV Debugging

Enable this debugging mode when you encounter specific runtime issues:

• Unexpected GPU OOM during long sequences – Detect if the KV cache grows unbounded rather than reusing entries.
• Sudden slowdown after specific frames – Identify low hit-rates that force repeated recomputation of attention weights.
• Verifying windowed cache behavior – Confirm that gct_stream_window_v2.py correctly reuses KV entries across sliding windows.
• Profiling memory scaling – Observe how mem values correlate with input resolution changes.

How to Enable and Use LINGBOT_DEBUG_KV

Export the variable before invoking any Python script that builds the model:

# Enable per-frame statistics (interval = 1)

export LINGBOT_DEBUG_KV=1
python demo.py --input data/example/loop/

# Reduce log verbosity by printing every 20 frames

export LINGBOT_DEBUG_KV=20
python -m lingbot_map.demo --seq your_sequence.bin

Capturing Output to a Log File

Redirect both stdout and stderr to preserve the debug lines for analysis:

export LINGBOT_DEBUG_KV=1
python demo.py 2>&1 | tee kv_debug.log

You can then extract statistics using grep:

grep "\[KV DEBUG\]" kv_debug.log > kv_stats_only.log

Understanding the Output Format

When enabled, the streaming transformers emit lines formatted as:

[KV DEBUG] frame=23  kv_entries=128  hit_rate=97.7%  mem=84.2MiB
[KV DEBUG] frame=24  kv_entries=128  hit_rate=98.0%  mem=84.2MiB

These values reflect the actual state of the cache managed by flashinfer_cache.py. Stable kv_entries with high hit_rate values indicate healthy cache reuse, while monotonically increasing entries suggest memory leaks or improper cache clearing.

Performance Considerations

The debug printing incurs minimal overhead—only string formatting and a print call. It does not modify KV-cache allocation strategy, window sizes, or model behavior, making it safe for use in production debugging sessions where you need to observe real memory pressure.

Summary

• Set LINGBOT_DEBUG_KV to an integer N to print KV cache statistics every N frames.
• Statistics include frame index, entry count, hit rate percentage, and memory usage in MiB.
• Implemented in lingbot_map/models/gct_stream.py and gct_stream_window_v2.py with data sourced from lingbot_map/layers/flashinfer_cache.py.
• Zero impact on model correctness; adds only observational overhead through conditional print statements.

Frequently Asked Questions

What values can I set for LINGBOT_DEBUG_KV?

Any positive integer. The value represents the interval between printed frames. Setting it to 1 outputs every frame, while 10 outputs every 10th frame. An empty or unset value disables the output entirely.

Does enabling this variable change how the KV cache is allocated?

No. The variable is strictly observational. According to the source code in gct_stream.py, the debug logic wraps existing statistics in conditional print statements without modifying the underlying cache management in flashinfer_cache.py or altering the model's forward pass behavior.

Why don't I see any output after setting the variable?

Ensure the variable is exported in your shell environment before the Python process starts. Since the code reads the variable at import time (line 24 in gct_stream.py), setting it after importing the module will have no effect. Also verify you are running a script that instantiates the streaming transformer classes, as the logging occurs inside their forward methods.

Can I use this to debug memory leaks in long-running sequences?

Yes. By logging the mem and kv_entries fields over time, you can correlate frame index with memory growth. If kv_entries increases monotonically without bound while processing a fixed window size, it indicates the cache is not releasing old entries, pointing to a management issue in the cache layer rather than a gradual system leak.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}