How ByteTrack Handles Object Occlusion and Track Recovery in Python

ByteTrack handles object occlusion and track recovery through a temporal buffering system that maintains lost tracks for a configurable number of frames and a two-stage re-association pipeline that matches occluded tracks with new detections.

ByteTrack is a multi-object tracking algorithm implemented in the roboflow/supervision repository that solves the challenging problem of maintaining object identity during temporary occlusions. Unlike simpler trackers that immediately discard missed detections, ByteTrack uses a sophisticated state management system to preserve track identities and recover them when objects reappear.

ByteTrack Track State Management

At the core of ByteTrack’s occlusion handling are three distinct containers that categorize every object’s current status.

According to the source code in src/supervision/tracker/byte_tracker/core.py (lines 61-64), ByteTrack maintains:

• tracked_tracks – Objects currently visible and actively matched with detections.
• lost_tracks – Objects that have disappeared temporarily and are waiting for potential recovery.
• removed_tracks – Objects that have exceeded the maximum allowed disappearance time and are permanently discarded.

This three-state architecture ensures that briefly occluded objects are not immediately deleted, preserving their trajectory history and unique IDs for potential re-linking.

Temporal Buffer Configuration

The key parameter controlling occlusion tolerance is lost_track_buffer, which defines how many frames a track can remain missing before being considered permanently lost.

In src/supervision/tracker/byte_tracker/core.py (lines 45-48), the buffer size is calculated based on the user-provided lost_track_buffer value and the video frame rate. By default, ByteTrack keeps tracks in the lost_tracks container for 30 frames. This temporal window provides the algorithm time to recover objects that pass behind obstacles, move out of frame temporarily, or experience detection failures.

You can configure this behavior during initialization:

import supervision as sv

# Keep lost tracks alive for 50 frames (approx. 1.6 seconds at 30 FPS)

tracker = sv.ByteTrack(lost_track_buffer=50)

Two-Stage Association and Re-Activation Logic

ByteTrack’s recovery mechanism operates through a sophisticated two-stage data association process that runs on every new frame.

Stage 1: High-Score Detection Association

First, ByteTrack predicts the positions of all existing tracks using a Kalman filter. It then attempts to match these predictions with high-confidence detections. If a track currently residing in lost_tracks successfully matches a detection, the system calls the re_activate() method (defined in src/supervision/tracker/byte_tracker/core.py, lines 26-34).

This method:

1. Updates the track’s Kalman filter state with the new detection.
2. Moves the track from lost_tracks back to tracked_tracks.
3. Preserves the original track ID, maintaining trajectory continuity.

Stage 2: Low-Score Detection Recovery

The second association round, implemented in src/supervision/tracker/byte_tracker/core.py (lines 36-70), processes lower-scoring detections to rescue tracks that were not matched in the first round. This "refind" step is critical for recovering objects that may have been partially occluded or detected with lower confidence during the occlusion event.

The matching logic in src/supervision/tracker/byte_tracker/matching.py implements the IoU-based cost matrix and linear assignment algorithm used in both stages.

Maximum Time Lost Enforcement

If a track remains in lost_tracks longer than the calculated max_time_lost (derived from lost_track_buffer), it is promoted to removed_tracks (see src/supervision/tracker/byte_tracker/core.py, lines 30-36). At this point, the track ID is permanently retired and will not be considered for future re-activation.

Complete Implementation Example

The following example demonstrates how ByteTrack automatically handles occlusion during video processing:

import supervision as sv
from ultralytics import YOLO
import numpy as np

model = YOLO("yolov8n.pt")
tracker = sv.ByteTrack(lost_track_buffer=50)
box_annotator = sv.BoxAnnotator()
label_annotator = sv.LabelAnnotator()

def process_frame(frame: np.ndarray) -> np.ndarray:
    # Detection

    result = model(frame)[0]
    detections = sv.Detections.from_ultralytics(result)
    
    # Update tracker - handles occlusion internally via Kalman filter

    # and two-stage association

    detections = tracker.update_with_detections(detections)
    
    # Visualize with persistent IDs

    labels = [f"Track #{i}" for i in detections.tracker_id]
    annotated = box_annotator.annotate(frame.copy(), detections)
    annotated = label_annotator.annotate(annotated, detections, labels)
    
    return annotated

In this implementation, lost_track_buffer=50 allows the tracker to maintain track IDs for approximately 50 frames after an object disappears, enabling seamless recovery when the object reappears from occlusion.

Summary

• ByteTrack in the roboflow/supervision repository uses a three-state system (tracked_tracks, lost_tracks, removed_tracks) to manage object lifecycle.
• The lost_track_buffer parameter (default 30 frames) defines how long occluded tracks remain available for recovery.
• A two-stage association pipeline first matches high-confidence detections, then attempts to recover remaining tracks using lower-confidence detections.
• The re_activate() method in STrack (defined in single_object_track.py) restores lost tracks to active status while preserving their original IDs.
• Tracks exceeding max_time_lost are permanently moved to removed_tracks and cannot be recovered.

Frequently Asked Questions

How long does ByteTrack keep a track alive after it disappears?

ByteTrack retains lost tracks in the lost_tracks container for the duration specified by the lost_track_buffer parameter. By default, this is 30 frames, though you can configure it based on your video frame rate and expected occlusion duration. Once this time expires, the track moves to removed_tracks and the ID is permanently retired.

What happens when an occluded object reappears in ByteTrack?

When a previously lost object reappears, ByteTrack’s Kalman filter predicts its expected position. If the prediction aligns with a new detection during either the first or second association round, the re_activate() method restores the track to tracked_tracks with its original ID intact, preserving trajectory continuity without creating a new track identity.

Does ByteTrack use appearance features to recover occluded tracks?

No, the ByteTrack implementation in roboflow/supervision relies strictly on motion information via Kalman filter predictions and IoU-based matching (implemented in matching.py). It does not use appearance embeddings or re-identification features, making it lightweight but potentially less robust to long-term occlusions where motion cues are unreliable.

Where is the track state defined in the source code?

The track state logic resides in src/supervision/tracker/byte_tracker/single_object_track.py, which defines the STrack class. This class manages the Kalman filter state, activation status, and the re_activate() method. The container management and association logic are implemented in src/supervision/tracker/byte_tracker/core.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}