How to Optimize the Non-Maximum Suppression (NMS) Threshold for Specific Detection Scenarios

Optimize your NMS threshold by selecting 0.3–0.5 for dense crowds, 0.5–0.7 for sparse scenes, and 0.2–0.4 for small objects, using the iou_threshold parameter in box_non_max_suppression or mask_non_max_suppression from src/supervision/detection/utils/iou_and_nms.py.

Non-Maximum Suppression (NMS) is the critical post-processing step that eliminates duplicate detections by retaining only the highest-scoring prediction when bounding boxes or masks overlap beyond a configurable IoU (Intersection-over-Union) threshold. In the Roboflow supervision library, optimizing this threshold allows you to balance precision and recall based on whether you are detecting densely packed crowds, isolated objects, or small visual targets. Understanding how to adjust the iou_threshold parameter for your specific detection scenario ensures you maximize model performance without manual trial and error.

How NMS Works in Supervision

The library implements NMS in src/supervision/detection/utils/iou_and_nms.py through two primary functions that process predictions after model inference.

Box-based NMS

The box_non_max_suppression function operates on NumPy arrays of bounding box coordinates. It first validates that the provided iou_threshold falls within the closed range [0, 1], then sorts all predictions by confidence score in descending order. For each detection, it computes pairwise Intersection-over-Union using box_iou_batch and discards any subsequent prediction whose IoU with a higher-scoring box exceeds the threshold.

Mask-based NMS

The mask_non_max_suppression function applies the same logic to segmentation masks, first resizing them to a common dimension (default 640) to ensure consistent comparison. After validating the IoU range, the function iterates through sorted predictions and removes any mask whose overlap with a higher-confidence detection exceeds the threshold while belonging to the same category.

Both functions support an optional overlap_metric parameter, accepting IOU or IOS (Intersection over Smaller), and handle class-aware suppression automatically when the input includes a class column. Omitting the class column triggers class-agnostic mode, allowing cross-class overlap handling.

NMS Threshold Recommendations by Detection Scenario

Choosing the optimal threshold requires matching the value to your scene's spatial density and object characteristics. The following guidelines help you optimize the Non-Maximum Suppression threshold for specific detection scenarios:

Scenario	Recommended IoU Threshold	Rationale
Dense crowds / overlapping objects (e.g., people in queues, vehicles in traffic)	0.3 – 0.5	Lower thresholds prevent legitimate nearby detections from being discarded as duplicates.
Sparse scenes / well-separated objects (e.g., isolated wildlife, traffic signs)	0.5 – 0.7	Higher thresholds aggressively remove spurious duplicates while preserving true positives.
Small objects (e.g., traffic lights, facial landmarks)	0.2 – 0.4	Small boxes often overlap tightly; lower thresholds retain all valid detections.
Large objects (e.g., vehicles, buildings)	0.6 – 0.8	Large boxes rarely overlap incorrectly; higher thresholds reduce false positives without sacrificing recall.
Class-agnostic merging (e.g., ensemble model outputs)	0.4 – 0.6	Balances cross-class overlap tolerance while suppressing obvious duplicates.
Mask-based segmentation (detailed instance masks)	0.3 – 0.5	Mask IoU tends to be stricter than box IoU, requiring modest thresholds to avoid over-pruning.

Practical Tips for Fine-Tuning NMS

When implementing NMS in production pipelines, consider these optimization strategies:

• Start with 0.5: The default threshold works well for generic COCO-style datasets and provides a baseline for comparison.

• Validate empirically: Run precision-recall curves on a representative validation subset, varying the threshold in 0.05 increments to identify the inflection point for your metric of choice.

• Match metrics to thresholds: If optimizing for [email protected], use a slightly higher threshold (0.55–0.65) to boost precision. For mAP@[0.5:0.95], maintain a balanced threshold around 0.5.

• Use IOS for nested objects: Set overlap_metric=sv.OverlapMetric.IOS when detecting small objects inside larger ones, as this metric favors retaining the smaller box.

• Leverage class-aware mode: Ensure your predictions array includes a class column to prevent distinct classes from suppressing each other, which is essential for multi-class detection workflows.

• Resize masks intelligently: Adjust the mask_dimension parameter (default 640) to balance computational speed against mask fidelity before running mask_non_max_suppression.

Implementing Custom NMS Thresholds in Code

The following examples demonstrate how to apply these optimization strategies using the supervision Python package.

Basic Box NMS with Custom Threshold

For crowded scenes, use a lower threshold to preserve overlapping detections:

import numpy as np
import supervision as sv

# Format: (x_min, y_min, x_max, y_max, confidence, class_id)

detections = np.array([
    [100, 120, 200, 220, 0.95, 0],   # person

    [105, 125, 195, 215, 0.90, 0],   # overlapping person

    [300, 400, 350, 450, 0.80, 1],   # car

])

# Low threshold for crowded scenes (0.35)

keep = sv.box_non_max_suppression(detections, iou_threshold=0.35)

print("Indices to keep:", np.where(keep)[0])

# → Keeps highest-scoring person and the car

Mask NMS with Class-Agnostic Mode

When merging detections across categories, omit the class column to suppress overlaps regardless of class:

import numpy as np
import supervision as sv

# Dummy binary masks: (N, H, W)

masks = np.random.randint(0, 2, size=(3, 640, 640), dtype=np.uint8)

# Predictions without class_id column triggers class-agnostic mode

preds = np.array([
    [50, 60, 150, 160, 0.92],
    [55, 65, 155, 165, 0.88],
    [400, 420, 460, 480, 0.80],
])

keep = sv.mask_non_max_suppression(preds, masks, iou_threshold=0.4)
print("Keep mask:", keep)

Using the IOS Metric for Small-Object Emphasis

To prioritize smaller boxes within overlapping regions, switch the overlap calculation:

import supervision as sv

keep_ios = sv.box_non_max_suppression(
    detections,
    iou_threshold=0.5,
    overlap_metric=sv.OverlapMetric.IOS,  # Intersection over Smaller

)

print("Kept indices with IOS:", np.where(keep_ios)[0])

Grid Search Threshold Optimization

Systematically evaluate performance across threshold values:

import numpy as np
import supervision as sv
from sklearn.metrics import precision_score, recall_score

def evaluate_threshold(thresh, preds, gt):
    keep = sv.box_non_max_suppression(preds, iou_threshold=thresh)
    selected = preds[keep]
    # Simplified matching logic for demonstration

    ious = sv.box_iou_batch(selected[:, :4], gt[:, :4])
    matches = (ious.max(axis=1) > 0.5).astype(int)
    prec = precision_score(np.ones_like(matches), matches)
    rec = recall_score(np.ones_like(matches), matches)
    return prec, rec

# Example ground truth

gt_boxes = np.array([[100, 120, 200, 220], [300, 400, 350, 450]])

for t in np.arange(0.2, 0.9, 0.1):
    p, r = evaluate_threshold(t, detections, gt_boxes)
    print(f"Threshold {t:.2f}: precision={p:.2f}, recall={r:.2f}")

Summary

Optimizing the NMS threshold in roboflow/supervision requires matching the IoU value to your specific detection scenario:

• Dense or small objects: Use 0.2–0.5 to maintain recall and prevent over-suppression of legitimate overlaps.
• Sparse or large objects: Use 0.6–0.8 to maximize precision by aggressively removing false duplicates.
• Implementation: Adjust the iou_threshold parameter in box_non_max_suppression or mask_non_max_suppression from src/supervision/detection/utils/iou_and_nms.py, ensuring values remain within [0, 1].
• Advanced options: Leverage overlap_metric="IOS" for nested objects and class-aware suppression for multi-class workflows.

Frequently Asked Questions

What happens if I set the NMS threshold too low or too high?

Setting the threshold below 0.3 preserves too many overlapping boxes, inflating false positives and reducing precision. Conversely, setting it above 0.8 causes aggressive pruning that eliminates true positives in crowded scenes, significantly degrading recall. The optimal range typically falls between 0.3 and 0.7 depending on object density.

Should I use different thresholds for box-based versus mask-based NMS?

Yes. Mask-based NMS generally requires lower thresholds (0.3–0.5) because mask IoU calculations are stricter than bounding box IoU—pixels must align exactly rather than just bounding box coordinates. Box-based NMS can tolerate higher thresholds (up to 0.7) for sparse scenes, as boxes offer more spatial forgiveness.

How does class-aware NMS differ from class-agnostic NMS in Supervision?

Class-aware NMS (the default when a class column is present) only suppresses detections belonging to the same category, preventing a "person" detection from eliminating a "car" detection at the same coordinates. Class-agnostic NMS (triggered by omitting the class column) suppresses all overlapping detections regardless of category, useful for merging model outputs or when treating all objects as generic targets.

When should I use IoU versus IoS as the overlap metric?

Use IoU (Intersection over Union) for standard suppression when boxes should not overlap significantly. Use IoS (Intersection over Smaller) when detecting small objects inside larger ones—such as faces within bodies or logos on products—as it favors retaining the smaller box even when it represents only a fraction of the larger box's area.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}