# How to Optimize the Non-Maximum Suppression (NMS) Threshold for Specific Detection Scenarios

> Learn how to optimize the NMS threshold for object detection. Adjust iou_threshold for dense crowds, sparse scenes, or small objects to improve your model's performance. Find the right settings for your specific detection scena...

- Repository: [Roboflow/supervision](https://github.com/roboflow/supervision)
- Tags: deep-dive
- Published: 2026-04-06

---

**Optimize your NMS threshold by selecting 0.3–0.5 for dense crowds, 0.5–0.7 for sparse scenes, and 0.2–0.4 for small objects, using the `iou_threshold` parameter in `box_non_max_suppression` or `mask_non_max_suppression` from [`src/supervision/detection/utils/iou_and_nms.py`](https://github.com/roboflow/supervision/blob/main/src/supervision/detection/utils/iou_and_nms.py).**

Non-Maximum Suppression (NMS) is the critical post-processing step that eliminates duplicate detections by retaining only the highest-scoring prediction when bounding boxes or masks overlap beyond a configurable **IoU (Intersection-over-Union) threshold**. In the Roboflow `supervision` library, optimizing this threshold allows you to balance precision and recall based on whether you are detecting densely packed crowds, isolated objects, or small visual targets. Understanding how to adjust the `iou_threshold` parameter for your specific detection scenario ensures you maximize model performance without manual trial and error.

## How NMS Works in Supervision

The library implements NMS in [`src/supervision/detection/utils/iou_and_nms.py`](https://github.com/roboflow/supervision/blob/main/src/supervision/detection/utils/iou_and_nms.py) through two primary functions that process predictions after model inference.

### Box-based NMS

The `box_non_max_suppression` function operates on NumPy arrays of bounding box coordinates. It first validates that the provided `iou_threshold` falls within the closed range `[0, 1]`, then sorts all predictions by confidence score in descending order. For each detection, it computes pairwise Intersection-over-Union using `box_iou_batch` and discards any subsequent prediction whose IoU with a higher-scoring box exceeds the threshold.

### Mask-based NMS

The `mask_non_max_suppression` function applies the same logic to segmentation masks, first resizing them to a common dimension (default `640`) to ensure consistent comparison. After validating the IoU range, the function iterates through sorted predictions and removes any mask whose overlap with a higher-confidence detection exceeds the threshold while belonging to the same category.

Both functions support an optional `overlap_metric` parameter, accepting `IOU` or `IOS` (Intersection over Smaller), and handle class-aware suppression automatically when the input includes a class column. Omitting the class column triggers class-agnostic mode, allowing cross-class overlap handling.

## NMS Threshold Recommendations by Detection Scenario

Choosing the optimal threshold requires matching the value to your scene's spatial density and object characteristics. The following guidelines help you optimize the Non-Maximum Suppression threshold for specific detection scenarios:

| Scenario | Recommended IoU Threshold | Rationale |
|----------|---------------------------|-----------|
| **Dense crowds / overlapping objects** (e.g., people in queues, vehicles in traffic) | **0.3 – 0.5** | Lower thresholds prevent legitimate nearby detections from being discarded as duplicates. |
| **Sparse scenes / well-separated objects** (e.g., isolated wildlife, traffic signs) | **0.5 – 0.7** | Higher thresholds aggressively remove spurious duplicates while preserving true positives. |
| **Small objects** (e.g., traffic lights, facial landmarks) | **0.2 – 0.4** | Small boxes often overlap tightly; lower thresholds retain all valid detections. |
| **Large objects** (e.g., vehicles, buildings) | **0.6 – 0.8** | Large boxes rarely overlap incorrectly; higher thresholds reduce false positives without sacrificing recall. |
| **Class-agnostic merging** (e.g., ensemble model outputs) | **0.4 – 0.6** | Balances cross-class overlap tolerance while suppressing obvious duplicates. |
| **Mask-based segmentation** (detailed instance masks) | **0.3 – 0.5** | Mask IoU tends to be stricter than box IoU, requiring modest thresholds to avoid over-pruning. |

## Practical Tips for Fine-Tuning NMS

When implementing NMS in production pipelines, consider these optimization strategies:

- **Start with 0.5**: The default threshold works well for generic COCO-style datasets and provides a baseline for comparison.

- **Validate empirically**: Run precision-recall curves on a representative validation subset, varying the threshold in `0.05` increments to identify the inflection point for your metric of choice.

- **Match metrics to thresholds**: If optimizing for **mAP@0.5**, use a slightly higher threshold (0.55–0.65) to boost precision. For **mAP@[0.5:0.95]**, maintain a balanced threshold around `0.5`.

- **Use IOS for nested objects**: Set `overlap_metric=sv.OverlapMetric.IOS` when detecting small objects inside larger ones, as this metric favors retaining the smaller box.

- **Leverage class-aware mode**: Ensure your predictions array includes a class column to prevent distinct classes from suppressing each other, which is essential for multi-class detection workflows.

- **Resize masks intelligently**: Adjust the `mask_dimension` parameter (default `640`) to balance computational speed against mask fidelity before running `mask_non_max_suppression`.

## Implementing Custom NMS Thresholds in Code

The following examples demonstrate how to apply these optimization strategies using the `supervision` Python package.

### Basic Box NMS with Custom Threshold

For crowded scenes, use a lower threshold to preserve overlapping detections:

```python
import numpy as np
import supervision as sv

# Format: (x_min, y_min, x_max, y_max, confidence, class_id)

detections = np.array([
    [100, 120, 200, 220, 0.95, 0],   # person

    [105, 125, 195, 215, 0.90, 0],   # overlapping person

    [300, 400, 350, 450, 0.80, 1],   # car

])

# Low threshold for crowded scenes (0.35)

keep = sv.box_non_max_suppression(detections, iou_threshold=0.35)

print("Indices to keep:", np.where(keep)[0])

# → Keeps highest-scoring person and the car

```

### Mask NMS with Class-Agnostic Mode

When merging detections across categories, omit the class column to suppress overlaps regardless of class:

```python
import numpy as np
import supervision as sv

# Dummy binary masks: (N, H, W)

masks = np.random.randint(0, 2, size=(3, 640, 640), dtype=np.uint8)

# Predictions without class_id column triggers class-agnostic mode

preds = np.array([
    [50, 60, 150, 160, 0.92],
    [55, 65, 155, 165, 0.88],
    [400, 420, 460, 480, 0.80],
])

keep = sv.mask_non_max_suppression(preds, masks, iou_threshold=0.4)
print("Keep mask:", keep)

```

### Using the IOS Metric for Small-Object Emphasis

To prioritize smaller boxes within overlapping regions, switch the overlap calculation:

```python
import supervision as sv

keep_ios = sv.box_non_max_suppression(
    detections,
    iou_threshold=0.5,
    overlap_metric=sv.OverlapMetric.IOS,  # Intersection over Smaller

)

print("Kept indices with IOS:", np.where(keep_ios)[0])

```

### Grid Search Threshold Optimization

Systematically evaluate performance across threshold values:

```python
import numpy as np
import supervision as sv
from sklearn.metrics import precision_score, recall_score

def evaluate_threshold(thresh, preds, gt):
    keep = sv.box_non_max_suppression(preds, iou_threshold=thresh)
    selected = preds[keep]
    # Simplified matching logic for demonstration

    ious = sv.box_iou_batch(selected[:, :4], gt[:, :4])
    matches = (ious.max(axis=1) > 0.5).astype(int)
    prec = precision_score(np.ones_like(matches), matches)
    rec = recall_score(np.ones_like(matches), matches)
    return prec, rec

# Example ground truth

gt_boxes = np.array([[100, 120, 200, 220], [300, 400, 350, 450]])

for t in np.arange(0.2, 0.9, 0.1):
    p, r = evaluate_threshold(t, detections, gt_boxes)
    print(f"Threshold {t:.2f}: precision={p:.2f}, recall={r:.2f}")

```

## Summary

Optimizing the NMS threshold in `roboflow/supervision` requires matching the IoU value to your specific detection scenario:

- **Dense or small objects**: Use **0.2–0.5** to maintain recall and prevent over-suppression of legitimate overlaps.
- **Sparse or large objects**: Use **0.6–0.8** to maximize precision by aggressively removing false duplicates.
- **Implementation**: Adjust the `iou_threshold` parameter in `box_non_max_suppression` or `mask_non_max_suppression` from [`src/supervision/detection/utils/iou_and_nms.py`](https://github.com/roboflow/supervision/blob/main/src/supervision/detection/utils/iou_and_nms.py), ensuring values remain within `[0, 1]`.
- **Advanced options**: Leverage `overlap_metric="IOS"` for nested objects and class-aware suppression for multi-class workflows.

## Frequently Asked Questions

### What happens if I set the NMS threshold too low or too high?

Setting the threshold **below 0.3** preserves too many overlapping boxes, inflating false positives and reducing precision. Conversely, setting it **above 0.8** causes aggressive pruning that eliminates true positives in crowded scenes, significantly degrading recall. The optimal range typically falls between 0.3 and 0.7 depending on object density.

### Should I use different thresholds for box-based versus mask-based NMS?

Yes. **Mask-based NMS** generally requires lower thresholds (0.3–0.5) because mask IoU calculations are stricter than bounding box IoU—pixels must align exactly rather than just bounding box coordinates. **Box-based NMS** can tolerate higher thresholds (up to 0.7) for sparse scenes, as boxes offer more spatial forgiveness.

### How does class-aware NMS differ from class-agnostic NMS in Supervision?

**Class-aware NMS** (the default when a class column is present) only suppresses detections belonging to the same category, preventing a "person" detection from eliminating a "car" detection at the same coordinates. **Class-agnostic NMS** (triggered by omitting the class column) suppresses all overlapping detections regardless of category, useful for merging model outputs or when treating all objects as generic targets.

### When should I use IoU versus IoS as the overlap metric?

Use **IoU** (Intersection over Union) for standard suppression when boxes should not overlap significantly. Use **IoS** (Intersection over Smaller) when detecting small objects inside larger ones—such as faces within bodies or logos on products—as it favors retaining the smaller box even when it represents only a fraction of the larger box's area.