How to Perform Box-Based Segmentation with SAM: A Complete Technical Guide

Box-based segmentation with SAM converts rectangular bounding box prompts into precise object masks by encoding the box coordinates as sparse embeddings and processing them through the Mask Decoder.

The Segment Anything Model (SAM) from the facebookresearch/segment-anything repository supports multiple prompt types, including rectangular bounding boxes. This guide explains the complete pipeline for performing box-based segmentation with SAM, from coordinate transformation to mask generation.

Understanding the Box-Based Segmentation Pipeline

Box-based segmentation follows the same high-level architecture as point-based prompting, but processes rectangular regions instead of individual coordinates.

Image Preprocessing and Coordinate Transformation

Before processing, SAM resizes the input image so its longest side matches the model's expected input size (typically 1024 pixels). This transformation is handled by ResizeLongestSide in segment_anything/utils/transforms.py.

The transformer provides methods to scale box coordinates from the original image space to the resized model space. Boxes are specified in XYXY format (x-min, y-min, x-max, y-max) in the original image coordinate system.

Encoding Box Prompts as Sparse Embeddings

The transformed box is passed to the PromptEncoder in segment_anything/modeling/prompt_encoder.py. Inside the _embed_boxes method, SAM applies a 0.5-pixel offset, reshapes the box into its two corner points, and injects learned corner embeddings (self.point_embeddings[2] and self.point_embeddings[3]) into the positional encoding.

These sparse embeddings represent the box prompt and are fed into the Mask Decoder to condition the segmentation output.

Mask Decoding and Post-Processing

The Sam class in segment_anything/modeling/sam.py orchestrates the forward pass. It combines the sparse box embeddings with dense image embeddings from the Vision Transformer encoder via the MaskDecoder in segment_anything/modeling/mask_decoder.py.

The decoder predicts low-resolution mask logits (256×256), which are then up-sampled and cropped back to the original image resolution using postprocess_masks. By default, SAM thresholds the up-sampled masks with mask_threshold (0.0 in the reference implementation) to produce binary segmentation masks.

Implementing Box-Based Segmentation with SamPredictor

The SamPredictor class in segment_anything/predictor.py provides a high-level API that handles image setting, box scaling, and mask post-processing automatically.

import torch
import numpy as np
from segment_anything import build_sam, SamPredictor

# 1. Build the SAM model using a pre-trained checkpoint

sam = build_sam(checkpoint="sam_vit_h_4b8939.pth")

# 2. Wrap it in the high-level predictor

predictor = SamPredictor(sam)

# 3. Load your image (H × W × C, uint8)

image = np.array(...)  # Your image as a NumPy array

# 4. Set the image - computes the embedding once

predictor.set_image(image)

# 5. Define the box in original image coordinates (XYXY format)

# Example: top-left (50, 30), bottom-right (400, 350)

box = np.array([50, 30, 400, 350], dtype=np.float32)

# 6. Run prediction with box prompt only

masks, scores, low_res = predictor.predict(box=box)

# 7. masks is a binary array of shape (C, H, W) where C is 1 or 3

# depending on multimask_output setting

Low-Level Implementation: Direct Model Inference

For batch processing or custom workflows, you can interact directly with the model components without the SamPredictor wrapper.

from segment_anything.modeling.sam import Sam
from segment_anything.utils.transforms import ResizeLongestSide
import torch.nn.functional as F

# Assume sam is an initialized Sam model and image is loaded

transform = ResizeLongestSide(sam.image_encoder.img_size)

# Preprocess image tensor (C × H × W, float)

input_image = torch.from_numpy(transform.apply_image(image)).permute(2, 0, 1).unsqueeze(0)
input_image = (input_image - sam.pixel_mean) / sam.pixel_std

# Pad to square input size

h, w = input_image.shape[-2:]
pad_h = sam.image_encoder.img_size - h
pad_w = sam.image_encoder.img_size - w
input_image = F.pad(input_image, (0, pad_w, 0, pad_h))

# Generate image embeddings

image_emb = sam.image_encoder(input_image)

# Transform box to model space

box_tensor = torch.as_tensor(
    transform.apply_boxes(box[None, :], image.shape[:2]), 
    dtype=torch.float
)

# Encode prompts

sparse_embed, dense_embed = sam.prompt_encoder(
    points=None, 
    boxes=box_tensor, 
    masks=None
)

# Decode masks

low_res_masks, iou_pred = sam.mask_decoder(
    image_embeddings=image_emb,
    image_pe=sam.prompt_encoder.get_dense_pe(),
    sparse_prompt_embeddings=sparse_embed,
    dense_prompt_embeddings=dense_embed,
    multimask_output=False,
)

# Post-process to original resolution

masks = sam.postprocess_masks(low_res_masks, input_image.shape[-2:], image.shape[:2])
binary_mask = (masks > sam.mask_threshold).cpu().numpy()

Key Source Files and Components

Understanding these source files helps debug and extend box-based segmentation workflows:

• segment_anything/predictor.py – Contains SamPredictor, the high-level API that manages image embeddings, coordinate transformation, and the predict method accepting box arguments.

• segment_anything/utils/transforms.py – Implements ResizeLongestSide, which handles image resizing and the critical coordinate mapping from original image space to model input space for box prompts.

• segment_anything/modeling/prompt_encoder.py – Houses PromptEncoder and the _embed_boxes method, which converts XYXY box coordinates into sparse embeddings using learned corner point embeddings.

• segment_anything/modeling/sam.py – The core Sam class that orchestrates image encoding, prompt encoding, mask decoding, and post-processing through its forward pass and helper methods.

• segment_anything/modeling/mask_decoder.py – Contains MaskDecoder, which generates low-resolution mask logits from combined image and box prompt embeddings.

• segment_anything/build_sam.py – Factory function for loading pre-trained model checkpoints (ViT-H, ViT-L, ViT-B variants).

Summary

• Box-based segmentation with SAM uses rectangular prompts in XYXY format to generate object masks without requiring point annotations.
• The SamPredictor class in segment_anything/predictor.py provides the simplest API, handling coordinate transformation and mask post-processing automatically.
• Under the hood, ResizeLongestSide scales box coordinates to model space, while _embed_boxes in the prompt encoder converts boxes into sparse embeddings using learned corner representations.
• For advanced use cases, the low-level Sam API allows direct control over batch processing, custom preprocessing, and embedding caching strategies.

Frequently Asked Questions

What format should box coordinates be in for SAM?

SAM expects box coordinates in XYXY format (x-min, y-min, x-max, y-max) relative to the original image dimensions. The SamPredictor automatically handles scaling these coordinates to the model's input space using ResizeLongestSide, while low-level implementations require manual transformation via apply_boxes.

Can I use multiple boxes in a single prediction?

Yes, the low-level Sam model supports batch processing of multiple boxes by passing a tensor of shape (N, 4) to the prompt_encoder. However, the high-level SamPredictor.predict() method is designed for single-image, single-prompt inference; for multiple boxes on the same image, you should either loop through boxes with the predictor or use the low-level API with batched embeddings.

How does SAM convert a box into embeddings?

Inside segment_anything/modeling/prompt_encoder.py, the _embed_boxes method applies a 0.5-pixel offset to the input box, reshapes it into two corner points, and maps these to learned positional embeddings point_embeddings[2] and point_embeddings[3]. These corner embeddings become the sparse prompt embeddings that condition the mask decoder.

What is the difference between using SamPredictor and the low-level Sam API?

SamPredictor (predictor.py) provides a stateful, high-level interface that manages image embeddings, coordinate transformations, and result post-processing automatically. It is ideal for interactive applications and single-image inference. The low-level Sam API (modeling/sam.py) exposes the raw forward pass, enabling custom batch processing, manual control over tensor operations, and integration into larger pipelines where you manage embeddings and coordinate scaling yourself.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}