# How to Perform Box-Based Segmentation with SAM: A Complete Technical Guide

> Learn how to perform box-based segmentation with SAM. This technical guide shows you how to convert bounding box prompts into precise object masks using SAM's robust architecture.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: how-to-guide
- Published: 2026-03-07

---

**Box-based segmentation with SAM converts rectangular bounding box prompts into precise object masks by encoding the box coordinates as sparse embeddings and processing them through the Mask Decoder.**

The Segment Anything Model (SAM) from the `facebookresearch/segment-anything` repository supports multiple prompt types, including rectangular bounding boxes. This guide explains the complete pipeline for performing box-based segmentation with SAM, from coordinate transformation to mask generation.

## Understanding the Box-Based Segmentation Pipeline

Box-based segmentation follows the same high-level architecture as point-based prompting, but processes rectangular regions instead of individual coordinates.

### Image Preprocessing and Coordinate Transformation

Before processing, SAM resizes the input image so its longest side matches the model's expected input size (typically 1024 pixels). This transformation is handled by **`ResizeLongestSide`** in **[`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py)**.

The transformer provides methods to scale box coordinates from the original image space to the resized model space. Boxes are specified in **XYXY format** (x-min, y-min, x-max, y-max) in the original image coordinate system.

### Encoding Box Prompts as Sparse Embeddings

The transformed box is passed to the **`PromptEncoder`** in **[`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py)**. Inside the **`_embed_boxes`** method, SAM applies a 0.5-pixel offset, reshapes the box into its two corner points, and injects learned corner embeddings (`self.point_embeddings[2]` and `self.point_embeddings[3]`) into the positional encoding.

These **sparse embeddings** represent the box prompt and are fed into the Mask Decoder to condition the segmentation output.

### Mask Decoding and Post-Processing

The **`Sam`** class in **[`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py)** orchestrates the forward pass. It combines the sparse box embeddings with dense image embeddings from the Vision Transformer encoder via the **`MaskDecoder`** in **[`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py)**.

The decoder predicts low-resolution mask logits (256×256), which are then up-sampled and cropped back to the original image resolution using **`postprocess_masks`**. By default, SAM thresholds the up-sampled masks with `mask_threshold` (0.0 in the reference implementation) to produce binary segmentation masks.

## Implementing Box-Based Segmentation with SamPredictor

The **`SamPredictor`** class in **[`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py)** provides a high-level API that handles image setting, box scaling, and mask post-processing automatically.

```python
import torch
import numpy as np
from segment_anything import build_sam, SamPredictor

# 1. Build the SAM model using a pre-trained checkpoint

sam = build_sam(checkpoint="sam_vit_h_4b8939.pth")

# 2. Wrap it in the high-level predictor

predictor = SamPredictor(sam)

# 3. Load your image (H × W × C, uint8)

image = np.array(...)  # Your image as a NumPy array

# 4. Set the image - computes the embedding once

predictor.set_image(image)

# 5. Define the box in original image coordinates (XYXY format)

# Example: top-left (50, 30), bottom-right (400, 350)

box = np.array([50, 30, 400, 350], dtype=np.float32)

# 6. Run prediction with box prompt only

masks, scores, low_res = predictor.predict(box=box)

# 7. masks is a binary array of shape (C, H, W) where C is 1 or 3

# depending on multimask_output setting

```

## Low-Level Implementation: Direct Model Inference

For batch processing or custom workflows, you can interact directly with the model components without the `SamPredictor` wrapper.

```python
from segment_anything.modeling.sam import Sam
from segment_anything.utils.transforms import ResizeLongestSide
import torch.nn.functional as F

# Assume sam is an initialized Sam model and image is loaded

transform = ResizeLongestSide(sam.image_encoder.img_size)

# Preprocess image tensor (C × H × W, float)

input_image = torch.from_numpy(transform.apply_image(image)).permute(2, 0, 1).unsqueeze(0)
input_image = (input_image - sam.pixel_mean) / sam.pixel_std

# Pad to square input size

h, w = input_image.shape[-2:]
pad_h = sam.image_encoder.img_size - h
pad_w = sam.image_encoder.img_size - w
input_image = F.pad(input_image, (0, pad_w, 0, pad_h))

# Generate image embeddings

image_emb = sam.image_encoder(input_image)

# Transform box to model space

box_tensor = torch.as_tensor(
    transform.apply_boxes(box[None, :], image.shape[:2]), 
    dtype=torch.float
)

# Encode prompts

sparse_embed, dense_embed = sam.prompt_encoder(
    points=None, 
    boxes=box_tensor, 
    masks=None
)

# Decode masks

low_res_masks, iou_pred = sam.mask_decoder(
    image_embeddings=image_emb,
    image_pe=sam.prompt_encoder.get_dense_pe(),
    sparse_prompt_embeddings=sparse_embed,
    dense_prompt_embeddings=dense_embed,
    multimask_output=False,
)

# Post-process to original resolution

masks = sam.postprocess_masks(low_res_masks, input_image.shape[-2:], image.shape[:2])
binary_mask = (masks > sam.mask_threshold).cpu().numpy()

```

## Key Source Files and Components

Understanding these source files helps debug and extend box-based segmentation workflows:

- **[`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py)** – Contains `SamPredictor`, the high-level API that manages image embeddings, coordinate transformation, and the `predict` method accepting box arguments.

- **[`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py)** – Implements `ResizeLongestSide`, which handles image resizing and the critical coordinate mapping from original image space to model input space for box prompts.

- **[`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py)** – Houses `PromptEncoder` and the `_embed_boxes` method, which converts XYXY box coordinates into sparse embeddings using learned corner point embeddings.

- **[`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py)** – The core `Sam` class that orchestrates image encoding, prompt encoding, mask decoding, and post-processing through its forward pass and helper methods.

- **[`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py)** – Contains `MaskDecoder`, which generates low-resolution mask logits from combined image and box prompt embeddings.

- **[`segment_anything/build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py)** – Factory function for loading pre-trained model checkpoints (ViT-H, ViT-L, ViT-B variants).

## Summary

- **Box-based segmentation with SAM** uses rectangular prompts in XYXY format to generate object masks without requiring point annotations.
- The **`SamPredictor`** class in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) provides the simplest API, handling coordinate transformation and mask post-processing automatically.
- Under the hood, **`ResizeLongestSide`** scales box coordinates to model space, while **`_embed_boxes`** in the prompt encoder converts boxes into sparse embeddings using learned corner representations.
- For advanced use cases, the low-level **`Sam`** API allows direct control over batch processing, custom preprocessing, and embedding caching strategies.

## Frequently Asked Questions

### What format should box coordinates be in for SAM?

SAM expects box coordinates in **XYXY format** (x-min, y-min, x-max, y-max) relative to the original image dimensions. The `SamPredictor` automatically handles scaling these coordinates to the model's input space using `ResizeLongestSide`, while low-level implementations require manual transformation via `apply_boxes`.

### Can I use multiple boxes in a single prediction?

Yes, the low-level `Sam` model supports batch processing of multiple boxes by passing a tensor of shape **(N, 4)** to the `prompt_encoder`. However, the high-level `SamPredictor.predict()` method is designed for single-image, single-prompt inference; for multiple boxes on the same image, you should either loop through boxes with the predictor or use the low-level API with batched embeddings.

### How does SAM convert a box into embeddings?

Inside **[`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py)**, the `_embed_boxes` method applies a 0.5-pixel offset to the input box, reshapes it into two corner points, and maps these to learned positional embeddings `point_embeddings[2]` and `point_embeddings[3]`. These corner embeddings become the **sparse prompt embeddings** that condition the mask decoder.

### What is the difference between using SamPredictor and the low-level Sam API?

**`SamPredictor`** ([`predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/predictor.py)) provides a stateful, high-level interface that manages image embeddings, coordinate transformations, and result post-processing automatically. It is ideal for interactive applications and single-image inference. The low-level **`Sam`** API ([`modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/modeling/sam.py)) exposes the raw forward pass, enabling custom batch processing, manual control over tensor operations, and integration into larger pipelines where you manage embeddings and coordinate scaling yourself.