How to Encode Point and Box Prompts for SAM: Implementation Guide

Point and box prompts for the Segment Anything Model (SAM) are encoded by the PromptEncoder class in segment_anything/modeling/prompt_encoder.py, which converts normalized coordinates into high-dimensional embeddings using random positional encoding and adds learned vectors specific to foreground points, background points, and box corners.

The facebookresearch/segment-anything repository processes spatial prompts through a dedicated encoding pipeline before they reach the mask decoder. Understanding how to encode prompts (points, boxes) for SAM enables precise control over segmentation behavior and facilitates custom batch inference implementations. This article examines the internal mechanics of prompt encoding and provides runnable code examples for both low-level and high-level APIs.

How the PromptEncoder Initializes Embedding Components

The PromptEncoder class serves as the central processing unit for all spatial prompts. According to the source code in segment_anything/modeling/prompt_encoder.py (lines 16‑48), the encoder initializes four critical components during construction:

• PositionEmbeddingRandom – A sinusoidal positional encoder that maps 2D coordinates to a vector of size embed_dim // 2, which is then concatenated to form the full embed_dim representation.
• point_embeddings – Four distinct nn.Embedding tables (indices 0‑3) that provide learned additive vectors for background points, foreground points, and the two corners of a bounding box.
• not_a_point_embed – A special embedding reserved for padding tokens (label -1) that indicates the absence of a valid point.
• Mask downsampler – A convolutional network for processing mask prompts (not covered in this guide).

from segment_anything.modeling.prompt_encoder import PromptEncoder

# Initialization parameters match the image encoder output dimensions

encoder = PromptEncoder(
    embed_dim=256,
    image_embedding_size=(64, 64),  # H, W of the image embedding

    input_image_size=(1024, 1024),  # Original padded image size

    mask_in_chans=16
)

Encoding Point Prompts in SAM

Point prompts are processed by the _embed_points method (lines 73‑92 in prompt_encoder.py). This function accepts coordinates in pixel space and transforms them into dense embeddings through a multi-step normalization pipeline.

Input specifications:

• Coordinates: Tensor of shape (B, N, 2) containing (x, y) pixel locations.
• Labels: Tensor of shape (B, N) where 1 indicates foreground, 0 indicates background, and -1 indicates padding ("not a point").

The encoding pipeline executes the following transformations:

1. Pixel centering: Adds 0.5 to coordinates to align with pixel centers (points + 0.5).
2. Conditional padding: If no box prompt is provided, appends a dummy point [0, 0] with label -1 to ensure at least one point exists for the decoder.
3. Positional encoding: Normalizes coordinates to [0, 1] using input_image_size and applies PositionEmbeddingRandom.forward_with_coords.
4. Label-specific embedding addition:
   • Background points (label == 0) add self.point_embeddings[0].weight
   • Foreground points (label == 1) add self.point_embeddings[1].weight
   • Padding points (label == -1) receive self.not_a_point_embed.weight (and have their positional embedding zeroed)

The output is a tensor of shape (B, N, embed_dim) representing the sparse embeddings consumed by the mask decoder.

def encode_points_example(encoder):
    import torch
    
    # Batch size 1, 2 points: one foreground, one background

    point_coords = torch.tensor([[[200.0, 300.0], [400.0, 500.0]]])  # (1, 2, 2)

    point_labels = torch.tensor([[1, 0]])  # 1=foreground, 0=background

    
    # The _embed_points method is called internally by forward()

    sparse_embeddings, _ = encoder(
        points=(point_coords, point_labels),
        boxes=None,
        masks=None
    )
    
    return sparse_embeddings  # Shape: (1, 2, 256)

Encoding Box Prompts in SAM

Box prompts are treated as two corner points and processed by _embed_boxes (lines 93‑100). A bounding box tensor of shape (B, 4) in [x1, y1, x2, y2] format undergoes the following transformation:

1. Pixel centering: Adds 0.5 to all coordinates (boxes + 0.5).
2. Reshaping: Converts (B, 4) to (B, 2, 2) to represent two corner points per box.
3. Positional encoding: Applies the same forward_with_coords normalization used for points.
4. Corner-specific embeddings:
   • The first corner (top-left) adds self.point_embeddings[2].weight
   • The second corner (bottom-right) adds self.point_embeddings[3].weight

The resulting tensor has shape (B, 2, embed_dim) and is concatenated with point embeddings to form the complete sparse prompt representation.

def encode_boxes_example(encoder):
    import torch
    
    # Single box: [x1, y1, x2, y2]

    boxes = torch.tensor([[150.0, 250.0, 600.0, 800.0]])  # (1, 4)

    
    sparse_embeddings, _ = encoder(
        points=None,
        boxes=boxes,
        masks=None
    )
    
    return sparse_embeddings  # Shape: (1, 2, 256)

Integrating Prompts into the SAM Forward Pass

During inference, the Sam class in segment_anything/modeling/sam.py (lines 100‑112) orchestrates the prompt encoding workflow. The model accepts a list of input dictionaries containing raw coordinates, then delegates to PromptEncoder to produce sparse and dense embeddings.

The data flow follows this contract:

• Input dictionary keys: "point_coords" (optional), "point_labels" (optional), "boxes" (optional), "mask_inputs" (optional).
• Sparse embeddings: Concatenation of all point and box embeddings (B, total_prompts, embed_dim).
• Dense embeddings: Encoded mask input or a learned "no-mask" token.

from segment_anything.modeling.sam import Sam

# Example batched input format expected by Sam.forward()

batched_input = [{
    "image": torch.randn(3, 1024, 1024),  # Preprocessed and padded

    "original_size": (720, 1280),
    "point_coords": torch.tensor([[[600.0, 350.0]]]),  # (B, N, 2)

    "point_labels": torch.tensor([[1]]),  # (B, N)

    "boxes": torch.tensor([[400.0, 200.0, 900.0, 600.0]]),  # (B, 4)

    "mask_inputs": None,
}]

# The forward pass internally calls:

# sparse_emb, dense_emb = self.prompt_encoder(points=..., boxes=..., masks=...)

output = sam(batched_input, multimask_output=False)

Practical Implementation Examples

Direct PromptEncoder Usage

For custom architectures requiring explicit control over the embedding process, instantiate PromptEncoder directly and supply normalized tensors.

import torch
from segment_anything.modeling.prompt_encoder import PromptEncoder

# Initialize with ViT-H dimensions

encoder = PromptEncoder(
    embed_dim=256,
    image_embedding_size=(64, 64),
    input_image_size=(1024, 1024),
    mask_in_chans=16
)

# Combined point and box prompts

point_coords = torch.tensor([[[200.0, 300.0], [400.0, 500.0]]])  # (1, 2, 2)

point_labels = torch.tensor([[1, 0]])
box = torch.tensor([[150.0, 250.0, 600.0, 800.0]])  # (1, 4)

sparse_embeddings, dense_embeddings = encoder(
    points=(point_coords, point_labels),
    boxes=box,
    masks=None
)

print(sparse_embeddings.shape)  # torch.Size([1, 4, 256]) -> 2 points + 2 box corners

High-Level SamPredictor API

The SamPredictor wrapper in segment_anything/predictor.py handles coordinate transformation and encoding automatically for single-image inference.

import numpy as np
from segment_anything import SamPredictor, sam_model_registry

# Load pretrained weights

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Set image once to cache image embeddings

image = np.random.randint(0, 255, (720, 1280, 3), dtype=np.uint8)
predictor.set_image(image)

# Define prompts in original image coordinates

point_coords = np.array([[600, 350]])  # Foreground click

point_labels = np.array([1])
box = np.array([400, 200, 900, 600])  # Bounding box

# The predictor internally encodes prompts via PromptEncoder

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    box=box,
    multimask_output=False
)

print(masks.shape)  # (1, 720, 1280) - binary mask at original resolution

Batch Processing with Sam.forward

For efficient batch inference across multiple images, use the low-level Sam.forward method with properly formatted dictionaries.

import torch

# Prepare multiple records

batched_input = [
    {
        "image": torch.randn(3, 1024, 1024),
        "original_size": (480, 640),
        "point_coords": torch.tensor([[[300.0, 200.0]]]),
        "point_labels": torch.tensor([[1]]),
        "boxes": None,
    },
    {
        "image": torch.randn(3, 1024, 1024),
        "original_size": (800, 1200),
        "point_coords": None,
        "point_labels": None,
        "boxes": torch.tensor([[100.0, 100.0, 500.0, 500.0]]),
    }
]

# Process batch (returns list of output dicts)

outputs = sam(batched_input, multimask_output=True)

# Each output contains low-resolution masks

print(outputs[0]["masks"].shape)  # (3, 256, 256) for multimask_output=True

Summary

• The PromptEncoder in segment_anything/modeling/prompt_encoder.py serves as the exclusive interface for converting spatial coordinates to learned embeddings.
• Point prompts use labels 1 (foreground), 0 (background), and -1 (padding), with specific embedding tables for each type plus a "not-a-point" token for padding.
• Box prompts are decomposed into two corner points and encoded using dedicated embedding tables at indices 2 and 3 of point_embeddings.
• All coordinates undergo a +0.5 shift to align with pixel centers and normalization to [0, 1] range before positional encoding.
• Sparse embeddings from points and boxes are concatenated and fed to the MaskDecoder alongside image features.

Frequently Asked Questions

What coordinate format does SAM expect for point prompts?

SAM expects point coordinates as floating-point tensors of shape (B, N, 2) representing (x, y) pixel locations in the original image coordinate system (not the padded 1024×1024 space). The SamPredictor class automatically handles the transformation to normalized coordinates, while low-level PromptEncoder usage requires manual normalization to [0, 1] range based on input_image_size.

How does SAM distinguish between foreground and background points?

The model distinguishes point types through the point_labels tensor provided alongside coordinates. A value of 1 triggers the addition of point_embeddings[1] (foreground), while 0 adds point_embeddings[0] (background). Internally, the _embed_points method uses these labels to index the correct learned embedding vector and add it to the positional encoding base.

Can I combine point and box prompts in a single prediction?

Yes, the PromptEncoder concatenates point and box embeddings automatically when both are provided. The resulting sparse embedding tensor contains the point embeddings followed by the two box corner embeddings, producing a shape of (B, N_points + 2, embed_dim). Both the SamPredictor.predict() method and the low-level encoder() call accept simultaneous points and boxes arguments.

Why does SAM add 0.5 to coordinates during encoding?

The +0.5 offset in _embed_points and _embed_boxes shifts coordinates from the top-left corner of pixels to their centers, ensuring that integer pixel indices map to the spatial middle of the respective pixel. This alignment improves the accuracy of the random positional encoding, which expects coordinates to represent continuous spatial positions rather than discrete grid corners.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}