# How to Encode Point and Box Prompts for SAM: Implementation Guide

> Learn how to encode point and box prompts for SAM. This guide details the PromptEncoder class, converting coordinates into high-dimensional embeddings for efficient segmentation.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: how-to-guide
- Published: 2026-03-07

---

**Point and box prompts for the Segment Anything Model (SAM) are encoded by the `PromptEncoder` class in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py), which converts normalized coordinates into high-dimensional embeddings using random positional encoding and adds learned vectors specific to foreground points, background points, and box corners.**

The `facebookresearch/segment-anything` repository processes spatial prompts through a dedicated encoding pipeline before they reach the mask decoder. Understanding how to encode prompts (points, boxes) for SAM enables precise control over segmentation behavior and facilitates custom batch inference implementations. This article examines the internal mechanics of prompt encoding and provides runnable code examples for both low-level and high-level APIs.

## How the PromptEncoder Initializes Embedding Components

The `PromptEncoder` class serves as the central processing unit for all spatial prompts. According to the source code in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) (lines 16‑48), the encoder initializes four critical components during construction:

*   **`PositionEmbeddingRandom`** – A sinusoidal positional encoder that maps 2D coordinates to a vector of size `embed_dim // 2`, which is then concatenated to form the full `embed_dim` representation.
*   **`point_embeddings`** – Four distinct `nn.Embedding` tables (indices 0‑3) that provide learned additive vectors for background points, foreground points, and the two corners of a bounding box.
*   **`not_a_point_embed`** – A special embedding reserved for padding tokens (label `-1`) that indicates the absence of a valid point.
*   **Mask downsampler** – A convolutional network for processing mask prompts (not covered in this guide).

```python
from segment_anything.modeling.prompt_encoder import PromptEncoder

# Initialization parameters match the image encoder output dimensions

encoder = PromptEncoder(
    embed_dim=256,
    image_embedding_size=(64, 64),  # H, W of the image embedding

    input_image_size=(1024, 1024),  # Original padded image size

    mask_in_chans=16
)

```

## Encoding Point Prompts in SAM

Point prompts are processed by the `_embed_points` method (lines 73‑92 in [`prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/prompt_encoder.py)). This function accepts coordinates in pixel space and transforms them into dense embeddings through a multi-step normalization pipeline.

**Input specifications:**
*   **Coordinates**: Tensor of shape `(B, N, 2)` containing `(x, y)` pixel locations.
*   **Labels**: Tensor of shape `(B, N)` where `1` indicates **foreground**, `0` indicates **background**, and `-1` indicates **padding** ("not a point").

The encoding pipeline executes the following transformations:

1.  **Pixel centering**: Adds `0.5` to coordinates to align with pixel centers (`points + 0.5`).
2.  **Conditional padding**: If no box prompt is provided, appends a dummy point `[0, 0]` with label `-1` to ensure at least one point exists for the decoder.
3.  **Positional encoding**: Normalizes coordinates to `[0, 1]` using `input_image_size` and applies `PositionEmbeddingRandom.forward_with_coords`.
4.  **Label-specific embedding addition**:
    *   Background points (`label == 0`) add `self.point_embeddings[0].weight`
    *   Foreground points (`label == 1`) add `self.point_embeddings[1].weight`
    *   Padding points (`label == -1`) receive `self.not_a_point_embed.weight` (and have their positional embedding zeroed)

The output is a tensor of shape `(B, N, embed_dim)` representing the **sparse embeddings** consumed by the mask decoder.

```python
def encode_points_example(encoder):
    import torch
    
    # Batch size 1, 2 points: one foreground, one background

    point_coords = torch.tensor([[[200.0, 300.0], [400.0, 500.0]]])  # (1, 2, 2)

    point_labels = torch.tensor([[1, 0]])  # 1=foreground, 0=background

    
    # The _embed_points method is called internally by forward()

    sparse_embeddings, _ = encoder(
        points=(point_coords, point_labels),
        boxes=None,
        masks=None
    )
    
    return sparse_embeddings  # Shape: (1, 2, 256)

```

## Encoding Box Prompts in SAM

Box prompts are treated as two corner points and processed by `_embed_boxes` (lines 93‑100). A bounding box tensor of shape `(B, 4)` in `[x1, y1, x2, y2]` format undergoes the following transformation:

1.  **Pixel centering**: Adds `0.5` to all coordinates (`boxes + 0.5`).
2.  **Reshaping**: Converts `(B, 4)` to `(B, 2, 2)` to represent two corner points per box.
3.  **Positional encoding**: Applies the same `forward_with_coords` normalization used for points.
4.  **Corner-specific embeddings**:
    *   The first corner (top-left) adds `self.point_embeddings[2].weight`
    *   The second corner (bottom-right) adds `self.point_embeddings[3].weight`

The resulting tensor has shape `(B, 2, embed_dim)` and is concatenated with point embeddings to form the complete sparse prompt representation.

```python
def encode_boxes_example(encoder):
    import torch
    
    # Single box: [x1, y1, x2, y2]

    boxes = torch.tensor([[150.0, 250.0, 600.0, 800.0]])  # (1, 4)

    
    sparse_embeddings, _ = encoder(
        points=None,
        boxes=boxes,
        masks=None
    )
    
    return sparse_embeddings  # Shape: (1, 2, 256)

```

## Integrating Prompts into the SAM Forward Pass

During inference, the `Sam` class in [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) (lines 100‑112) orchestrates the prompt encoding workflow. The model accepts a list of input dictionaries containing raw coordinates, then delegates to `PromptEncoder` to produce sparse and dense embeddings.

**The data flow follows this contract:**

*   **Input dictionary keys**: `"point_coords"` (optional), `"point_labels"` (optional), `"boxes"` (optional), `"mask_inputs"` (optional).
*   **Sparse embeddings**: Concatenation of all point and box embeddings `(B, total_prompts, embed_dim)`.
*   **Dense embeddings**: Encoded mask input or a learned "no-mask" token.

```python
from segment_anything.modeling.sam import Sam

# Example batched input format expected by Sam.forward()

batched_input = [{
    "image": torch.randn(3, 1024, 1024),  # Preprocessed and padded

    "original_size": (720, 1280),
    "point_coords": torch.tensor([[[600.0, 350.0]]]),  # (B, N, 2)

    "point_labels": torch.tensor([[1]]),  # (B, N)

    "boxes": torch.tensor([[400.0, 200.0, 900.0, 600.0]]),  # (B, 4)

    "mask_inputs": None,
}]

# The forward pass internally calls:

# sparse_emb, dense_emb = self.prompt_encoder(points=..., boxes=..., masks=...)

output = sam(batched_input, multimask_output=False)

```

## Practical Implementation Examples

### Direct PromptEncoder Usage

For custom architectures requiring explicit control over the embedding process, instantiate `PromptEncoder` directly and supply normalized tensors.

```python
import torch
from segment_anything.modeling.prompt_encoder import PromptEncoder

# Initialize with ViT-H dimensions

encoder = PromptEncoder(
    embed_dim=256,
    image_embedding_size=(64, 64),
    input_image_size=(1024, 1024),
    mask_in_chans=16
)

# Combined point and box prompts

point_coords = torch.tensor([[[200.0, 300.0], [400.0, 500.0]]])  # (1, 2, 2)

point_labels = torch.tensor([[1, 0]])
box = torch.tensor([[150.0, 250.0, 600.0, 800.0]])  # (1, 4)

sparse_embeddings, dense_embeddings = encoder(
    points=(point_coords, point_labels),
    boxes=box,
    masks=None
)

print(sparse_embeddings.shape)  # torch.Size([1, 4, 256]) -> 2 points + 2 box corners

```

### High-Level SamPredictor API

The `SamPredictor` wrapper in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) handles coordinate transformation and encoding automatically for single-image inference.

```python
import numpy as np
from segment_anything import SamPredictor, sam_model_registry

# Load pretrained weights

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Set image once to cache image embeddings

image = np.random.randint(0, 255, (720, 1280, 3), dtype=np.uint8)
predictor.set_image(image)

# Define prompts in original image coordinates

point_coords = np.array([[600, 350]])  # Foreground click

point_labels = np.array([1])
box = np.array([400, 200, 900, 600])  # Bounding box

# The predictor internally encodes prompts via PromptEncoder

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    box=box,
    multimask_output=False
)

print(masks.shape)  # (1, 720, 1280) - binary mask at original resolution

```

### Batch Processing with Sam.forward

For efficient batch inference across multiple images, use the low-level `Sam.forward` method with properly formatted dictionaries.

```python
import torch

# Prepare multiple records

batched_input = [
    {
        "image": torch.randn(3, 1024, 1024),
        "original_size": (480, 640),
        "point_coords": torch.tensor([[[300.0, 200.0]]]),
        "point_labels": torch.tensor([[1]]),
        "boxes": None,
    },
    {
        "image": torch.randn(3, 1024, 1024),
        "original_size": (800, 1200),
        "point_coords": None,
        "point_labels": None,
        "boxes": torch.tensor([[100.0, 100.0, 500.0, 500.0]]),
    }
]

# Process batch (returns list of output dicts)

outputs = sam(batched_input, multimask_output=True)

# Each output contains low-resolution masks

print(outputs[0]["masks"].shape)  # (3, 256, 256) for multimask_output=True

```

## Summary

*   The `PromptEncoder` in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) serves as the exclusive interface for converting spatial coordinates to learned embeddings.
*   **Point prompts** use labels `1` (foreground), `0` (background), and `-1` (padding), with specific embedding tables for each type plus a "not-a-point" token for padding.
*   **Box prompts** are decomposed into two corner points and encoded using dedicated embedding tables at indices `2` and `3` of `point_embeddings`.
*   All coordinates undergo a `+0.5` shift to align with pixel centers and normalization to `[0, 1]` range before positional encoding.
*   Sparse embeddings from points and boxes are concatenated and fed to the `MaskDecoder` alongside image features.

## Frequently Asked Questions

### What coordinate format does SAM expect for point prompts?

SAM expects point coordinates as floating-point tensors of shape `(B, N, 2)` representing `(x, y)` pixel locations in the **original image coordinate system** (not the padded 1024×1024 space). The `SamPredictor` class automatically handles the transformation to normalized coordinates, while low-level `PromptEncoder` usage requires manual normalization to `[0, 1]` range based on `input_image_size`.

### How does SAM distinguish between foreground and background points?

The model distinguishes point types through the `point_labels` tensor provided alongside coordinates. A value of `1` triggers the addition of `point_embeddings[1]` (foreground), while `0` adds `point_embeddings[0]` (background). Internally, the `_embed_points` method uses these labels to index the correct learned embedding vector and add it to the positional encoding base.

### Can I combine point and box prompts in a single prediction?

Yes, the `PromptEncoder` concatenates point and box embeddings automatically when both are provided. The resulting sparse embedding tensor contains the point embeddings followed by the two box corner embeddings, producing a shape of `(B, N_points + 2, embed_dim)`. Both the `SamPredictor.predict()` method and the low-level `encoder()` call accept simultaneous `points` and `boxes` arguments.

### Why does SAM add 0.5 to coordinates during encoding?

The `+0.5` offset in `_embed_points` and `_embed_boxes` shifts coordinates from the top-left corner of pixels to their **centers**, ensuring that integer pixel indices map to the spatial middle of the respective pixel. This alignment improves the accuracy of the random positional encoding, which expects coordinates to represent continuous spatial positions rather than discrete grid corners.