# How to Perform Point-Based Segmentation with SAM: A Complete Guide

> Learn how to perform point-based segmentation with SAM. This guide explains encoding clicks, combining features, and decoding masks for precise image segmentation.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: how-to-guide
- Published: 2026-03-07

---

**Point-based segmentation with SAM works by encoding click coordinates into sparse embeddings via the `PromptEncoder`, combining them with pre-computed image features, and decoding them into binary masks through the `MaskDecoder`.**

The Segment Anything Model (SAM) from `facebookresearch/segment-anything` enables precise object segmentation through interactive point prompts. This guide explains the complete workflow for generating masks from foreground and background clicks using the `SamPredictor` interface and the underlying architecture in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py).

## Prerequisites: Loading the SAM Model

Before performing point-based segmentation, you must load a pretrained checkpoint and initialize the model components (`ImageEncoderViT`, `PromptEncoder`, `MaskDecoder`). The `sam_model_registry` provides convenient access to model variants (`vit_h`, `vit_l`, `vit_b`).

```python
import torch
from segment_anything import sam_model_registry, SamPredictor

# Load the ViT-H checkpoint

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda" if torch.cuda.is_available() else "cpu")

# Create the predictor wrapper

predictor = SamPredictor(sam)

```

## Step-by-Step Point-Based Segmentation Workflow

### Step 1: Encode the Image with set_image()

The `SamPredictor.set_image()` method in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) preprocesses the input and generates the **image embedding** (`self.features`). This embedding is computed once and reused for all subsequent point prompts on that image.

```python
import cv2
import numpy as np

# Load image as H×W×3 uint8 array

image = cv2.imread("example.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Compute image embeddings (stored in predictor.features)

predictor.set_image(image)

```

### Step 2: Prepare Point Prompts

SAM accepts two arrays for point-based segmentation:

- **`point_coords`**: An `N×2` NumPy array of `(x, y)` pixel locations in the **original image** coordinate system.
- **`point_labels`**: An `N` array where `1` indicates **foreground** (object) and `0` indicates **background** (exclude).

```python

# Foreground click at (150, 200), background click at (300, 400)

point_coords = np.array([[150, 200], [300, 400]], dtype=np.float32)
point_labels = np.array([1, 0], dtype=np.int32)  # 1=fg, 0=bg

```

### Step 3: Generate Masks with predict()

The `SamPredictor.predict()` method orchestrates the segmentation pipeline:

1. **Coordinate Transformation**: `ResizeLongestSide.apply_coords` in [`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py) scales the point coordinates from the original image resolution to the encoder's input size (1024×1024).
2. **Point Embedding**: `PromptEncoder._embed_points` in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) shifts each point to the pixel center, applies random positional encoding, and adds learned foreground/background embeddings.
3. **Mask Decoding**: The `MaskDecoder` in [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) combines sparse point embeddings with dense image features to predict low-resolution masks and IoU scores.
4. **Post-processing**: `Sam.postprocess_masks` upsamples masks to the original image size and applies thresholding.

```python

# Run prediction

masks, iou_scores, low_res_masks = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=False,  # Set True to get 3 candidate masks

)

# masks: (C, H, W) binary array

# iou_scores: (C,) quality scores

# low_res_masks: (C, 256, 256) logits for refinement

```

## Iterative Refinement Using Low-Resolution Logits

SAM supports iterative refinement by feeding previous mask predictions back into the model. Pass the `low_res_masks` from a previous call as `mask_input` to the next `predict` call. This allows the model to "remember" the previous segmentation while incorporating new point prompts.

```python

# First prediction

masks, iou, low_res = predictor.predict(
    point_coords=np.array([[250, 250]], dtype=np.float32),
    point_labels=np.array([1], dtype=np.int32),
    multimask_output=False,
)

# Refinement with additional point and previous mask context

masks2, iou2, _ = predictor.predict(
    point_coords=np.array([[260, 260]], dtype=np.float32),
    point_labels=np.array([1], dtype=np.int32),
    mask_input=low_res,  # Feed previous logits

    multimask_output=False,
)

```

## Key Source Files and Architecture

Understanding the following files in `facebookresearch/segment-anything` helps debug and extend point-based segmentation:

- **[`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py)**: Contains `SamPredictor`, the high-level API that manages image preprocessing, coordinate transformation, and prompt formatting.
- **[`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py)**: Implements the core `Sam` class that orchestrates the image encoder, prompt encoder, and mask decoder.
- **[`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py)**: Houses `PromptEncoder` and the `_embed_points` method that converts click coordinates into sparse embeddings with learned foreground/background tokens.
- **[`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py)**: Contains `MaskDecoder`, the transformer-based module that fuses image and prompt embeddings to predict masks and IoU scores.
- **[`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py)**: Implements `ImageEncoderViT`, the Vision Transformer that generates the dense image features consumed by the decoder.
- **[`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py)**: Provides `ResizeLongestSide`, the utility that rescales images and prompt coordinates to the model's 1024×1024 input resolution.

## Summary

- **Point-based segmentation with SAM** requires initializing a `SamPredictor`, encoding the image once with `set_image()`, and passing `point_coords` and `point_labels` to `predict()`.
- **Coordinate systems** use `(x, y)` pixel locations in the original image resolution; `ResizeLongestSide.apply_coords` handles scaling to the encoder's input size.
- **Prompt encoding** happens in `PromptEncoder._embed_points`, which assigns learned foreground (`1`) and background (`0`) embeddings to each click.
- **Iterative refinement** is supported by passing `low_res_masks` as `mask_input` to subsequent predictions, allowing the model to refine boundaries with additional clicks.

## Frequently Asked Questions

### What coordinate system does SAM use for point prompts?

SAM expects `point_coords` as an `N×2` array of `(x, y)` pixel locations in the **original image coordinate system** (not the resized 1024×1024 input). The `SamPredictor` automatically scales these coordinates to the encoder's resolution using `ResizeLongestSide.apply_coords` in [`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py) before passing them to the `PromptEncoder`.

### How does SAM distinguish between foreground and background clicks?

The `PromptEncoder._embed_points` method in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) uses the `point_labels` array to assign learned embedding vectors. Label `1` triggers the **foreground embedding** (indicating the object to segment), while label `0` triggers the **background embedding** (indicating areas to exclude). These embeddings are added to the positional encoding of each point before being passed to the `MaskDecoder`.

### Can I combine point prompts with box prompts in SAM?

Yes. The `SamPredictor.predict()` method accepts both `point_coords`/`point_labels` and `box` parameters simultaneously. When both are provided, the `PromptEncoder` in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) encodes the box as a pair of points (top-left and bottom-right corners) and concatenates them with the user-provided point embeddings before feeding the combined sparse embeddings to the `MaskDecoder`.

### What is the difference between multimask_output=True and False?

When `multimask_output=True`, the `MaskDecoder` in [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) returns **three candidate masks** (at different resolutions) along with their IoU scores, allowing you to select the best match for ambiguous prompts. When `multimask_output=False` (the default for single-point prompts), the decoder returns only the **highest-scoring mask** (index 0), which is sufficient for clear foreground/background distinctions. The `low_res_masks` returned by `predict()` are always 256×256 logits suitable for iterative refinement regardless of this setting.