How to Perform Point-Based Segmentation with SAM: A Complete Guide

Point-based segmentation with SAM works by encoding click coordinates into sparse embeddings via the PromptEncoder, combining them with pre-computed image features, and decoding them into binary masks through the MaskDecoder.

The Segment Anything Model (SAM) from facebookresearch/segment-anything enables precise object segmentation through interactive point prompts. This guide explains the complete workflow for generating masks from foreground and background clicks using the SamPredictor interface and the underlying architecture in segment_anything/predictor.py.

Prerequisites: Loading the SAM Model

Before performing point-based segmentation, you must load a pretrained checkpoint and initialize the model components (ImageEncoderViT, PromptEncoder, MaskDecoder). The sam_model_registry provides convenient access to model variants (vit_h, vit_l, vit_b).

import torch
from segment_anything import sam_model_registry, SamPredictor

# Load the ViT-H checkpoint

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda" if torch.cuda.is_available() else "cpu")

# Create the predictor wrapper

predictor = SamPredictor(sam)

Step-by-Step Point-Based Segmentation Workflow

Step 1: Encode the Image with set_image()

The SamPredictor.set_image() method in segment_anything/predictor.py preprocesses the input and generates the image embedding (self.features). This embedding is computed once and reused for all subsequent point prompts on that image.

import cv2
import numpy as np

# Load image as H×W×3 uint8 array

image = cv2.imread("example.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Compute image embeddings (stored in predictor.features)

predictor.set_image(image)

Step 2: Prepare Point Prompts

SAM accepts two arrays for point-based segmentation:

• point_coords: An N×2 NumPy array of (x, y) pixel locations in the original image coordinate system.
• point_labels: An N array where 1 indicates foreground (object) and 0 indicates background (exclude).

# Foreground click at (150, 200), background click at (300, 400)

point_coords = np.array([[150, 200], [300, 400]], dtype=np.float32)
point_labels = np.array([1, 0], dtype=np.int32)  # 1=fg, 0=bg

Step 3: Generate Masks with predict()

The SamPredictor.predict() method orchestrates the segmentation pipeline:

1. Coordinate Transformation: ResizeLongestSide.apply_coords in segment_anything/utils/transforms.py scales the point coordinates from the original image resolution to the encoder's input size (1024×1024).
2. Point Embedding: PromptEncoder._embed_points in segment_anything/modeling/prompt_encoder.py shifts each point to the pixel center, applies random positional encoding, and adds learned foreground/background embeddings.
3. Mask Decoding: The MaskDecoder in segment_anything/modeling/mask_decoder.py combines sparse point embeddings with dense image features to predict low-resolution masks and IoU scores.
4. Post-processing: Sam.postprocess_masks upsamples masks to the original image size and applies thresholding.

# Run prediction

masks, iou_scores, low_res_masks = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=False,  # Set True to get 3 candidate masks

)

# masks: (C, H, W) binary array

# iou_scores: (C,) quality scores

# low_res_masks: (C, 256, 256) logits for refinement

Iterative Refinement Using Low-Resolution Logits

SAM supports iterative refinement by feeding previous mask predictions back into the model. Pass the low_res_masks from a previous call as mask_input to the next predict call. This allows the model to "remember" the previous segmentation while incorporating new point prompts.

# First prediction

masks, iou, low_res = predictor.predict(
    point_coords=np.array([[250, 250]], dtype=np.float32),
    point_labels=np.array([1], dtype=np.int32),
    multimask_output=False,
)

# Refinement with additional point and previous mask context

masks2, iou2, _ = predictor.predict(
    point_coords=np.array([[260, 260]], dtype=np.float32),
    point_labels=np.array([1], dtype=np.int32),
    mask_input=low_res,  # Feed previous logits

    multimask_output=False,
)

Key Source Files and Architecture

Understanding the following files in facebookresearch/segment-anything helps debug and extend point-based segmentation:

• segment_anything/predictor.py: Contains SamPredictor, the high-level API that manages image preprocessing, coordinate transformation, and prompt formatting.
• segment_anything/modeling/sam.py: Implements the core Sam class that orchestrates the image encoder, prompt encoder, and mask decoder.
• segment_anything/modeling/prompt_encoder.py: Houses PromptEncoder and the _embed_points method that converts click coordinates into sparse embeddings with learned foreground/background tokens.
• segment_anything/modeling/mask_decoder.py: Contains MaskDecoder, the transformer-based module that fuses image and prompt embeddings to predict masks and IoU scores.
• segment_anything/modeling/image_encoder.py: Implements ImageEncoderViT, the Vision Transformer that generates the dense image features consumed by the decoder.
• segment_anything/utils/transforms.py: Provides ResizeLongestSide, the utility that rescales images and prompt coordinates to the model's 1024×1024 input resolution.

Summary

• Point-based segmentation with SAM requires initializing a SamPredictor, encoding the image once with set_image(), and passing point_coords and point_labels to predict().
• Coordinate systems use (x, y) pixel locations in the original image resolution; ResizeLongestSide.apply_coords handles scaling to the encoder's input size.
• Prompt encoding happens in PromptEncoder._embed_points, which assigns learned foreground (1) and background (0) embeddings to each click.
• Iterative refinement is supported by passing low_res_masks as mask_input to subsequent predictions, allowing the model to refine boundaries with additional clicks.

Frequently Asked Questions

What coordinate system does SAM use for point prompts?

SAM expects point_coords as an N×2 array of (x, y) pixel locations in the original image coordinate system (not the resized 1024×1024 input). The SamPredictor automatically scales these coordinates to the encoder's resolution using ResizeLongestSide.apply_coords in segment_anything/utils/transforms.py before passing them to the PromptEncoder.

How does SAM distinguish between foreground and background clicks?

The PromptEncoder._embed_points method in segment_anything/modeling/prompt_encoder.py uses the point_labels array to assign learned embedding vectors. Label 1 triggers the foreground embedding (indicating the object to segment), while label 0 triggers the background embedding (indicating areas to exclude). These embeddings are added to the positional encoding of each point before being passed to the MaskDecoder.

Can I combine point prompts with box prompts in SAM?

Yes. The SamPredictor.predict() method accepts both point_coords/point_labels and box parameters simultaneously. When both are provided, the PromptEncoder in segment_anything/modeling/prompt_encoder.py encodes the box as a pair of points (top-left and bottom-right corners) and concatenates them with the user-provided point embeddings before feeding the combined sparse embeddings to the MaskDecoder.

What is the difference between multimask_output=True and False?

When multimask_output=True, the MaskDecoder in segment_anything/modeling/mask_decoder.py returns three candidate masks (at different resolutions) along with their IoU scores, allowing you to select the best match for ambiguous prompts. When multimask_output=False (the default for single-point prompts), the decoder returns only the highest-scoring mask (index 0), which is sufficient for clear foreground/background distinctions. The low_res_masks returned by predict() are always 256×256 logits suitable for iterative refinement regardless of this setting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}