What Is the SA-1B Dataset and How Does It Train SAM?

The SA-1B dataset comprises 1.1 billion segmentation masks across 11 million images, providing the visual diversity required to train the Segment Anything Model (SAM) for zero-shot, promptable segmentation.

The SA-1B dataset serves as the massive training corpus behind Meta's Segment Anything Model (SAM). Released through the facebookresearch/segment-anything repository, this dataset provides the diverse visual supervision necessary for SAM's ability to segment virtually any object from prompts such as points, boxes, or text without task-specific fine-tuning.

What Is the SA-1B Dataset?

The SA-1B dataset (Segment Anything 1 Billion) is a large-scale image collection specifically constructed to train foundation models for segmentation. Unlike traditional datasets with manual annotations, SA-1B was created using an automated pipeline that generated high-quality masks at unprecedented scale.

Scale and Composition

SA-1B contains approximately 11 million images paired with roughly 1.1 billion masks. This massive scale provides the visual diversity required for zero-shot generalization, allowing SAM to recognize and segment objects it has never explicitly seen during training. Each image in the dataset is accompanied by multiple object masks representing various visual concepts, from common objects to obscure patterns.

Data Format and Licensing

The mask data are stored as per-image JSON files using COCO Run-Length Encoding (RLE) format. According to the repository's README.md, the dataset is covered by the SA-1B Dataset Research License and must be downloaded separately from Meta's dataset portal. The RLE format efficiently compresses binary masks into compact string representations suitable for large-scale storage.

How SAM Trains on the SA-1B Dataset

SAM was trained directly on the masks contained within SA-1B, using the dataset's diversity to learn generalized segmentation features rather than class-specific patterns.

Direct Supervision from Mask Annotations

The model was trained on the full corpus of 11 million images and 1.1 billion masks, treating each mask as a supervision signal for the image encoder, prompt encoder, and mask decoder architecture defined in segment_anything/modeling/sam.py. This direct training approach enables SAM to predict valid masks for any prompt location, even in ambiguous scenarios where multiple objects overlap.

The Automatic Mask Generation Pipeline

The same infrastructure used to create SA-1B is available in the repository via segment_anything/automatic_mask_generator.py. This module implements the grid-based point sampling strategy that originally generated the 1.1 billion masks. By running this generator across millions of images at multiple scales, the authors produced the diverse training set without manual annotation.

Loading and Using SA-1B Data

Working with SA-1B requires decoding the COCO RLE format using pycocotools, then optionally running inference with SAM checkpoints pre-trained on this data.

Decoding COCO RLE Masks

Each annotation file contains masks encoded in COCO RLE format. Use the following pattern to load and decode these masks into binary numpy arrays:

import json
from pycocotools import mask as mask_utils

# Path to a per-image JSON file from SA-1B

json_path = "path/to/sa_image_00001.json"

with open(json_path) as f:
    data = json.load(f)

# Each entry in `annotations` holds a mask in COCO RLE format

annotation = data["annotations"][0]

# Decode the RLE to a binary mask (H × W numpy array)

binary_mask = mask_utils.decode(annotation["segmentation"])

Running Inference with SAM

Once trained on SA-1B, SAM can generate masks for new images using prompts. The SamPredictor class in segment_anything/predictor.py provides the inference interface:

from segment_anything import SamPredictor, sam_model_registry

# Load a pre-trained checkpoint (trained on SA-1B)

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Set the image (NumPy H×W×3 array)

predictor.set_image(my_image)

# Single-point prompt example

point = [[250, 300]]          # (x, y) coordinates

label = [1]                   # 1 → foreground point

masks, scores, logits = predictor.predict(
    point_coords=point,
    point_labels=label,
    multimask_output=False,
)

# masks[0] contains the binary segmentation mask

Regenerating Masks Automatically

To generate comprehensive masks for an entire image (replicating the SA-1B creation process), use SamAutomaticMaskGenerator:

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
mask_generator = SamAutomaticMaskGenerator(sam)

# Generates hundreds of masks across the image

all_masks = mask_generator.generate(my_image)

Key Implementation Files in the Repository

The facebookresearch/segment-anything repository contains several critical files that implement the training infrastructure and data handling for SA-1B:

• segment_anything/predictor.py – Implements SamPredictor, the core inference class that runs the image encoder and mask decoder trained on SA-1B.

• segment_anything/automatic_mask_generator.py – Contains SamAutomaticMaskGenerator, the pipeline used to generate the 1.1 billion masks in SA-1B through grid-based point prompting.

• segment_anything/modeling/sam.py – Defines the SAM architecture including the Vision Transformer image encoder and the lightweight mask decoder.

• README.md (Dataset section) – Documents the SA-1B download location, licensing requirements, and data format specifications.

Summary

• The SA-1B dataset contains 1.1 billion masks from 11 million images, making it the largest segmentation dataset available for research.
• Masks are stored in COCO RLE format within per-image JSON files and require pycocotools for decoding.
• SAM was trained directly on SA-1B masks, enabling its zero-shot segmentation capabilities across diverse visual domains.
• The automatic_mask_generator.py module implements the same pipeline used to create the dataset.
• The dataset is subject to the SA-1B Dataset Research License and hosted externally from the code repository.

Frequently Asked Questions

How large is the SA-1B dataset?

The SA-1B dataset contains approximately 11 million images with 1.1 billion associated segmentation masks. This makes it one of the largest publicly available image segmentation datasets by mask count, providing the scale necessary for training foundation models like SAM.

What format are the masks stored in?

Masks are encoded using COCO Run-Length Encoding (RLE) format, stored as strings within per-image JSON files. To work with these masks, you must use pycocotools.mask.decode() to convert the RLE strings into binary numpy arrays representing the segmentation bitmaps.

Can I use SA-1B for commercial applications?

No, the SA-1B dataset is distributed under the SA-1B Dataset Research License, which restricts usage to non-commercial research purposes. You must agree to these terms when downloading the dataset from Meta's portal, separate from the Apache-2.0 licensed code in the repository.

How does SA-1B enable zero-shot segmentation?

By training on 1.1 billion diverse masks covering a vast range of visual concepts, SAM learns generalizable features for "what constitutes an object" rather than memorizing specific classes. This allows the model, as implemented in segment_anything/predictor.py, to segment novel objects during inference based solely on geometric prompts like points or bounding boxes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}