SAM Model Sizes Explained: vit_h, vit_l, and vit_b Comparison in Segment Anything

The Segment Anything Model (SAM) offers three Vision Transformer backbone sizes—vit_h (Huge), vit_l (Large), and vit_b (Base)—that trade off between segmentation accuracy and computational cost through varying embedding dimensions, transformer depths, and attention head counts.

The facebookresearch/segment-anything repository implements these three SAM model sizes to support diverse deployment scenarios, from edge devices to high-end research workstations. Understanding the architectural differences between vit_h, vit_l, and vit_b enables you to select the optimal balance of inference speed and mask quality for your computer vision pipeline.

SAM Model Size Specifications and Architecture Differences

The three variants differ primarily in embedding dimension, transformer depth, and attention head configuration, which directly impacts GPU memory consumption and inference latency.

| Model | Backbone | Embed Dim | Depth | # Heads | Global-Attention Indices |

|-------|----------|-----------|-------|--------|-------------------------| | vit_h | ViT-Huge | 1280 | 32 | 16 | [7, 15, 23, 31] | | vit_l | ViT-Large | 1024 | 24 | 16 | [5, 11, 17, 23] | | vit_b | ViT-Base | 768 | 12 | 12 | [2, 5, 8, 11] |

The global-attention indices specify which transformer blocks use full global attention rather than windowed attention, affecting the receptive field and computational cost. vit_h provides the highest accuracy but requires the most memory, while vit_b offers the fastest inference with the smallest footprint.

Where SAM Model Sizes Are Defined in the Source Code

The model configurations are hardcoded in segment_anything/build_sam.py, where each size has a dedicated builder function that passes specific hyperparameters to the internal _build_sam helper.

The vit_h builder (lines 14-21) initializes the huge variant:

def build_sam_vit_h(checkpoint=None):
    return _build_sam(
        encoder_embed_dim=1280,
        encoder_depth=32,
        encoder_num_heads=16,
        encoder_global_attn_indexes=[7, 15, 23, 31],
        checkpoint=checkpoint,
    )

The vit_l builder (lines 27-34) configures the large variant:

def build_sam_vit_l(checkpoint=None):
    return _build_sam(
        encoder_embed_dim=1024,
        encoder_depth=24,
        encoder_num_heads=16,
        encoder_global_attn_indexes=[5, 11, 17, 23],
        checkpoint=checkpoint,
    )

The vit_b builder (lines 37-44) defines the base variant:

def build_sam_vit_b(checkpoint=None):
    return _build_sam(
        encoder_embed_dim=768,
        encoder_depth=12,
        encoder_num_heads=12,
        encoder_global_attn_indexes=[2, 5, 8, 11],
        checkpoint=checkpoint,
    )

All three functions delegate to _build_sam (lines 55-107), which assembles the complete architecture including the image encoder, prompt encoder, and mask decoder.

The model registry (lines 47-52) maps string identifiers to these builders:

sam_model_registry = {
    "default": build_sam_vit_h,
    "vit_h": build_sam_vit_h,
    "vit_l": build_sam_vit_l,
    "vit_b": build_sam_vit_b,
}

How to Load and Use Different SAM Model Sizes

Selecting a model size involves choosing the appropriate key from sam_model_registry and loading the corresponding checkpoint file.

Loading a Specific Model Size

from segment_anything import sam_model_registry

# Select model size: "vit_h", "vit_l", or "vit_b"

model_type = "vit_l"
checkpoint_path = "sam_vit_l.pth"

# Load model

sam = sam_model_registry[model_type](checkpoint=checkpoint_path)
sam.eval()  # Set to evaluation mode

Running Inference with SamPredictor

The SamPredictor class provides a high-level interface for generating masks:

from segment_anything import SamPredictor
import cv2
import numpy as np

# Initialize predictor with loaded model

predictor = SamPredictor(sam)

# Load and preprocess image

image = cv2.imread("example.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Set image (computes embeddings once)

predictor.set_image(image)

# Define point prompt (x, y) and label (1=foreground)

point_coords = np.array([[500, 375]])
point_labels = np.array([1])

# Generate masks

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True,
)

Exporting to ONNX for Deployment

For production environments, convert any model size to ONNX format:

python -m segment_anything.scripts.export_onnx_model \
    --checkpoint sam_vit_b.pth \
    --model-type vit_b \
    --output sam_vit_b.onnx

Summary

• The facebookresearch/segment-anything repository provides three SAM model sizes—vit_h, vit_l, and vit_b—differentiated by Vision Transformer backbone scale and configured in segment_anything/build_sam.py.
• vit_h (Huge) offers the highest accuracy with 1280 embedding dimensions and 32 transformer blocks, while vit_b (Base) provides the fastest inference with 768 dimensions and 12 blocks.
• The sam_model_registry dictionary maps string keys to builder functions, allowing you to instantiate any size by changing the model type parameter.
• Select vit_b for edge deployment, vit_l for balanced performance, and vit_h for maximum accuracy on high-end GPUs.

Frequently Asked Questions

What is the difference between vit_h, vit_l, and vit_b in SAM?

The three variants differ in their Vision Transformer backbone capacity as defined in segment_anything/build_sam.py. vit_h (Huge) uses 1280 embedding dimensions across 32 transformer blocks with 16 attention heads, providing the highest accuracy but requiring the most GPU memory. vit_l (Large) uses 1024 dimensions and 24 blocks, while vit_b (Base) uses 768 dimensions and 12 blocks, delivering the fastest inference with the smallest memory footprint.

Which SAM model size should I use for my application?

Choose vit_b if you are deploying on edge devices or consumer GPUs with limited VRAM, as it requires approximately 4x less memory than vit_h. Use vit_l for production servers where you need strong accuracy without the extreme memory overhead of the huge variant. Reserve vit_h for research environments or high-end workstations where maximum segmentation quality is prioritized over inference speed.

How do I switch between SAM model sizes in code?

You can switch models by changing the key passed to sam_model_registry in segment_anything/build_sam.py. The registry accepts "vit_h", "vit_l", or "vit_b" as strings, mapping each to its respective builder function (build_sam_vit_h, build_sam_vit_l, or build_sam_vit_b). Simply load the corresponding checkpoint file and instantiate the model using the registry key that matches your hardware constraints.

Can I export smaller SAM models like vit_b to ONNX for faster deployment?

Yes, the repository includes a dedicated export script at scripts/export_onnx_model.py that supports all three model sizes. When exporting vit_b, the resulting ONNX model will have significantly smaller file size and faster CPU inference compared to vit_h, making it ideal for deployment in resource-constrained environments. The export process remains identical across variants—simply specify --model-type vit_b when running the script.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}