# How to Choose the Right SAM Model Variant for Your Application

> Find the best SAM model variant for your application. Choose vit_h for max accuracy, vit_l for balanced performance, or vit_b for real-time edge deployment based on your needs.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: best-practices
- Published: 2026-03-07

---

**Select `vit_h` for maximum segmentation accuracy, `vit_l` for balanced GPU performance, or `vit_b` for real-time edge deployment based on your latency and memory constraints.**

The Segment Anything Model (SAM) from Meta AI's `facebookresearch/segment-anything` repository provides three distinct backbone architectures that share the same inference pipeline but differ significantly in computational requirements. Learning how to choose the right SAM model variant ensures optimal performance whether you're processing high-resolution medical imagery or deploying interactive segmentation in browser-based applications.

## SAM Model Variants Overview

The SAM architecture consists of an image encoder, prompt encoder, and mask decoder. All variants use the same prompt encoder and decoder, but differ in the Vision Transformer (ViT) backbone used for image encoding. The [`segment_anything/build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py) file defines three constructor functions that instantiate these variants: `build_sam_vit_h` (lines 14-20), `build_sam_vit_l` (lines 27-33), and `build_sam_vit_b` (lines 36-44).

### vit_h: High Accuracy

The `vit_h` variant uses a ViT-Huge backbone with 632 million parameters, making it the default choice in the `sam_model_registry`. It provides the highest mask quality and excels at fine-grained segmentation tasks such as medical imaging or high-resolution satellite analysis. However, it requires substantial GPU memory and longer inference times.

### vit_l: Balanced Performance

The `vit_l` variant employs a ViT-Large backbone with approximately 300 million parameters. It offers a strong middle ground between accuracy and computational efficiency, making it suitable for desktop GPU deployments and batch processing workflows where `vit_h` would be too slow.

### vit_b: Lightweight and Fast

The `vit_b` variant uses a ViT-Base backbone with only 120 million parameters. It is the fastest and most memory-efficient option, designed specifically for real-time applications, edge devices, and web demos where latency is critical.

## Technical Architecture and Specifications

The following table details the architectural differences between variants as defined in the source code:

| Variant | Backbone | Embedding Dim | Depth | # Heads | Global Attention Layers | Parameters | Typical Use Case |

|---------|----------|---------------|-------|---------|------------------------|------------|------------------|
| `vit_h` | ViT-H | 1280 | 32 | 16 | `[7, 15, 23, 31]` | ~632 M | Research, medical imaging, high-res analysis |
| `vit_l` | ViT-L | 1024 | 24 | 16 | `[5, 11, 17, 23]` | ~300 M | Desktop GPUs, balanced workloads |
| `vit_b` | ViT-B | 768 | 12 | 12 | `[2, 5, 8, 11]` | ~120 M | Real-time apps, edge deployment |

The image encoder processes inputs at 1024×1024 resolution with a patch size of 16, resulting in 64×64 spatial positions. Larger embedding dimensions directly increase memory consumption for these feature maps.

## How to Load Different SAM Model Variants in Python

The `sam_model_registry` dictionary in [`segment_anything/build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py) (lines 47-52) provides a convenient interface for loading any variant by string key. This allows you to switch between models without changing your inference code.

```python
from segment_anything import SamPredictor, sam_model_registry
import cv2

# Select variant: "vit_h", "vit_l", or "vit_b"

model_type = "vit_l"
checkpoint_path = "sam_vit_l_0b3195.pth"  # Download from official README

# Load model through registry

sam = sam_model_registry[model_type](checkpoint=checkpoint_path)
predictor = SamPredictor(sam)

# Prepare image (numpy HWC uint8)

image = cv2.imread("example.jpg")
predictor.set_image(image)

# Run inference with point prompt

masks, scores, _ = predictor.predict(
    point_coords=[[250, 300]],
    point_labels=[1],  # 1 = foreground

    multimask_output=False
)

```

Changing `model_type` from `"vit_h"` to `"vit_l"` or `"vit_b"` instantly switches the backbone while preserving identical prompting and decoding logic.

## Performance Trade-offs and Selection Criteria

When choosing the right SAM model variant, evaluate these three primary constraints:

**Accuracy Requirements**
- Use **`vit_h`** when mask boundary precision is paramount, such as in medical imaging, satellite analysis, or research benchmarks.
- Use **`vit_l`** for general-purpose segmentation where high quality is needed but extreme precision is optional.
- Use **`vit_b`** for coarse segmentation or when speed matters more than pixel-perfect boundaries.

**GPU Memory Constraints**
The image encoder's feature map size scales with embedding dimension. If you encounter CUDA out-of-memory errors during `set_image()`, downgrade from `vit_h` to `vit_l` or `vit_b`. The base variant requires approximately 4× less memory than the huge variant for the encoder features.

**Latency and Throughput**
- **`vit_h`**: ~1-2 seconds per image on high-end GPUs (slower on consumer hardware).
- **`vit_l`**: ~0.5-1 seconds per image; suitable for interactive desktop applications.
- **`vit_b`**: ~0.1-0.3 seconds per image; ideal for real-time video processing or web demos.

## Command-Line Usage for Batch Processing

For batch processing without writing Python code, use the [`scripts/amg.py`](https://github.com/facebookresearch/segment-anything/blob/main/scripts/amg.py) CLI tool, which accepts the `--model-type` argument to specify the variant:

```bash
python scripts/amg.py \
  --checkpoint sam_vit_b_01ec64.pth \
  --model-type vit_b \
  --input /path/to/images/ \
  --output /path/to/masks/ \
  --points-per-side 32

```

Reducing `--points-per-side` further increases speed for the `vit_b` variant, making it suitable for rapid prototyping on large datasets.

## Summary

- **Three variants** are available via `sam_model_registry`: `vit_h` (632M params), `vit_l` (300M params), and `vit_b` (120M params).
- **Architecture** differs only in the image encoder's Vision Transformer backbone; prompt encoding and mask decoding remain identical across all variants.
- **Selection criteria**: Use `vit_h` for maximum accuracy, `vit_l` for balanced desktop performance, and `vit_b` for real-time or memory-constrained edge deployment.
- **Implementation** requires only changing the `model_type` string and checkpoint path when calling `sam_model_registry[model_type](checkpoint=...)`.

## Frequently Asked Questions

### What is the difference between SAM model variants?

The SAM model variants (`vit_h`, `vit_l`, and `vit_b`) differ exclusively in their image encoder backbones. The `vit_h` variant uses a ViT-Huge architecture with 1280 embedding dimensions and 632 million parameters, providing the highest segmentation accuracy. The `vit_l` variant uses ViT-Large with 1024 dimensions and 300 million parameters, while `vit_b` uses ViT-Base with 768 dimensions and 120 million parameters for maximum speed.

### How much GPU memory does each SAM variant require?

GPU memory scales primarily with the image encoder's embedding dimension and the 64×64 spatial feature map. The `vit_h` variant requires the most memory due to its 1280-dimensional features, often needing 8GB+ VRAM for high-resolution inputs. The `vit_l` variant typically requires 4-6GB, while `vit_b` can run comfortably on 2-4GB, making it suitable for consumer GPUs and edge devices.

### Can I switch between SAM variants without changing my inference code?

Yes, you can switch between variants by only modifying the `model_type` string and checkpoint path when loading through `sam_model_registry`. The `SamPredictor` API remains identical across all variants, including the `set_image()`, `predict()`, and `predict_torch()` methods. This means you can prototype with `vit_h` for accuracy, then deploy with `vit_b` for speed using the exact same prompting logic.

### Which SAM variant is best for real-time applications?

The `vit_b` (ViT-Base) variant is specifically designed for real-time applications, offering inference speeds of 0.1-0.3 seconds per image on modern GPUs. With only 120 million parameters and 768 embedding dimensions, it minimizes latency while maintaining usable segmentation quality. For video processing or web-based demos, combine `vit_b` with reduced `--points-per-side` values in the automatic mask generation pipeline to achieve interactive frame rates.