# Performance Characteristics of SAM Models: Architecture, Speed, and Memory Analysis

> Explore SAM model performance characteristics. Analyze ViT-H, ViT-L, and ViT-B variants for speed, memory, and accuracy trade-offs. Optimize your segmentation tasks.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: performance
- Published: 2026-03-07

---

**The Segment Anything Model (SAM) offers three variants—ViT-H (~1.4B parameters, 4.8GB), ViT-L (~0.4B, 2.3GB), and ViT-B (~0.1B, 1.0GB)—with inference latencies ranging from 0.05s to 0.20s on an NVIDIA A100, trading off segmentation accuracy for computational efficiency.**

The performance characteristics of SAM models are determined by their Vision Transformer (ViT) backbones, which scale from 0.1 billion to 1.4 billion parameters across the facebookresearch/segment-anything repository. Understanding these performance characteristics helps developers select the optimal model variant for real-time applications versus high-accuracy segmentation tasks.

## SAM Architecture and Model Variants

### Core Components

Every SAM model consists of three frozen components that determine its performance profile:

| Component | Role | Configuration (ViT-H Example) | Source File |
|-----------|------|------------------------------|-------------|
| **Image encoder** | Extracts dense visual features using a ViT backbone, producing a 1024×1024 image embedding. | `embed_dim=1280`, `depth=32`, `num_heads=16`, `global_attn_indexes=[7,15,23,31]` | `ImageEncoderViT` in [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py) |
| **Prompt encoder** | Converts points, boxes, or mask inputs into sparse and dense prompt embeddings. | `embed_dim=256`, `image_embedding_size=(64,64)`, `mask_in_chans=16` | [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) |
| **Mask decoder** | Lightweight transformer that fuses image and prompt embeddings to predict masks. | `num_multimask_outputs=3`, `TwoWayTransformer(depth=2, embedding_dim=256, mlp_dim=2048, num_heads=8)` | [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) |

The three public model sizes differ **only** in the image encoder. The prompt encoder and mask decoder are shared across all variants, meaning the performance bottleneck always lies in the ViT backbone.

### Parameter Counts and Storage Requirements

| Model | Backbone | Parameters | Checkpoint Size | A100 Latency (1024×1024) | Peak VRAM |
|-------|----------|------------|-----------------|-------------------------|-----------|
| **ViT-H** (`sam_vit_h`) | ViT-H (1280-dim, 32 layers) | ~1.4 B | 4.8 GB | ~0.20 s (≈5 FPS) | 2.6 GB |
| **ViT-L** (`sam_vit_l`) | ViT-L (1024-dim, 24 layers) | ~0.4 B | 2.3 GB | ~0.10 s (≈10 FPS) | 1.2 GB |
| **ViT-B** (`sam_vit_b`) | ViT-B (768-dim, 12 layers) | ~0.1 B | 1.0 GB | ~0.05 s (≈20 FPS) | 0.6 GB |

These metrics assume **FP16 inference** on an NVIDIA A100 GPU. On consumer GPUs (e.g., RTX 4090), expect roughly 1.5–2× higher latency. CPU inference is not recommended for the image encoder; latency grows to several seconds per image.

## Speed vs. Quality Trade-offs

**Zero-shot segmentation accuracy** scales with model size. On standard benchmarks (COCO, ADE20K), ViT-H consistently outperforms ViT-L and ViT-B by 2–5 mean IoU points. However, the **inference latency** is roughly linear with parameter count because the image encoder dominates compute (≈1 TFLOP for ViT-H).

For **real-time applications** requiring 15–30 FPS, ViT-B is the only viable option among native PyTorch checkpoints. For **high-accuracy offline processing**, ViT-H provides state-of-the-art zero-shot performance at the cost of 2.6 GB VRAM and 200 ms latency per image.

## Optimization Strategies

### Batching Support

The `Sam.forward` method in [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) accepts a `batched_input` list. In practice, an A100 GPU can process **2–4 images** (1024×1024) simultaneously using ViT-H before exhausting 40 GB of VRAM. Batching improves throughput for video processing but increases peak memory linearly.

### ONNX Export and Quantization

For edge deployment, the repository provides [`scripts/export_onnx_model.py`](https://github.com/facebookresearch/segment-anything/blob/main/scripts/export_onnx_model.py) to convert the mask decoder (and optionally the image encoder) to ONNX format. Quantization reduces weights from FP32 to UINT8 with **<1% IoU loss** and yields approximately **2× speed-up** on CPU and JavaScript runtimes, as documented in `notebooks/onnx_model_example.ipynb`.

## Implementation Examples

### Loading a Model and Running Single-Point Inference

```python
from segment_anything import sam_model_registry, SamPredictor
import cv2
import numpy as np

# Select variant: 'vit_h', 'vit_l', or 'vit_b'

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Load image (H×W×3 RGB)

predictor.set_image(cv2.imread("scene.jpg"))

# Single point prompt: (x, y) coordinates

point_coords = np.array([[500, 300]])
point_labels = np.array([1])  # 1 = foreground

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=False,
)

```

The `sam_model_registry` dictionary is defined in [`segment_anything/build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py), while `SamPredictor` resides in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py).

### Automatic Mask Generation for Entire Images

```python
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator
import cv2

# Use ViT-L for balanced speed/quality

sam = sam_model_registry["vit_l"](checkpoint="sam_vit_l_0b3195.pth")
mask_generator = SamAutomaticMaskGenerator(sam)

image = cv2.imread("landscape.jpg")
masks = mask_generator.generate(image)

print(f"Generated {len(masks)} masks")

# Each mask contains: 'segmentation' (RLE), 'bbox', 'area', 'predicted_iou'

```

This functionality is implemented in [`segment_anything/automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py), which internally calls `Sam.forward` with dense point sampling.

### Exporting to ONNX for Edge Deployment

```bash
python scripts/export_onnx_model.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --output sam_vit_h.onnx \
    --export_params True \
    --opset 15 \
    --quantize

```

The resulting ONNX model can be executed via `onnxruntime` in Python, C++, or WebAssembly environments. The `--quantize` flag enables UINT8 weight quantization for 2× faster CPU inference.

## Key Source Files

| File | Purpose | Location |
|------|---------|----------|
| [`build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/build_sam.py) | Factory functions (`build_sam_vit_h`, `build_sam_vit_l`, `build_sam_vit_b`) | [`segment_anything/build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py) |
| [`sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/sam.py) | Core `Sam` module implementing forward pass and preprocessing | [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) |
| [`image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/image_encoder.py) | Vision Transformer backbone (`ImageEncoderViT`) | [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py) |
| [`prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/prompt_encoder.py) | Point, box, and mask prompt encoding | [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) |
| [`mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/mask_decoder.py) | Lightweight transformer for mask prediction | [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) |
| [`predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/predictor.py) | High-level `SamPredictor` wrapper | [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) |
| [`automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/automatic_mask_generator.py) | Whole-image mask generation | [`segment_anything/automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py) |
| [`export_onnx_model.py`](https://github.com/facebookresearch/segment-anything/blob/main/export_onnx_model.py) | ONNX conversion and quantization | [`scripts/export_onnx_model.py`](https://github.com/facebookresearch/segment-anything/blob/main/scripts/export_onnx_model.py) |

## Summary

- **SAM provides three model sizes** (ViT-H, ViT-L, ViT-B) that differ only in the image encoder backbone, ranging from 0.1B to 1.4B parameters.
- **Inference latency scales linearly** with model size: ViT-B achieves ~20 FPS on an A100, while ViT-H runs at ~5 FPS with 2.6 GB VRAM usage.
- **Zero-shot accuracy improves with scale**: ViT-H outperforms smaller variants by 2–5 IoU points on COCO and ADE20K benchmarks.
- **Optimization paths include batching** (2–4 images on A100), **ONNX export**, and **UINT8 quantization** for 2× CPU speedup with <1% accuracy loss.

## Frequently Asked Questions

### Which SAM model variant offers the best speed-to-accuracy ratio?

**ViT-L provides the optimal balance** for most production use cases, delivering 10 FPS on an A100 with 1.2 GB VRAM while maintaining zero-shot performance within 2–3 IoU points of the largest ViT-H model. ViT-B is preferable for real-time applications above 15 FPS, while ViT-H is reserved for offline high-accuracy segmentation tasks.

### How much GPU memory is required to run SAM ViT-H?

**ViT-H requires approximately 2.6 GB of peak VRAM** for a single 1024×1024 image in FP16 mode on an NVIDIA A100. Memory usage scales linearly with batch size; you can process 2–4 images simultaneously before exhausting a 40 GB A100. For consumer GPUs with 8–12 GB VRAM, ViT-L or ViT-B are recommended to leave headroom for other operations.

### Can SAM models be quantized for faster inference?

**Yes, the mask decoder can be quantized to UINT8 with minimal accuracy loss.** The [`scripts/export_onnx_model.py`](https://github.com/facebookresearch/segment-anything/blob/main/scripts/export_onnx_model.py) utility supports `--quantize` flags that convert FP32 weights to 8-bit integers, yielding approximately **2× speedup on CPU and JavaScript runtimes** with less than 1% IoU degradation. The image encoder can also be exported to ONNX, though quantization of the encoder requires careful calibration to maintain spatial feature quality.

### What is the difference between SamPredictor and SamAutomaticMaskGenerator?

**SamPredictor is designed for interactive prompting**, accepting specific point coordinates, bounding boxes, or mask inputs to segment targeted objects, while **SamAutomaticMaskGenerator performs dense grid sampling** over the entire image to generate all possible masks without user input. The automatic generator internally calls `Sam.forward` with a dense point grid and applies non-maximum suppression, making it computationally heavier (processing time scales with image complexity) compared to the single-pass predictor.