Performance Characteristics of SAM Models: Architecture, Speed, and Memory Analysis

The Segment Anything Model (SAM) offers three variants—ViT-H (~1.4B parameters, 4.8GB), ViT-L (~0.4B, 2.3GB), and ViT-B (~0.1B, 1.0GB)—with inference latencies ranging from 0.05s to 0.20s on an NVIDIA A100, trading off segmentation accuracy for computational efficiency.

The performance characteristics of SAM models are determined by their Vision Transformer (ViT) backbones, which scale from 0.1 billion to 1.4 billion parameters across the facebookresearch/segment-anything repository. Understanding these performance characteristics helps developers select the optimal model variant for real-time applications versus high-accuracy segmentation tasks.

SAM Architecture and Model Variants

Core Components

Every SAM model consists of three frozen components that determine its performance profile:

Component	Role	Configuration (ViT-H Example)	Source File
Image encoder	Extracts dense visual features using a ViT backbone, producing a 1024×1024 image embedding.	embed_dim=1280, depth=32, num_heads=16, global_attn_indexes=[7,15,23,31]	ImageEncoderViT in segment_anything/modeling/image_encoder.py
Prompt encoder	Converts points, boxes, or mask inputs into sparse and dense prompt embeddings.	embed_dim=256, image_embedding_size=(64,64), mask_in_chans=16	segment_anything/modeling/prompt_encoder.py
Mask decoder	Lightweight transformer that fuses image and prompt embeddings to predict masks.	num_multimask_outputs=3, TwoWayTransformer(depth=2, embedding_dim=256, mlp_dim=2048, num_heads=8)	segment_anything/modeling/mask_decoder.py

The three public model sizes differ only in the image encoder. The prompt encoder and mask decoder are shared across all variants, meaning the performance bottleneck always lies in the ViT backbone.

Parameter Counts and Storage Requirements

Model	Backbone	Parameters	Checkpoint Size	A100 Latency (1024×1024)	Peak VRAM
ViT-H (sam_vit_h)	ViT-H (1280-dim, 32 layers)	~1.4 B	4.8 GB	~0.20 s (≈5 FPS)	2.6 GB
ViT-L (sam_vit_l)	ViT-L (1024-dim, 24 layers)	~0.4 B	2.3 GB	~0.10 s (≈10 FPS)	1.2 GB
ViT-B (sam_vit_b)	ViT-B (768-dim, 12 layers)	~0.1 B	1.0 GB	~0.05 s (≈20 FPS)	0.6 GB

These metrics assume FP16 inference on an NVIDIA A100 GPU. On consumer GPUs (e.g., RTX 4090), expect roughly 1.5–2× higher latency. CPU inference is not recommended for the image encoder; latency grows to several seconds per image.

Speed vs. Quality Trade-offs

Zero-shot segmentation accuracy scales with model size. On standard benchmarks (COCO, ADE20K), ViT-H consistently outperforms ViT-L and ViT-B by 2–5 mean IoU points. However, the inference latency is roughly linear with parameter count because the image encoder dominates compute (≈1 TFLOP for ViT-H).

For real-time applications requiring 15–30 FPS, ViT-B is the only viable option among native PyTorch checkpoints. For high-accuracy offline processing, ViT-H provides state-of-the-art zero-shot performance at the cost of 2.6 GB VRAM and 200 ms latency per image.

Optimization Strategies

Batching Support

The Sam.forward method in segment_anything/modeling/sam.py accepts a batched_input list. In practice, an A100 GPU can process 2–4 images (1024×1024) simultaneously using ViT-H before exhausting 40 GB of VRAM. Batching improves throughput for video processing but increases peak memory linearly.

ONNX Export and Quantization

For edge deployment, the repository provides scripts/export_onnx_model.py to convert the mask decoder (and optionally the image encoder) to ONNX format. Quantization reduces weights from FP32 to UINT8 with <1% IoU loss and yields approximately 2× speed-up on CPU and JavaScript runtimes, as documented in notebooks/onnx_model_example.ipynb.

Implementation Examples

Loading a Model and Running Single-Point Inference

from segment_anything import sam_model_registry, SamPredictor
import cv2
import numpy as np

# Select variant: 'vit_h', 'vit_l', or 'vit_b'

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Load image (H×W×3 RGB)

predictor.set_image(cv2.imread("scene.jpg"))

# Single point prompt: (x, y) coordinates

point_coords = np.array([[500, 300]])
point_labels = np.array([1])  # 1 = foreground

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=False,
)

The sam_model_registry dictionary is defined in segment_anything/build_sam.py, while SamPredictor resides in segment_anything/predictor.py.

Automatic Mask Generation for Entire Images

from segment_anything import sam_model_registry, SamAutomaticMaskGenerator
import cv2

# Use ViT-L for balanced speed/quality

sam = sam_model_registry["vit_l"](checkpoint="sam_vit_l_0b3195.pth")
mask_generator = SamAutomaticMaskGenerator(sam)

image = cv2.imread("landscape.jpg")
masks = mask_generator.generate(image)

print(f"Generated {len(masks)} masks")

# Each mask contains: 'segmentation' (RLE), 'bbox', 'area', 'predicted_iou'

This functionality is implemented in segment_anything/automatic_mask_generator.py, which internally calls Sam.forward with dense point sampling.

Exporting to ONNX for Edge Deployment

python scripts/export_onnx_model.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --output sam_vit_h.onnx \
    --export_params True \
    --opset 15 \
    --quantize

The resulting ONNX model can be executed via onnxruntime in Python, C++, or WebAssembly environments. The --quantize flag enables UINT8 weight quantization for 2× faster CPU inference.

Key Source Files

File	Purpose	Location
build_sam.py	Factory functions (build_sam_vit_h, build_sam_vit_l, build_sam_vit_b)	segment_anything/build_sam.py
sam.py	Core Sam module implementing forward pass and preprocessing	segment_anything/modeling/sam.py
image_encoder.py	Vision Transformer backbone (ImageEncoderViT)	segment_anything/modeling/image_encoder.py
prompt_encoder.py	Point, box, and mask prompt encoding	segment_anything/modeling/prompt_encoder.py
mask_decoder.py	Lightweight transformer for mask prediction	segment_anything/modeling/mask_decoder.py
predictor.py	High-level SamPredictor wrapper	segment_anything/predictor.py
automatic_mask_generator.py	Whole-image mask generation	segment_anything/automatic_mask_generator.py
export_onnx_model.py	ONNX conversion and quantization	scripts/export_onnx_model.py

Summary

• SAM provides three model sizes (ViT-H, ViT-L, ViT-B) that differ only in the image encoder backbone, ranging from 0.1B to 1.4B parameters.
• Inference latency scales linearly with model size: ViT-B achieves ~20 FPS on an A100, while ViT-H runs at ~5 FPS with 2.6 GB VRAM usage.
• Zero-shot accuracy improves with scale: ViT-H outperforms smaller variants by 2–5 IoU points on COCO and ADE20K benchmarks.
• Optimization paths include batching (2–4 images on A100), ONNX export, and UINT8 quantization for 2× CPU speedup with <1% accuracy loss.

Frequently Asked Questions

Which SAM model variant offers the best speed-to-accuracy ratio?

ViT-L provides the optimal balance for most production use cases, delivering 10 FPS on an A100 with 1.2 GB VRAM while maintaining zero-shot performance within 2–3 IoU points of the largest ViT-H model. ViT-B is preferable for real-time applications above 15 FPS, while ViT-H is reserved for offline high-accuracy segmentation tasks.

How much GPU memory is required to run SAM ViT-H?

ViT-H requires approximately 2.6 GB of peak VRAM for a single 1024×1024 image in FP16 mode on an NVIDIA A100. Memory usage scales linearly with batch size; you can process 2–4 images simultaneously before exhausting a 40 GB A100. For consumer GPUs with 8–12 GB VRAM, ViT-L or ViT-B are recommended to leave headroom for other operations.

Can SAM models be quantized for faster inference?

Yes, the mask decoder can be quantized to UINT8 with minimal accuracy loss. The scripts/export_onnx_model.py utility supports --quantize flags that convert FP32 weights to 8-bit integers, yielding approximately 2× speedup on CPU and JavaScript runtimes with less than 1% IoU degradation. The image encoder can also be exported to ONNX, though quantization of the encoder requires careful calibration to maintain spatial feature quality.

What is the difference between SamPredictor and SamAutomaticMaskGenerator?

SamPredictor is designed for interactive prompting, accepting specific point coordinates, bounding boxes, or mask inputs to segment targeted objects, while SamAutomaticMaskGenerator performs dense grid sampling over the entire image to generate all possible masks without user input. The automatic generator internally calls Sam.forward with a dense point grid and applies non-maximum suppression, making it computationally heavier (processing time scales with image complexity) compared to the single-pass predictor.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}