How to Integrate SAM into a Web Application: A Complete Production Guide

You can integrate SAM into a web application by exporting the mask decoder to ONNX, pre-computing image embeddings on a Python backend, and running interactive inference in the browser using ONNX Runtime Web with React.

The facebookresearch/segment-anything repository provides a complete pipeline for deploying the Segment Anything Model (SAM) in browser-based applications. This guide walks through the production-ready architecture used in the official demo, covering Python backend setup, ONNX export, and React frontend integration.

Architecture Overview for Web Deployment

Integrating SAM into a web application requires splitting the workload between server and client. The heavy image encoder runs once on the backend to generate embeddings, while the lightweight mask decoder runs interactively in the browser.

The architecture follows three distinct stages:

1. Python Backend – Loads a SAM checkpoint via sam_model_registry, generates image embeddings using SamPredictor, and exports the mask decoder to ONNX format.
2. ONNX Optimization – Quantizes the exported model to reduce file size for faster web delivery.
3. React Frontend – Loads the ONNX model with onnxruntime-web, ingests pre-computed embeddings, and performs real-time mask prediction based on user interactions.

Setting Up the Python Environment

Begin by cloning the repository and installing dependencies required for model inference and ONNX export.

git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .
pip install opencv-python pycocotools matplotlib onnxruntime onnx

Download a SAM checkpoint. The repository supports three model types: vit_h (largest), vit_l, and vit_b (smallest).

wget https://dl.fbaipublicfiles.com/segment-anything/sam_vit_h_4b8939.pth -O sam_vit_h.pth

Generating Image Embeddings on the Backend

The SamPredictor class in segment_anything/predictor.py handles image preprocessing and embedding generation. The set_image() method applies ResizeLongestSide preprocessing (lines 34-59), while get_image_embedding() extracts the feature tensor (lines 45-56).

Create a script to generate and save embeddings for frontend consumption:

# generate_embedding.py

import cv2
import numpy as np
from segment_anything import sam_model_registry, SamPredictor

# Load model

checkpoint = "sam_vit_h.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=checkpoint)
sam.to(device="cuda")

# Initialize predictor

predictor = SamPredictor(sam)

# Load and process image

image = cv2.imread("src/assets/data/dogs.jpg")
predictor.set_image(image, image_format="BGR")

# Extract and save embedding

embedding = predictor.get_image_embedding().cpu().numpy()
np.save("demo/src/assets/data/dogs_embedding.npy", embedding.squeeze())
print("Embedding saved successfully")

Exporting the Mask Decoder to ONNX

The scripts/export_onnx_model.py script exports the prompt encoder and mask decoder to ONNX format, enabling browser-based inference. The script wraps these components in SamOnnxModel and performs Torch-ONNX export (lines 97-160). Dynamic quantization (lines 87-99) reduces model size for web deployment.

Run the export with quantization:

python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h.pth \
  --model-type vit_h \
  --output demo/model/sam_onnx_quantized_example.onnx \
  --return-single-mask \
  --opset 17 \
  --quantize-out demo/model/sam_onnx_quantized_example_quant.onnx

The quantized ONNX file (*_quant.onnx) is typically 100-400MB depending on the model type, suitable for CDN distribution.

Integrating the Frontend with React and ONNX Runtime Web

The demo application in demo/src/App.tsx demonstrates the complete integration pattern. The frontend loads the quantized ONNX model using onnxruntime-web and performs inference using pre-computed embeddings.

Configuring Asset Paths

Update the constants in demo/src/App.tsx to point to your generated assets:

const IMAGE_PATH = "/assets/data/dogs.jpg";
const IMAGE_EMBEDDING = "/assets/data/dogs_embedding.npy";
const MODEL_DIR = "/model/sam_onnx_quantized_example.onnx";

Running the Inference Pipeline

The onnxModelAPI.tsx helper (located at demo/src/components/helpers/onnxModelAPI.tsx) handles the interaction with ONNX Runtime Web:

import * as ort from "onnxruntime-web";

export async function initModel(modelUrl: string) {
  const session = await ort.InferenceSession.create(modelUrl, {
    executionProviders: ["wasm"],
    graphOptimizationLevel: "all",
  });
  return session;
}

export async function runMask(
  session: ort.InferenceSession,
  imageEmbedding: Float32Array,
  pointCoords: number[][],
  pointLabels: number[]
) {
  const feeds = {
    image_embeddings: new ort.Tensor("float32", imageEmbedding, [1, 256, 64, 64]),
    point_coords: new ort.Tensor("float32", Float32Array.from(pointCoords.flat()), [1, pointCoords.length, 2]),
    point_labels: new ort.Tensor("float32", Float32Array.from(pointLabels), [1, pointLabels.length]),
    has_mask_input: new ort.Tensor("float32", new Float32Array([0]), [1]),
    orig_im_size: new ort.Tensor("float32", new Float32Array([imageHeight, imageWidth]), [2]),
  };
  
  const results = await session.run(feeds);
  return results.masks.data as Float32Array;
}

The Stage.tsx component captures mouse coordinates and converts them to model inputs, while maskUtils.tsx converts the raw mask output into displayable HTML image elements.

Deployment Considerations for Production

When deploying the SAM web integration, ensure your server sends the required cross-origin isolation headers to enable SharedArrayBuffer and multi-threaded WebAssembly execution.

Configure your web server or CDN to include these headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

The demo's webpack configuration in demo/configs/webpack/dev.js (lines 89-94) demonstrates how to set these headers during development. Without these headers, ONNX Runtime Web falls back to single-threaded execution, significantly impacting mask generation performance.

Summary

• Split the workload: Run the SAM image encoder on your Python backend to generate embeddings, then export only the mask decoder to ONNX for browser inference.
• Use the official tools: Leverage SamPredictor in segment_anything/predictor.py for embedding generation and scripts/export_onnx_model.py for ONNX conversion with quantization.
• Frontend integration: Load the quantized ONNX model using onnxruntime-web in React, feed pre-computed embeddings and user prompts to runMask, and render results with utilities from maskUtils.tsx.
• Production requirements: Serve assets with cross-origin isolation headers to enable multi-threaded WebAssembly and optimal performance in the browser.

Frequently Asked Questions

What is the difference between the SAM image encoder and mask decoder?

The image encoder is a heavy Vision Transformer (ViT) that processes the input image once to produce a feature embedding. The mask decoder is a lightweight transformer that takes this embedding plus user prompts (points, boxes) to generate segmentation masks. For web applications, you run the encoder on the backend and export only the decoder to ONNX for browser inference.

Why do I need to pre-compute image embeddings?

Pre-computing embeddings eliminates the need to run the large ViT encoder in the browser, reducing the client-side model size from gigabytes to hundreds of megabytes. The embedding is a fixed-size tensor (1×256×64×64) that the lightweight ONNX decoder uses to generate masks based on user interactions, enabling real-time performance without GPU requirements on the client.

How do I enable multi-threading in the browser for SAM inference?

Multi-threading requires cross-origin isolation headers. Your server must send Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp headers. These enable SharedArrayBuffer, which ONNX Runtime Web uses for WebAssembly threads. Without these headers, the model runs single-threaded, significantly slowing mask generation. The demo's webpack configuration in demo/configs/webpack/dev.js shows how to set these headers for development.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}