# How to Integrate SAM into a Web Application: A Complete Production Guide

> Integrate Segment Anything Model SAM into your web app. Export mask decoder to ONNX, pre compute embeddings, and run inference in browser with ONNX Runtime Web and React. A complete production guide.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: how-to-guide
- Published: 2026-03-07

---

**You can integrate SAM into a web application by exporting the mask decoder to ONNX, pre-computing image embeddings on a Python backend, and running interactive inference in the browser using ONNX Runtime Web with React.**

The **facebookresearch/segment-anything** repository provides a complete pipeline for deploying the Segment Anything Model (SAM) in browser-based applications. This guide walks through the production-ready architecture used in the official demo, covering Python backend setup, ONNX export, and React frontend integration.

## Architecture Overview for Web Deployment

Integrating SAM into a web application requires splitting the workload between server and client. The heavy image encoder runs once on the backend to generate embeddings, while the lightweight mask decoder runs interactively in the browser.

The architecture follows three distinct stages:

1. **Python Backend** – Loads a SAM checkpoint via `sam_model_registry`, generates image embeddings using `SamPredictor`, and exports the mask decoder to ONNX format.
2. **ONNX Optimization** – Quantizes the exported model to reduce file size for faster web delivery.
3. **React Frontend** – Loads the ONNX model with `onnxruntime-web`, ingests pre-computed embeddings, and performs real-time mask prediction based on user interactions.

## Setting Up the Python Environment

Begin by cloning the repository and installing dependencies required for model inference and ONNX export.

```bash
git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .
pip install opencv-python pycocotools matplotlib onnxruntime onnx

```

Download a SAM checkpoint. The repository supports three model types: `vit_h` (largest), `vit_l`, and `vit_b` (smallest).

```bash
wget https://dl.fbaipublicfiles.com/segment-anything/sam_vit_h_4b8939.pth -O sam_vit_h.pth

```

## Generating Image Embeddings on the Backend

The `SamPredictor` class in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) handles image preprocessing and embedding generation. The `set_image()` method applies `ResizeLongestSide` preprocessing (lines 34-59), while `get_image_embedding()` extracts the feature tensor (lines 45-56).

Create a script to generate and save embeddings for frontend consumption:

```python

# generate_embedding.py

import cv2
import numpy as np
from segment_anything import sam_model_registry, SamPredictor

# Load model

checkpoint = "sam_vit_h.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=checkpoint)
sam.to(device="cuda")

# Initialize predictor

predictor = SamPredictor(sam)

# Load and process image

image = cv2.imread("src/assets/data/dogs.jpg")
predictor.set_image(image, image_format="BGR")

# Extract and save embedding

embedding = predictor.get_image_embedding().cpu().numpy()
np.save("demo/src/assets/data/dogs_embedding.npy", embedding.squeeze())
print("Embedding saved successfully")

```

## Exporting the Mask Decoder to ONNX

The [`scripts/export_onnx_model.py`](https://github.com/facebookresearch/segment-anything/blob/main/scripts/export_onnx_model.py) script exports the prompt encoder and mask decoder to ONNX format, enabling browser-based inference. The script wraps these components in `SamOnnxModel` and performs Torch-ONNX export (lines 97-160). Dynamic quantization (lines 87-99) reduces model size for web deployment.

Run the export with quantization:

```bash
python scripts/export_onnx_model.py \
  --checkpoint sam_vit_h.pth \
  --model-type vit_h \
  --output demo/model/sam_onnx_quantized_example.onnx \
  --return-single-mask \
  --opset 17 \
  --quantize-out demo/model/sam_onnx_quantized_example_quant.onnx

```

The quantized ONNX file (`*_quant.onnx`) is typically 100-400MB depending on the model type, suitable for CDN distribution.

## Integrating the Frontend with React and ONNX Runtime Web

The demo application in [`demo/src/App.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/demo/src/App.tsx) demonstrates the complete integration pattern. The frontend loads the quantized ONNX model using `onnxruntime-web` and performs inference using pre-computed embeddings.

### Configuring Asset Paths

Update the constants in [`demo/src/App.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/demo/src/App.tsx) to point to your generated assets:

```tsx
const IMAGE_PATH = "/assets/data/dogs.jpg";
const IMAGE_EMBEDDING = "/assets/data/dogs_embedding.npy";
const MODEL_DIR = "/model/sam_onnx_quantized_example.onnx";

```

### Running the Inference Pipeline

The [`onnxModelAPI.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/onnxModelAPI.tsx) helper (located at [`demo/src/components/helpers/onnxModelAPI.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/demo/src/components/helpers/onnxModelAPI.tsx)) handles the interaction with ONNX Runtime Web:

```tsx
import * as ort from "onnxruntime-web";

export async function initModel(modelUrl: string) {
  const session = await ort.InferenceSession.create(modelUrl, {
    executionProviders: ["wasm"],
    graphOptimizationLevel: "all",
  });
  return session;
}

export async function runMask(
  session: ort.InferenceSession,
  imageEmbedding: Float32Array,
  pointCoords: number[][],
  pointLabels: number[]
) {
  const feeds = {
    image_embeddings: new ort.Tensor("float32", imageEmbedding, [1, 256, 64, 64]),
    point_coords: new ort.Tensor("float32", Float32Array.from(pointCoords.flat()), [1, pointCoords.length, 2]),
    point_labels: new ort.Tensor("float32", Float32Array.from(pointLabels), [1, pointLabels.length]),
    has_mask_input: new ort.Tensor("float32", new Float32Array([0]), [1]),
    orig_im_size: new ort.Tensor("float32", new Float32Array([imageHeight, imageWidth]), [2]),
  };
  
  const results = await session.run(feeds);
  return results.masks.data as Float32Array;
}

```

The [`Stage.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/Stage.tsx) component captures mouse coordinates and converts them to model inputs, while [`maskUtils.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/maskUtils.tsx) converts the raw mask output into displayable HTML image elements.

## Deployment Considerations for Production

When deploying the SAM web integration, ensure your server sends the required **cross-origin isolation** headers to enable `SharedArrayBuffer` and multi-threaded WebAssembly execution.

Configure your web server or CDN to include these headers:

```

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

```

The demo's webpack configuration in [`demo/configs/webpack/dev.js`](https://github.com/facebookresearch/segment-anything/blob/main/demo/configs/webpack/dev.js) (lines 89-94) demonstrates how to set these headers during development. Without these headers, ONNX Runtime Web falls back to single-threaded execution, significantly impacting mask generation performance.

## Summary

- **Split the workload**: Run the SAM image encoder on your Python backend to generate embeddings, then export only the mask decoder to ONNX for browser inference.
- **Use the official tools**: Leverage `SamPredictor` in [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) for embedding generation and [`scripts/export_onnx_model.py`](https://github.com/facebookresearch/segment-anything/blob/main/scripts/export_onnx_model.py) for ONNX conversion with quantization.
- **Frontend integration**: Load the quantized ONNX model using `onnxruntime-web` in React, feed pre-computed embeddings and user prompts to `runMask`, and render results with utilities from [`maskUtils.tsx`](https://github.com/facebookresearch/segment-anything/blob/main/maskUtils.tsx).
- **Production requirements**: Serve assets with cross-origin isolation headers to enable multi-threaded WebAssembly and optimal performance in the browser.

## Frequently Asked Questions

### What is the difference between the SAM image encoder and mask decoder?

The **image encoder** is a heavy Vision Transformer (ViT) that processes the input image once to produce a feature embedding. The **mask decoder** is a lightweight transformer that takes this embedding plus user prompts (points, boxes) to generate segmentation masks. For web applications, you run the encoder on the backend and export only the decoder to ONNX for browser inference.

### Why do I need to pre-compute image embeddings?

Pre-computing embeddings eliminates the need to run the large ViT encoder in the browser, reducing the client-side model size from gigabytes to hundreds of megabytes. The embedding is a fixed-size tensor (1×256×64×64) that the lightweight ONNX decoder uses to generate masks based on user interactions, enabling real-time performance without GPU requirements on the client.

### How do I enable multi-threading in the browser for SAM inference?

Multi-threading requires **cross-origin isolation** headers. Your server must send `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp` headers. These enable `SharedArrayBuffer`, which ONNX Runtime Web uses for WebAssembly threads. Without these headers, the model runs single-threaded, significantly slowing mask generation. The demo's webpack configuration in [`demo/configs/webpack/dev.js`](https://github.com/facebookresearch/segment-anything/blob/main/demo/configs/webpack/dev.js) shows how to set these headers for development.