Setting Up a Reverse Proxy for Open Notebook: A Complete Guide

You only need to expose port 8502 (the Next.js frontend) to the public internet, because Open Notebook's frontend automatically rewrites all /api/* requests to the FastAPI backend internally.

Open Notebook is an open-source, three-tier application consisting of a Next.js frontend, FastAPI backend, and SurrealDB database. When deploying this system behind a reverse proxy, understanding how the frontend handles API routing is critical to securing your deployment. This guide walks through the architecture, environment variables, and complete configuration examples for Nginx, Caddy, and Traefik.

Understanding the Three-Tier Architecture

Open Notebook runs as three distinct services with specific networking requirements:

• Frontend (Next.js): Runs on port 8502 and serves the user interface, static assets, and handles all client-side routing. This is the only port that should be exposed to the public internet.
• Backend (FastAPI): Runs on port 5055 and provides REST endpoints under /api/*. It should never be directly exposed; instead, it receives traffic through the frontend's internal proxy.
• Database (SurrealDB): Runs on port 8000 and stores notebooks, sources, and embeddings. It is accessed exclusively by the backend service.

According to the architecture documentation in open_notebook/CLAUDE.md, this design ensures that external traffic never directly hits your API or database layers, significantly reducing the attack surface.

Why You Only Need to Expose Port 8502

Starting with version 1.1, Open Notebook uses Next.js rewrites to handle all API traffic internally. This eliminates the need for complex proxy rules that route /api/* paths separately.

How Next.js Handles API Rewrites

In frontend/next.config.ts, the configuration defines an internal rewrite that forwards any request matching /api/* to the backend service:

// frontend/next.config.ts (lines 15-31)
async rewrites() {
  return [
    {
      source: '/api/:path*',
      destination: `${process.env.INTERNAL_API_URL || 'http://localhost:5055'}/api/:path*`,
    },
  ];
}

This means when a user visits https://your-domain.com/api/health, the Next.js server (port 8502) receives the request and internally proxies it to http://localhost:5055/api/health. Your reverse proxy only needs to route to port 8502, and Next.js handles the rest.

Essential Environment Variables

Configure these variables in your docker-compose.yml or container environment to ensure proper proxy behavior:

Variable	Purpose	Example Value
API_URL	The public URL the frontend uses to contact the backend (must include scheme, no /api suffix)	https://notebook.example.com
INTERNAL_API_URL	URL the Next.js server uses for internal API rewrites (only needed for multi-container setups)	http://api-service:5055
OPEN_NOTEBOOK_ENCRYPTION_KEY	Secret key for encrypting stored API credentials	change-me-to-a-secret-string

The API_URL variable is critical for client-side rendering. If unset, the frontend attempts auto-detection, but explicitly setting it to your HTTPS domain prevents mixed-content errors.

Reverse Proxy Configuration Examples

All examples assume your Open Notebook container is named open-notebook and runs on the same Docker network as your proxy.

Nginx

Nginx requires standard proxy headers and extended timeouts for Open Notebook's long-running operations (like AI transformations and podcast generation):

server {
    listen 443 ssl http2;
    server_name notebook.example.com;

    ssl_certificate /etc/nginx/ssl/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/privkey.pem;

    # Match Next.js max body size (100MB)

    client_max_body_size 100M;

    location / {
        proxy_pass http://open-notebook:8502;
        proxy_http_version 1.1;
        
        # Essential headers

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_cache_bypass $http_upgrade;
        
        # Extended timeouts for long operations

        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}

Caddy

Caddy provides automatic HTTPS and requires explicit timeout settings to handle long-running tasks:

notebook.example.com {
    reverse_proxy open-notebook:8502 {
        transport http {
            read_timeout 600s
            write_timeout 600s
        }
    }
}

Caddy automatically handles the Upgrade headers for WebSocket support and manages TLS certificates without additional configuration.

Traefik

For Traefik users, use Docker labels to configure the router and service:

services:
  open-notebook:
    image: lfnovo/open_notebook:v1-latest
    environment:
      - API_URL=https://notebook.example.com
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.notebook.rule=Host(`notebook.example.com`)"
      - "traefik.http.routers.notebook.entrypoints=websecure"
      - "traefik.http.routers.notebook.tls.certresolver=myresolver"
      - "traefik.http.services.notebook.loadbalancer.server.port=8502"

Traefik's default timeouts are typically sufficient, but you can add middleware to increase them if needed for specific long-running endpoints.

Docker Compose Deployment Pattern

Here is a complete single-container deployment pattern with Nginx as the reverse proxy:

services:
  open-notebook:
    image: lfnovo/open_notebook:v1-latest
    ports:
      - "8502:8502"   # Exposed to reverse proxy only

    environment:
      - API_URL=https://notebook.example.com
      - INTERNAL_API_URL=http://localhost:5055
      - OPEN_NOTEBOOK_ENCRYPTION_KEY=${OPEN_NOTEBOOK_ENCRYPTION_KEY}
    volumes:
      - ./notebook_data:/app/data
    depends_on:
      - surrealdb

  surrealdb:
    image: surrealdb/surrealdb:latest
    volumes:
      - ./surreal_data:/data
    command: start --user root --pass root file:/data/notebook.db

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./ssl:/etc/nginx/ssl:ro
    depends_on:
      - open-notebook

Note that port 5055 is not mapped to the host in this configuration because the FastAPI backend is accessed internally via the Docker network. The INTERNAL_API_URL uses localhost only when running the backend within the same container; for multi-container setups, use the service name (e.g., http://open-notebook:5055).

Troubleshooting Common Reverse Proxy Issues

Symptom	Cause	Solution
"Unable to connect to server"	API_URL missing or using wrong scheme	Set API_URL=https://your-domain.com (no trailing /api)
502 Bad Gateway	Proxy cannot reach container on port 8502	Verify container is running (docker ps) and both services share the same Docker network
Mixed Content errors	Frontend loaded over HTTPS but API_URL uses HTTP	Ensure API_URL uses https:// scheme
413 Payload Too Large	Proxy body size limit under 100MB	Increase client_max_body_size in Nginx or equivalent in Caddy/Traefik to match frontend/next.config.ts settings
Timeout after 30 seconds	Proxy default timeout shorter than Open Notebook operations	Set proxy_read_timeout and proxy_send_timeout to 600s (10 minutes) for long-running transformations

The timeout issue is particularly common with AI-powered features like document transformation and podcast generation, which can take several minutes to complete.

Summary

• Expose only port 8502 (Next.js frontend) to your reverse proxy; the backend on port 5055 should remain internal.
• Next.js automatically proxies /api/* requests to the FastAPI backend using the INTERNAL_API_URL rewrite rule defined in frontend/next.config.ts.
• Set API_URL to your public HTTPS domain to prevent mixed-content errors and ensure proper client-side routing.
• Configure 600-second timeouts in your proxy to accommodate long-running AI operations.
• Allow 100MB uploads by setting client_max_body_size (Nginx) or equivalent to match the Next.js configuration.

Frequently Asked Questions

Do I need to expose the FastAPI backend port 5055 to the internet?

No. You should only expose port 8502 (the Next.js frontend). The frontend handles API requests internally through Next.js rewrites defined in frontend/next.config.ts, forwarding /api/* paths to the backend service within the container network. Exposing port 5055 directly creates an unnecessary security risk.

Why am I getting "Unable to connect to server" after setting up HTTPS?

This typically occurs when the API_URL environment variable is missing or uses http:// instead of https://. The frontend needs API_URL to generate correct absolute URLs for client-side API calls. Ensure it matches your public domain exactly, including the https:// scheme, and contains no /api suffix.

How do I handle large file uploads through the reverse proxy?

Open Notebook supports uploads up to 100MB by default, defined in frontend/next.config.ts. You must configure your reverse proxy to match this limit. For Nginx, set client_max_body_size 100M; in your server block. Caddy and Traefik have similar configuration options to prevent 413 Payload Too Large errors.

What timeout settings are required for AI features?

Configure your reverse proxy with 600-second (10-minute) read and send timeouts. Open Notebook performs long-running operations like document transformation and podcast generation that can exceed standard 30-second proxy defaults. In Nginx, set proxy_read_timeout 600s; and proxy_send_timeout 600s; within your location block.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}