# How to Achieve Horizontal Scaling with Load Balancers and Shared Storage for the Copilot SDK

> Achieve horizontal scaling for the Copilot SDK using load balancers and shared storage. This guide explains how to deploy multiple instances for high availability and even load distribution.

- Repository: [GitHub/copilot-sdk](https://github.com/github/copilot-sdk)
- Tags: how-to-guide
- Published: 2026-08-02

---

**To serve many concurrent users, deploy multiple Copilot CLI server instances behind a load balancer with a shared storage location for session state, allowing any instance to resume any session for even load distribution and high availability.**

The `github/copilot-sdk` repository provides enterprise-grade infrastructure to scale Copilot services beyond a single instance. When traffic demands exceed the capacity of one server, implementing horizontal scaling with load balancers and shared storage for Copilot SDK ensures session continuity and fault tolerance across a distributed pool of CLI processes.

## Architecture Overview

A horizontally scaled deployment consists of independent CLI servers behind a load balancer, with all instances pointing to a shared network filesystem. According to [`docs/setup/scaling.md`](https://github.com/github/copilot-sdk/blob/main/docs/setup/scaling.md), this topology allows any server in the pool to handle any user request because session data is decoupled from the compute nodes.

The standard configuration includes:
- **CLI Server Pool**: Multiple processes listening on distinct ports (e.g., `:4321`, `:4322`, `:4323`)
- **Load Balancer**: Distributes incoming traffic via round-robin or sticky-session algorithms
- **Shared Storage**: Network File System (NFS) or cloud-mounted volumes at `~/.copilot/session-state/`

The [`go/session_fs_provider.go`](https://github.com/github/copilot-sdk/blob/main/go/session_fs_provider.go) file implements the filesystem-based session storage that makes this architecture possible. By placing this directory on shared storage, every CLI instance gains read/write access to the same session files.

## Configure Shared Storage for Session State

Session persistence requires mounting a shared volume to the CLI's state directory. The Copilot CLI writes session data to `~/.copilot/session-state/`, which must be accessible to every server in the pool.

For production deployments, mount persistent volumes using:
- **NFS**: Traditional network-attached storage for on-premise clusters
- **Cloud File Systems**: AWS EFS, Azure Files, or Google Cloud Filestore
- **Kubernetes PVCs**: Persistent Volume Claims that mount to each pod

Ensure the filesystem supports concurrent read/write operations, as multiple CLI servers may access the same session files simultaneously during high-load scenarios.

## Deploy CLI Servers in Headless Mode

Each CLI instance must run as a headless server binding to `0.0.0.0` so the load balancer can route traffic. Start servers with distinct ports as documented in [`docs/setup/backend-services.md`](https://github.com/github/copilot-sdk/blob/main/docs/setup/backend-services.md):

```bash
copilot --headless --host 0.0.0.0 --port 4321

```

Backend SDK clients connect using `RuntimeConnection.forUri()`:

```typescript
import { CopilotClient, RuntimeConnection } from "@github/copilot-sdk";

const client = new CopilotClient({
    connection: RuntimeConnection.forUri("cli-server-1:4321"),
    mode: "empty",
});

```

## Load Balancer Configuration Strategies

The scaling guide in [`docs/setup/scaling.md`](https://github.com/github/copilot-sdk/blob/main/docs/setup/scaling.md) outlines two routing approaches:

**Sticky Sessions (Session Affinity)**
- Pins a user to a specific CLI server using IP hashing or cookie-based routing
- Eliminates the need for shared storage but risks uneven load distribution if users have variable activity levels

**Shared Storage with Round-Robin**
- Routes requests to any available server via round-robin or least-connections algorithms
- Requires networked filesystems but achieves true load balancing and high availability

For external reverse proxies, configure NGINX or HAProxy to forward `localhost:<port>` requests to the CLI pool. In Kubernetes, a `Service` resource automatically load-balances traffic across pods.

## Client-Side Routing Logic

Your application can implement custom load balancing using the `CLILoadBalancer` pattern from the scaling guide. This TypeScript class provides both round-robin and sticky routing based on user ID hashing:

```typescript
class CLILoadBalancer {
    private servers: string[];
    private idx = 0;

    constructor(servers: string[]) {
        this.servers = servers;
    }

    // Simple round-robin
    next(): string {
        const srv = this.servers[this.idx];
        this.idx = (this.idx + 1) % this.servers.length;
        return srv;
    }

    // Sticky routing based on userId
    forUser(userId: string): string {
        const hash = this.hashCode(userId);
        return this.servers[hash % this.servers.length];
    }

    private hashCode(s: string): number {
        let h = 0;
        for (let i = 0; i < s.length; i++) {
            h = (h << 5) - h + s.charCodeAt(i);
            h |= 0;
        }
        return Math.abs(h);
    }
}

```

Integrate the load balancer with your HTTP framework:

```typescript
const lb = new CLILoadBalancer([
    "cli-1:4321",
    "cli-2:4321",
    "cli-3:4321",
]);

app.post("/chat", async (req, res) => {
    const server = lb.forUser(req.user.id);
    const client = new CopilotClient({
        connection: RuntimeConnection.forUri(server),
        mode: "empty",
    });
    
    const session = await client.createSession({
        sessionId: `user-${req.user.id}-${Date.now()}`,
        model: "gpt-5.4",
    });
    
    const reply = await session.sendAndWait({ prompt: req.body.message });
    res.json({ content: reply?.data.content });
});

```

## Production Deployment Examples

### Docker Compose with Shared Volume

```yaml
version: "3.8"
services:
  copilot-cli:
    image: copilot-cli:latest
    command: ["--headless", "--host", "0.0.0.0", "--port", "4321"]
    environment:
      - COPILOT_GITHUB_TOKEN=${COPILOT_GITHUB_TOKEN}
    ports:
      - "4321:4321"
    volumes:
      - session-data:/root/.copilot/session-state

  api:
    build: .
    environment:
      - CLI_URL=copilot-cli:4321
    depends_on:
      - copilot-cli
    ports:
      - "3000:3000"

volumes:
  session-data:

```

### Kubernetes Deployment with Persistent Storage

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: copilot-cli
spec:
  replicas: 3
  selector:
    matchLabels:
      app: copilot-cli
  template:
    metadata:
      labels:
        app: copilot-cli
    spec:
      containers:
      - name: copilot-cli
        image: your-registry/copilot-cli:latest
        args: ["--headless", "--host", "0.0.0.0", "--port", "4321"]
        env:
        - name: COPILOT_GITHUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: copilot-secrets
              key: github-token
        ports:
        - containerPort: 4321
        volumeMounts:
        - name: session-state
          mountPath: /root/.copilot/session-state
      volumes:
      - name: session-state
        persistentVolumeClaim:
          claimName: copilot-sessions-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: copilot-cli
spec:
  selector:
    app: copilot-cli
  ports:
  - port: 4321
    targetPort: 4321

```

## Summary

- **Horizontal scaling** requires running multiple CLI server instances behind a load balancer to distribute traffic across your Copilot SDK deployment.
- **Shared storage** at `~/.copilot/session-state/` is mandatory for stateless routing, allowing any server to resume any session.
- **Headless mode** enables the CLI to accept network connections via `--headless --host 0.0.0.0` flags.
- **Client routing** can be delegated to external load balancers or handled programmatically using the `CLILoadBalancer` pattern.
- **Production readiness** demands persistent volumes, health checks, graceful shutdowns, and secret management for GitHub tokens.

## Frequently Asked Questions

### What is the difference between sticky sessions and shared storage for Copilot SDK?

Sticky sessions pin a specific user to a single CLI server instance for the duration of their session, which eliminates the need for shared filesystems but can lead to uneven load distribution. Shared storage allows any CLI server to handle any request, achieving true horizontal scaling but requiring a networked filesystem like NFS or cloud storage mounted at `~/.copilot/session-state/`.

### How should I secure GitHub tokens when scaling horizontally?

Store sensitive credentials like `COPILOT_GITHUB_TOKEN` in secret managers rather than environment variables or configuration files. In Kubernetes, use `secretKeyRef` to inject tokens from Secrets into pod environments. The [`docs/setup/multi-tenancy.md`](https://github.com/github/copilot-sdk/blob/main/docs/setup/multi-tenancy.md) file provides additional guidance on per-user token isolation in multi-tenant deployments.

### Can I use cloud storage instead of NFS for the session state directory?

Yes, cloud-native filesystems such as AWS EFS, Azure Files, or Google Cloud Filestore work as replacements for traditional NFS. Ensure the storage solution supports concurrent access and maintains POSIX file semantics, as the [`go/session_fs_provider.go`](https://github.com/github/copilot-sdk/blob/main/go/session_fs_provider.go) implementation relies on standard filesystem operations.

### What monitoring is required for a production horizontal scaling deployment?

Monitor active session counts per CLI instance, request latency from the load balancer, error rates on the `/health` endpoints, and disk I/O on the shared storage volume. The production checklist in [`docs/setup/scaling.md`](https://github.com/github/copilot-sdk/blob/main/docs/setup/scaling.md) recommends implementing graceful shutdown handlers to allow in-flight requests to complete before terminating instances during scale-down events.