# How Shadowbroker Handles Scalability and Load Balancing for Backend Services

> Learn how Shadowbroker handles scalability and load balancing with its three-layer architecture. Discover stateless workers, container scaling, and internal rate limiting.

- Repository: [Shadowbroker/Shadowbroker](https://github.com/BigBodyCobain/Shadowbroker)
- Tags: performance
- Published: 2026-05-07

---

**Shadowbroker achieves scalability and load balancing through a three-layer architecture: stateless FastAPI workers running via Uvicorn, horizontal container scaling managed by Docker Compose or Kubernetes Helm charts, and internal rate limiting via a custom token-bucket algorithm to prevent external API overload.**

Shadowbroker is an open-source intelligence platform built by BigBodyCobain that processes decentralized data feeds through a modular FastAPI backend. According to the source code, the architecture treats scalability as a first-class concern, implementing horizontal scaling, vertical multi-processing, and intelligent back-pressure mechanisms without requiring application-level changes.

## Horizontal Scaling with Container Orchestration

The backend is designed to run as **independent, lightweight FastAPI services** inside Docker containers or Helm-deployed pods, allowing the infrastructure layer to handle load distribution.

### Docker Compose for Local Development

For local environments, [`docker-compose.yml`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/docker-compose.yml) supports rapid horizontal scaling through the `--scale` flag. The configuration exposes the service on a host port while allowing multiple backend replicas to share the load.

```bash

# Spin up three backend containers with built-in round-robin load balancing

docker compose up -d --scale backend=3

```

The frontend communicates with the backend through a single DNS name (`shadowbroker-backend`), enabling Docker’s internal load balancer to distribute requests across all running containers automatically.

### Kubernetes Helm for Production

In production environments, the Helm chart defined in [`helm/chart/values.yaml`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/helm/chart/values.yaml) provides declarative scaling through the `replicaCount` parameter.

```yaml

# helm/chart/values.yaml

replicaCount: 4          # Kubernetes maintains 4 backend pods

resources:
  limits:
    memory: "4Gi"
    cpu: "2000m"

```

Kubernetes’ kube-proxy handles request distribution across pods, while the `replicaCount` field ensures the cluster maintains the desired number of backend instances regardless of node failures or traffic spikes.

## Vertical Scaling via Uvicorn Workers

Inside each container, Shadowbroker leverages **multi-process worker pools** to maximize CPU utilization. In [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py), the application launches via Uvicorn with a configurable number of worker processes.

```python

# backend/main.py – start the FastAPI app with multiple uvicorn workers

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(
        "backend.main:app",
        host="0.0.0.0",
        port=int(os.getenv("BACKEND_PORT", 8000)),
        workers=int(os.getenv("UVICORN_WORKERS", 2)),
    )

```

The `UVICORN_WORKERS` environment variable allows operators to adjust concurrency without modifying container images. This approach keeps each request short and non-blocking while exploiting multi-core CPUs within a single pod.

## Back-Pressure Protection with Rate Limiting

To prevent cascade failures when scaling out, [`backend/limiter.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/limiter.py) implements a **token-bucket rate limiter** that caps outgoing feed fetches. This protects external APIs from excessive calls and ensures the service remains responsive under heavy load.

```python

# backend/limiter.py – token bucket limiter used by data fetchers

class RateLimiter:
    def __init__(self, max_calls: int, period: float):
        self._max_calls = max_calls
        self._period = period
        self._tokens = max_calls
        self._last = time.monotonic()

    async def acquire(self):
        while self._tokens < 1:
            await asyncio.sleep(self._period / self._max_calls)
            self._refill()
        self._tokens -= 1

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self._last
        added = int(elapsed / self._period * self._max_calls)
        self._tokens = min(self._tokens + added, self._max_calls)
        self._last = now

```

The limiter operates per-endpoint and per-feed, ensuring that adding more API workers does not amplify external API rate-limit violations. APScheduler jobs in the data ingestion layer utilize this limiter to manage back-pressure automatically.

## Stateless Architecture and Async Workers

Heavy-weight background operations are isolated from the request-serving path to prevent resource contention. Modules like [`backend/gate_sse.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/gate_sse.py) (which manages Server-Sent Events streams) and files under `backend/services/*` run as **independent async tasks**.

Because these modules are imported lazily and execute in their own event loop tasks, scaling the API layer does not increase the memory footprint of long-running jobs. The stateless route handlers in `backend/routers/*.py` can be called concurrently by any replica, ensuring that horizontal scaling remains efficient and predictable.

## Summary

- **Horizontal scaling**: Use Docker Compose `--scale` or Kubernetes `replicaCount` to distribute traffic across multiple backend containers behind a single DNS endpoint.
- **Vertical scaling**: Configure `UVICORN_WORKERS` in [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py) to spawn multiple processes per container, maximizing CPU utilization for concurrent HTTP requests.
- **Back-pressure control**: Implement the `RateLimiter` class from [`backend/limiter.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/limiter.py) to throttle external API calls and prevent overload when scaling up.
- **Stateless design**: Isolate long-running tasks in [`backend/gate_sse.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/gate_sse.py) and `backend/services/*` to ensure API replicas remain lightweight and independent.

## Frequently Asked Questions

### What orchestration platforms does Shadowbroker support for scaling?

Shadowbroker officially supports Docker Compose for local development and Kubernetes via Helm charts for production deployments. Both configurations allow you to define the number of backend replicas declaratively, utilizing built-in load balancing mechanisms to distribute HTTP requests across instances.

### How does Shadowbroker prevent overwhelming external APIs when scaling up?

The platform implements a token-bucket rate limiter in [`backend/limiter.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/limiter.py) that caps outgoing requests to external data feeds. Because the limiter works per-feed and per-endpoint, increasing the number of Uvicorn workers or container replicas does not linearly increase external API traffic, protecting both the service and third-party endpoints from rate-limit violations.

### Can I adjust the number of workers without rebuilding the container?

Yes. The [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py) entry point reads the `UVICORN_WORKERS` environment variable at runtime to configure the Uvicorn worker pool. You can modify this value in your Docker Compose file or Kubernetes deployment spec and restart the containers to apply changes without rebuilding the image.

### How does the backend handle long-running tasks without blocking API requests?

Long-running operations such as SSE stream management and periodic feed ingestion are isolated in separate modules like [`backend/gate_sse.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/gate_sse.py) and `backend/services/*`. These run as independent async tasks rather than blocking the main request workers, ensuring that scaling the API layer does not compound resource usage for background jobs.