How Shadowbroker Handles Scalability and Load Balancing for Backend Services

Shadowbroker achieves scalability and load balancing through a three-layer architecture: stateless FastAPI workers running via Uvicorn, horizontal container scaling managed by Docker Compose or Kubernetes Helm charts, and internal rate limiting via a custom token-bucket algorithm to prevent external API overload.

Shadowbroker is an open-source intelligence platform built by BigBodyCobain that processes decentralized data feeds through a modular FastAPI backend. According to the source code, the architecture treats scalability as a first-class concern, implementing horizontal scaling, vertical multi-processing, and intelligent back-pressure mechanisms without requiring application-level changes.

Horizontal Scaling with Container Orchestration

The backend is designed to run as independent, lightweight FastAPI services inside Docker containers or Helm-deployed pods, allowing the infrastructure layer to handle load distribution.

Docker Compose for Local Development

For local environments, docker-compose.yml supports rapid horizontal scaling through the --scale flag. The configuration exposes the service on a host port while allowing multiple backend replicas to share the load.

# Spin up three backend containers with built-in round-robin load balancing

docker compose up -d --scale backend=3

The frontend communicates with the backend through a single DNS name (shadowbroker-backend), enabling Docker’s internal load balancer to distribute requests across all running containers automatically.

Kubernetes Helm for Production

In production environments, the Helm chart defined in helm/chart/values.yaml provides declarative scaling through the replicaCount parameter.

# helm/chart/values.yaml

replicaCount: 4          # Kubernetes maintains 4 backend pods

resources:
  limits:
    memory: "4Gi"
    cpu: "2000m"

Kubernetes’ kube-proxy handles request distribution across pods, while the replicaCount field ensures the cluster maintains the desired number of backend instances regardless of node failures or traffic spikes.

Vertical Scaling via Uvicorn Workers

Inside each container, Shadowbroker leverages multi-process worker pools to maximize CPU utilization. In backend/main.py, the application launches via Uvicorn with a configurable number of worker processes.

# backend/main.py – start the FastAPI app with multiple uvicorn workers

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(
        "backend.main:app",
        host="0.0.0.0",
        port=int(os.getenv("BACKEND_PORT", 8000)),
        workers=int(os.getenv("UVICORN_WORKERS", 2)),
    )

The UVICORN_WORKERS environment variable allows operators to adjust concurrency without modifying container images. This approach keeps each request short and non-blocking while exploiting multi-core CPUs within a single pod.

Back-Pressure Protection with Rate Limiting

To prevent cascade failures when scaling out, backend/limiter.py implements a token-bucket rate limiter that caps outgoing feed fetches. This protects external APIs from excessive calls and ensures the service remains responsive under heavy load.

# backend/limiter.py – token bucket limiter used by data fetchers

class RateLimiter:
    def __init__(self, max_calls: int, period: float):
        self._max_calls = max_calls
        self._period = period
        self._tokens = max_calls
        self._last = time.monotonic()

    async def acquire(self):
        while self._tokens < 1:
            await asyncio.sleep(self._period / self._max_calls)
            self._refill()
        self._tokens -= 1

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self._last
        added = int(elapsed / self._period * self._max_calls)
        self._tokens = min(self._tokens + added, self._max_calls)
        self._last = now

The limiter operates per-endpoint and per-feed, ensuring that adding more API workers does not amplify external API rate-limit violations. APScheduler jobs in the data ingestion layer utilize this limiter to manage back-pressure automatically.

Stateless Architecture and Async Workers

Heavy-weight background operations are isolated from the request-serving path to prevent resource contention. Modules like backend/gate_sse.py (which manages Server-Sent Events streams) and files under backend/services/* run as independent async tasks.

Because these modules are imported lazily and execute in their own event loop tasks, scaling the API layer does not increase the memory footprint of long-running jobs. The stateless route handlers in backend/routers/*.py can be called concurrently by any replica, ensuring that horizontal scaling remains efficient and predictable.

Summary

• Horizontal scaling: Use Docker Compose --scale or Kubernetes replicaCount to distribute traffic across multiple backend containers behind a single DNS endpoint.
• Vertical scaling: Configure UVICORN_WORKERS in backend/main.py to spawn multiple processes per container, maximizing CPU utilization for concurrent HTTP requests.
• Back-pressure control: Implement the RateLimiter class from backend/limiter.py to throttle external API calls and prevent overload when scaling up.
• Stateless design: Isolate long-running tasks in backend/gate_sse.py and backend/services/* to ensure API replicas remain lightweight and independent.

Frequently Asked Questions

What orchestration platforms does Shadowbroker support for scaling?

Shadowbroker officially supports Docker Compose for local development and Kubernetes via Helm charts for production deployments. Both configurations allow you to define the number of backend replicas declaratively, utilizing built-in load balancing mechanisms to distribute HTTP requests across instances.

How does Shadowbroker prevent overwhelming external APIs when scaling up?

The platform implements a token-bucket rate limiter in backend/limiter.py that caps outgoing requests to external data feeds. Because the limiter works per-feed and per-endpoint, increasing the number of Uvicorn workers or container replicas does not linearly increase external API traffic, protecting both the service and third-party endpoints from rate-limit violations.

Can I adjust the number of workers without rebuilding the container?

Yes. The backend/main.py entry point reads the UVICORN_WORKERS environment variable at runtime to configure the Uvicorn worker pool. You can modify this value in your Docker Compose file or Kubernetes deployment spec and restart the containers to apply changes without rebuilding the image.

How does the backend handle long-running tasks without blocking API requests?

Long-running operations such as SSE stream management and periodic feed ingestion are isolated in separate modules like backend/gate_sse.py and backend/services/*. These run as independent async tasks rather than blocking the main request workers, ensuring that scaling the API layer does not compound resource usage for background jobs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}