# Shadowbroker Backend Microservices and Components: Complete Architecture Guide

> Explore the Shadowbroker backend architecture. Understand its FastAPI service plane, Python microservices, and how they deliver real-time geospatial intelligence via a unified API.

- Repository: [Shadowbroker/Shadowbroker](https://github.com/BigBodyCobain/Shadowbroker)
- Tags: architecture
- Published: 2026-05-07

---

**Shadowbroker's backend is a modular FastAPI service plane that stitches together specialized Python microservices to fetch, enrich, store, and expose real-time geospatial intelligence through a unified API.**

The BigBodyCobain/Shadowbroker repository implements a horizontally scalable architecture designed for intelligence operations. This backend separates concerns into distinct components—ranging from data ingestion schedulers to decentralized mesh routers—enabling independent deployment while maintaining tight integration through shared telemetry caches and cryptographic privacy controls. Understanding these microservices is essential for operators extending the platform's capabilities or deploying nodes across distributed environments.

## FastAPI Entry Point and Router Layer

The [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py) file serves as the application bootstrap, creating the FastAPI instance, configuring CORS middleware, and registering the modular router layer. During startup, the system validates environment secrets and lazily imports optional routers via the `_load_optional_router` pattern to keep the service plane lightweight.

The router layer in `backend/routers/*.py` exposes distinct API namespaces that map to functional domains:

- **`/health`** – System health checks and uptime metrics
- **`/cctv`** – Live camera feed aggregation and GeoJSON tile generation
- **`/radio`** – SDR tuning and radio intercept endpoints
- **`/sigint`** – Signal intelligence queries via Shodan and scanner aggregation
- **`/mesh_*`** – Decentralized peer synchronization and messaging
- **`/ai_intel`** – OpenClaw agentic AI command channel
- **`/sar`** – Synthetic Aperture Radar catalog services
- **`/wormhole`** – Wormhole relay control for private transport

Each router operates as an isolated component, allowing developers to disable entire functional domains by omitting the router file without affecting core system stability.

To verify router registration and system health, query the health endpoint:

```python
import httpx

resp = httpx.get("http://localhost:8000/health")
print(resp.json())

# → {"status":"ok","version":"0.9.75","uptime":123.4}

```

## Data Ingestion and Scheduler Services

The [`services/data_fetcher.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/data_fetcher.py) module orchestrates the backend's real-time intelligence pipeline using APScheduler to manage fast-tier (60-second) and slow-tier ingestion jobs. This component continuously pulls external telemetry—including ADS-B flight data, AIS vessel positions, satellite TLEs, and GDELT news feeds—and caches the latest snapshots in memory via [`services/telemetry.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/telemetry.py).

Feed-specific adapters in the `services/` directory provide thin wrappers for external APIs:

- **[`ais_stream.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/ais_stream.py)** – WebSocket client for aisstream.io vessel tracking
- **[`shodan_connector.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/shodan_connector.py)** – Device search and internet exposure queries
- **[`tinygs_fetcher.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/tinygs_fetcher.py)** – LoRa satellite position retrieval from TinyGS
- **[`satnogs_fetcher.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/satnogs_fetcher.py)** – Amateur ground station data from SatNOGS
- **[`kiwisdr_fetcher.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/kiwisdr_fetcher.py)** – Public SDR receiver catalogs
- **[`carrier_tracker.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/carrier_tracker.py)** – Carrier strike group estimation via GDELT news analysis

To manually seed caches for testing or development, invoke the data fetcher directly:

```python
from backend.services.data_fetcher import start_scheduler, seed_startup_caches

# Initialise caches with latest snapshots

seed_startup_caches()

# Kick off APScheduler jobs (runs every 60s for fast feeds)

start_scheduler()

```

Accessing cached AIS data programmatically:

```python
from backend.services.ais_stream import start_ais_stream
from backend.services.data_fetcher import get_latest_data

# Start websocket stream in background thread

start_ais_stream()

# Read vessel by MMSI from global cache

ais_snapshot = get_latest_data()["ais"]
print(ais_snapshot.get("123456789"))   # → {'lat': 51.5, 'lon': -0.12, ...}

```

## Mesh and InfoNet Networking Stack

The `services/mesh/` directory contains the decentralized messaging layer that powers the InfoNet test-net and Wormhole relay systems. This stack enables cryptographically signed, tier-gated peer-to-peer communication across public, private, and heavy transport lanes.

Core mesh components include:

- **[`mesh_router.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/mesh_router.py)** – Routing logic for peer-to-peer message propagation
- **[`mesh_wormhole_contacts.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/mesh_wormhole_contacts.py)** – Pre-key lookup and contact discovery
- **[`mesh_private_transport_manager.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/mesh_private_transport_manager.py)** – Onion and RNS (Reticulum) transport management
- **[`mesh_private_release_worker.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/mesh_private_release_worker.py)** – Private-lane release handling
- **[`mesh_privacy_policy.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/mesh_privacy_policy.py)** – Transport-tier enforcement and privacy gating

Peers discover each other via bootstrap manifests and negotiate encrypted transports using X25519 and Ed25519 primitives provided by the privacy-core integration.

To inspect private-lane peers:

```python
from backend.services.mesh.mesh_peer_store import PeerStore, DEFAULT_PEER_STORE_PATH

store = PeerStore(DEFAULT_PEER_STORE_PATH)
store.load()
print([p for p in store.peers if p.transport == "onion"])

```

## AI Integration: The OpenClaw Channel

The OpenClaw subsystem provides a signed, tier-gated API surface for external LLM agents to interact with the map data layer. Implemented across [`services/openclaw_channel.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/openclaw_channel.py), [`openclaw_bridge.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/openclaw_bridge.py), and [`openclaw_watchdog.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/openclaw_watchdog.py), this channel accepts HMAC-signed HTTP commands that allow AI agents to place pins, control viewports, and read telemetry.

The signing protocol requires timestamped, nonced payloads verified against a pre-shared secret, preventing replay attacks and ensuring command authenticity.

Sending a signed OpenClaw command:

```python
import hmac, hashlib, json, time, uuid, httpx

SECRET = "••••‑your‑hmac‑secret‑••••"
path = "/api/ai/channel/command"
body = json.dumps({"cmd": "pin_place", "args": {"lat": 37.7749, "lon": -122.4194, "category": "threat"}})
ts = str(int(time.time()))
nonce = uuid.uuid4().hex
payload = f"POST|{path}|{ts}|{nonce}|{hashlib.sha256(body.encode()).hexdigest()}"
sig = hmac.new(SECRET.encode(), payload.encode(), hashlib.sha256).hexdigest()

headers = {
    "X-SB-Timestamp": ts,
    "X-SB-Nonce": nonce,
    "X-SB-Signature": sig,
    "Content-Type": "application/json",
}
resp = httpx.post(f"http://localhost:8000{path}", data=body, headers=headers)
print(resp.json())

```

## Specialized Intelligence Modules

Beyond core data ingestion, the backend includes specialized services for specific intelligence disciplines:

**CCTV Pipeline** ([`services/cctv_pipeline.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/cctv_pipeline.py)) aggregates live camera feeds across multiple protocols (MJPEG, HLS, embedded streams) and normalizes them into GeoJSON tiles for frontend rendering.

**Radio and SIGINT** ([`services/radio_intercept.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/radio_intercept.py), [`services/sigint_bridge.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/sigint_bridge.py)) interface with KiwiSDR instances, police/fire scanners, Meshtastic MQTT brokers, and APRS-IS networks to capture and correlate signal intelligence.

**SAR Services** (`services/sar/`) handle Synthetic Aperture Radar catalog queries and anomaly detection feeds, providing all-weather surveillance capabilities independent of optical sensors.

**Telemetry and Correlation** ([`services/telemetry.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/telemetry.py), [`services/correlation_engine.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/correlation_engine.py)) deduplicate cross-feed data and compute derived intelligence layers such as GPS-jamming heatmaps and vessel anomaly detection.

## Privacy and Security Infrastructure

The [`services/privacy_core_client.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/privacy_core_client.py) module provides the Python interface to a compiled Rust crate (`privacy-core`) that implements cryptographic primitives including Ed25519 signatures, X25519 key exchange, and RingCT privacy mechanisms. At startup, [`services/privacy_core_attestation.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/privacy_core_attestation.py) verifies the Rust binary hash against expected values to prevent tampering.

Governance and release management flow through [`services/privacy_claims.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/privacy_claims.py) and [`services/release_profiles.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/release_profiles.py), which expose the "release gate" status to frontend operators, indicating feature flags, compatibility debt, and Sovereign Shell readiness.

## Component Integration Architecture

During startup, [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py) initializes the service plane through a strict sequence: environment validation, privacy-core attestation, cache seeding, and router registration. The data fetcher spawns background APScheduler jobs that continuously populate [`services/telemetry.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/telemetry.py) caches, while mesh components establish peer connections via the transport manager.

Public-lane data (ADS-B, AIS, open-source intelligence) flows through standard REST endpoints, while private-lane mesh traffic utilizes onion routing or RNS transports with mandatory cryptographic signing. The OpenClaw channel bridges these worlds, allowing AI agents to query both public telemetry and private mesh events through a unified HMAC-authenticated interface.

This architecture enables horizontal scaling through containerized deployment, where individual microservices can be replicated or isolated based on operational requirements without compromising the cryptographic boundaries between public data and protected mesh communications.

## Summary

- **FastAPI Foundation**: The [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py) entry point initializes a modular router layer spanning health checks, CCTV, SIGINT, mesh networking, and AI channels.
- **Data Ingestion**: [`services/data_fetcher.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/data_fetcher.py) orchestrates APScheduler jobs across specialized fetchers for AIS, Shodan, satellite TLEs, and news feeds, caching results in [`telemetry.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/telemetry.py).
- **Mesh Networking**: The `services/mesh/` stack provides decentralized, cryptographically signed messaging with tier-gated privacy controls and transport management.
- **AI Integration**: OpenClaw services expose HMAC-signed endpoints allowing LLM agents to manipulate map data and query intelligence layers.
- **Security Model**: Rust-based privacy-core integration via [`privacy_core_client.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/privacy_core_client.py) provides Ed25519/X25519 primitives with runtime attestation and release-gate governance.

## Frequently Asked Questions

### What is the main entry point for the Shadowbroker backend?

The [`backend/main.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/backend/main.py) file serves as the primary entry point, creating the FastAPI application instance, loading environment variables, validating secrets, and registering the modular router layer through the `_load_optional_router` function. This file also initializes CORS middleware and prepares the service plane for horizontal deployment.

### How does Shadowbroker handle real-time data ingestion?

Real-time ingestion is managed by [`services/data_fetcher.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/data_fetcher.py), which uses APScheduler to coordinate fast-tier (60-second) and slow-tier background jobs. These jobs execute feed-specific adapters—such as [`services/ais_stream.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/ais_stream.py) for vessel tracking and [`services/shodan_connector.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/shodan_connector.py) for device search—storing the latest snapshots in an in-memory cache managed by [`services/telemetry.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/telemetry.py).

### What is the OpenClaw channel used for?

The OpenClaw channel provides a secure, signed API surface in [`services/openclaw_channel.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/openclaw_channel.py) that allows external LLM agents to interact with the backend's geospatial data. Using HMAC-SHA256 authentication with timestamped nonces, agents can place map pins, control viewports, and query intelligence layers without direct database access, enabling safe agentic AI integration.

### How does the backend ensure privacy and encryption?

Privacy guarantees are enforced through [`services/privacy_core_client.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/privacy_core_client.py), a Python wrapper around a compiled Rust crate providing Ed25519 signatures, X25519 key exchange, and RingCT privacy mechanisms. The [`services/privacy_core_attestation.py`](https://github.com/BigBodyCobain/Shadowbroker/blob/main/services/privacy_core_attestation.py) module verifies the Rust binary hash at startup, while mesh components utilize these primitives to encrypt peer-to-peer traffic across onion and RNS transports.