Shadowbroker Backend Microservices and Components: Complete Architecture Guide

Shadowbroker's backend is a modular FastAPI service plane that stitches together specialized Python microservices to fetch, enrich, store, and expose real-time geospatial intelligence through a unified API.

The BigBodyCobain/Shadowbroker repository implements a horizontally scalable architecture designed for intelligence operations. This backend separates concerns into distinct components—ranging from data ingestion schedulers to decentralized mesh routers—enabling independent deployment while maintaining tight integration through shared telemetry caches and cryptographic privacy controls. Understanding these microservices is essential for operators extending the platform's capabilities or deploying nodes across distributed environments.

FastAPI Entry Point and Router Layer

The backend/main.py file serves as the application bootstrap, creating the FastAPI instance, configuring CORS middleware, and registering the modular router layer. During startup, the system validates environment secrets and lazily imports optional routers via the _load_optional_router pattern to keep the service plane lightweight.

The router layer in backend/routers/*.py exposes distinct API namespaces that map to functional domains:

• /health – System health checks and uptime metrics
• /cctv – Live camera feed aggregation and GeoJSON tile generation
• /radio – SDR tuning and radio intercept endpoints
• /sigint – Signal intelligence queries via Shodan and scanner aggregation
• /mesh_* – Decentralized peer synchronization and messaging
• /ai_intel – OpenClaw agentic AI command channel
• /sar – Synthetic Aperture Radar catalog services
• /wormhole – Wormhole relay control for private transport

Each router operates as an isolated component, allowing developers to disable entire functional domains by omitting the router file without affecting core system stability.

To verify router registration and system health, query the health endpoint:

import httpx

resp = httpx.get("http://localhost:8000/health")
print(resp.json())

# → {"status":"ok","version":"0.9.75","uptime":123.4}

Data Ingestion and Scheduler Services

The services/data_fetcher.py module orchestrates the backend's real-time intelligence pipeline using APScheduler to manage fast-tier (60-second) and slow-tier ingestion jobs. This component continuously pulls external telemetry—including ADS-B flight data, AIS vessel positions, satellite TLEs, and GDELT news feeds—and caches the latest snapshots in memory via services/telemetry.py.

Feed-specific adapters in the services/ directory provide thin wrappers for external APIs:

• ais_stream.py – WebSocket client for aisstream.io vessel tracking
• shodan_connector.py – Device search and internet exposure queries
• tinygs_fetcher.py – LoRa satellite position retrieval from TinyGS
• satnogs_fetcher.py – Amateur ground station data from SatNOGS
• kiwisdr_fetcher.py – Public SDR receiver catalogs
• carrier_tracker.py – Carrier strike group estimation via GDELT news analysis

To manually seed caches for testing or development, invoke the data fetcher directly:

from backend.services.data_fetcher import start_scheduler, seed_startup_caches

# Initialise caches with latest snapshots

seed_startup_caches()

# Kick off APScheduler jobs (runs every 60s for fast feeds)

start_scheduler()

Accessing cached AIS data programmatically:

from backend.services.ais_stream import start_ais_stream
from backend.services.data_fetcher import get_latest_data

# Start websocket stream in background thread

start_ais_stream()

# Read vessel by MMSI from global cache

ais_snapshot = get_latest_data()["ais"]
print(ais_snapshot.get("123456789"))   # → {'lat': 51.5, 'lon': -0.12, ...}

Mesh and InfoNet Networking Stack

The services/mesh/ directory contains the decentralized messaging layer that powers the InfoNet test-net and Wormhole relay systems. This stack enables cryptographically signed, tier-gated peer-to-peer communication across public, private, and heavy transport lanes.

Core mesh components include:

• mesh_router.py – Routing logic for peer-to-peer message propagation
• mesh_wormhole_contacts.py – Pre-key lookup and contact discovery
• mesh_private_transport_manager.py – Onion and RNS (Reticulum) transport management
• mesh_private_release_worker.py – Private-lane release handling
• mesh_privacy_policy.py – Transport-tier enforcement and privacy gating

Peers discover each other via bootstrap manifests and negotiate encrypted transports using X25519 and Ed25519 primitives provided by the privacy-core integration.

To inspect private-lane peers:

from backend.services.mesh.mesh_peer_store import PeerStore, DEFAULT_PEER_STORE_PATH

store = PeerStore(DEFAULT_PEER_STORE_PATH)
store.load()
print([p for p in store.peers if p.transport == "onion"])

AI Integration: The OpenClaw Channel

The OpenClaw subsystem provides a signed, tier-gated API surface for external LLM agents to interact with the map data layer. Implemented across services/openclaw_channel.py, openclaw_bridge.py, and openclaw_watchdog.py, this channel accepts HMAC-signed HTTP commands that allow AI agents to place pins, control viewports, and read telemetry.

The signing protocol requires timestamped, nonced payloads verified against a pre-shared secret, preventing replay attacks and ensuring command authenticity.

Sending a signed OpenClaw command:

import hmac, hashlib, json, time, uuid, httpx

SECRET = "••••‑your‑hmac‑secret‑••••"
path = "/api/ai/channel/command"
body = json.dumps({"cmd": "pin_place", "args": {"lat": 37.7749, "lon": -122.4194, "category": "threat"}})
ts = str(int(time.time()))
nonce = uuid.uuid4().hex
payload = f"POST|{path}|{ts}|{nonce}|{hashlib.sha256(body.encode()).hexdigest()}"
sig = hmac.new(SECRET.encode(), payload.encode(), hashlib.sha256).hexdigest()

headers = {
    "X-SB-Timestamp": ts,
    "X-SB-Nonce": nonce,
    "X-SB-Signature": sig,
    "Content-Type": "application/json",
}
resp = httpx.post(f"http://localhost:8000{path}", data=body, headers=headers)
print(resp.json())

Specialized Intelligence Modules

Beyond core data ingestion, the backend includes specialized services for specific intelligence disciplines:

CCTV Pipeline (services/cctv_pipeline.py) aggregates live camera feeds across multiple protocols (MJPEG, HLS, embedded streams) and normalizes them into GeoJSON tiles for frontend rendering.

Radio and SIGINT (services/radio_intercept.py, services/sigint_bridge.py) interface with KiwiSDR instances, police/fire scanners, Meshtastic MQTT brokers, and APRS-IS networks to capture and correlate signal intelligence.

SAR Services (services/sar/) handle Synthetic Aperture Radar catalog queries and anomaly detection feeds, providing all-weather surveillance capabilities independent of optical sensors.

Telemetry and Correlation (services/telemetry.py, services/correlation_engine.py) deduplicate cross-feed data and compute derived intelligence layers such as GPS-jamming heatmaps and vessel anomaly detection.

Privacy and Security Infrastructure

The services/privacy_core_client.py module provides the Python interface to a compiled Rust crate (privacy-core) that implements cryptographic primitives including Ed25519 signatures, X25519 key exchange, and RingCT privacy mechanisms. At startup, services/privacy_core_attestation.py verifies the Rust binary hash against expected values to prevent tampering.

Governance and release management flow through services/privacy_claims.py and services/release_profiles.py, which expose the "release gate" status to frontend operators, indicating feature flags, compatibility debt, and Sovereign Shell readiness.

Component Integration Architecture

During startup, backend/main.py initializes the service plane through a strict sequence: environment validation, privacy-core attestation, cache seeding, and router registration. The data fetcher spawns background APScheduler jobs that continuously populate services/telemetry.py caches, while mesh components establish peer connections via the transport manager.

Public-lane data (ADS-B, AIS, open-source intelligence) flows through standard REST endpoints, while private-lane mesh traffic utilizes onion routing or RNS transports with mandatory cryptographic signing. The OpenClaw channel bridges these worlds, allowing AI agents to query both public telemetry and private mesh events through a unified HMAC-authenticated interface.

This architecture enables horizontal scaling through containerized deployment, where individual microservices can be replicated or isolated based on operational requirements without compromising the cryptographic boundaries between public data and protected mesh communications.

Summary

• FastAPI Foundation: The backend/main.py entry point initializes a modular router layer spanning health checks, CCTV, SIGINT, mesh networking, and AI channels.
• Data Ingestion: services/data_fetcher.py orchestrates APScheduler jobs across specialized fetchers for AIS, Shodan, satellite TLEs, and news feeds, caching results in telemetry.py.
• Mesh Networking: The services/mesh/ stack provides decentralized, cryptographically signed messaging with tier-gated privacy controls and transport management.
• AI Integration: OpenClaw services expose HMAC-signed endpoints allowing LLM agents to manipulate map data and query intelligence layers.
• Security Model: Rust-based privacy-core integration via privacy_core_client.py provides Ed25519/X25519 primitives with runtime attestation and release-gate governance.

Frequently Asked Questions

What is the main entry point for the Shadowbroker backend?

The backend/main.py file serves as the primary entry point, creating the FastAPI application instance, loading environment variables, validating secrets, and registering the modular router layer through the _load_optional_router function. This file also initializes CORS middleware and prepares the service plane for horizontal deployment.

How does Shadowbroker handle real-time data ingestion?

Real-time ingestion is managed by services/data_fetcher.py, which uses APScheduler to coordinate fast-tier (60-second) and slow-tier background jobs. These jobs execute feed-specific adapters—such as services/ais_stream.py for vessel tracking and services/shodan_connector.py for device search—storing the latest snapshots in an in-memory cache managed by services/telemetry.py.

What is the OpenClaw channel used for?

The OpenClaw channel provides a secure, signed API surface in services/openclaw_channel.py that allows external LLM agents to interact with the backend's geospatial data. Using HMAC-SHA256 authentication with timestamped nonces, agents can place map pins, control viewports, and query intelligence layers without direct database access, enabling safe agentic AI integration.

How does the backend ensure privacy and encryption?

Privacy guarantees are enforced through services/privacy_core_client.py, a Python wrapper around a compiled Rust crate providing Ed25519 signatures, X25519 key exchange, and RingCT privacy mechanisms. The services/privacy_core_attestation.py module verifies the Rust binary hash at startup, while mesh components utilize these primitives to encrypt peer-to-peer traffic across onion and RNS transports.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}