Hydrus Service System Architecture: Local Repository Server Design

The Hydrus service system architecture implements a Twisted-based HTTP/HTTPS server that manages distinct services (administration, file repository, and tag repository) through a centralized ServerController, SQLite persistence layer, and specialized networking resources.

The hydrusnetwork/hydrus repository uses this modular architecture of the service system to power its local repository server, allowing users to self-host file and metadata repositories with granular permission controls. This design separates bootstrapping, database operations, and HTTP request handling into discrete components that communicate through well-defined interfaces. The architecture supports multiple concurrent service types running on separate ports while maintaining a unified persistence layer under hydrus/server/.

Core Components of the Service System

Bootstrapping and Reactor Initialization

The entry point hydrus/hydrus_server_boot.py parses command-line options, decides whether to start, stop, or restart the daemon, and launches the Twisted reactor in a dedicated thread. This bootstrapping component initializes the global state and prepares the environment before handing control to the main orchestrator.

The ServerController Orchestrator

At the heart of the system sits hydrus/server/ServerController.py. The Controller class initializes the ServerDB instance, manages the session manager, and maintains the list of active services. During startup, it calls InitModel followed by InitView to prepare the data layer and networking stack. The controller exposes SetRunningTwistedServices(services), which iterates over each service definition—generated via HydrusNetwork.GenerateService—and binds them to SSL listeners.

Persistence and Storage Layers

Data durability relies on two primary abstractions:

• hydrus/server/ServerDB.py – Implements the SQLite-backed persistence layer for accounts, sessions, files, tags, and petitions. It provides the Read and WriteSynchronous methods that resources invoke to query or mutate state.
• hydrus/server/ServerFiles.py – Maps cryptographic file hashes to deterministic on-disk paths under server_files/, handling physical storage independently of the database schema.
• hydrus/server/ServerGlobals.py – Exposes a global reference (server_controller) used by networking resources to reach the controller without circular imports.

Networking and Resource Handling

The HTTP layer resides in hydrus/server/networking/:

• ServerServer.py – Constructs Twisted Resource trees for each service type and registers URL endpoints. It calls listenSSL for each service port (default admin port 45871).
• ServerServerResources.py – Contains concrete request handlers such as HydrusResourceAccessKey and HydrusResourceRestrictedRepositoryFile. These classes parse GET/POST arguments via HydrusNetworkVariableHandling, check HG.server_busy status, validate account permissions, and delegate to SG.server_controller.Read or WriteSynchronous.

Service Types and Registration Flow

The architecture supports three primary service types defined in the HC constants module:

• SERVER_ADMIN (HC.SERVER_ADMIN) – Runs on port 45871 by default; provides administration UI and maintenance actions (backup, lock, shutdown).
• FILE_REPOSITORY (HC.FILE_REPOSITORY) – Stores raw files and thumbnails; exposes endpoints like /file and /thumbnail.
• TAG_REPOSITORY (HC.TAG_REPOSITORY) – Manages tag-to-hash mappings; supports /update, /metadata, and /tag_filter.

During initialization, Controller.SetRunningTwistedServices creates a ServerServer.HydrusService* subclass for each configured service, builds a URL root (e.g., https://127.0.0.1:45871/), and registers it with the Twisted reactor.

Request Handling Lifecycle

A complete HTTP request follows this path through the architecture:

1. Startup – hydrus_server_boot.py spawns the reactor thread and instantiates the Controller.
2. Controller Init – The controller creates ServerDB, reads the admin service configuration, and invokes InitModel → InitView.
3. Service Registration – SetRunningTwistedServices iterates over service objects, binds each to a port via listenSSL, and attaches the appropriate resource tree.
4. Request Dispatch – Twisted receives an HTTPS request and routes it to the corresponding HydrusResource* class in ServerServerResources.py. The resource checks permissions and calls Read or WriteSynchronous on the controller.
5. Database Interaction – ServerDB executes SQL queries against tables such as files_info, hashes, tags, and accounts, while ServerFiles mirrors uploaded content to disk.
6. Shutdown – On SIGINT or a /shutdown request, the controller stops Twisted listeners, flushes pending writes, and releases global locks.

Starting the Server and Uploading Files

The following example demonstrates the bootstrap-to-upload pipeline:

# Start the local server (equivalent to running `hydrus_server.py start`)

import subprocess, os, time

# Assume the repository lives in ~/hydrus_repo

repo_dir = os.path.expanduser('~/hydrus_repo')
os.makedirs(repo_dir, exist_ok=True)

# Launch the server in a subprocess (the boot script will spawn Twisted)

proc = subprocess.Popen(
    ['python3', '-m', 'hydrus.hydrus_server_boot', 'start',
     '--db_dir', repo_dir],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
)

# Wait a few seconds for the server to bind its admin port

time.sleep(5)
print('Server started, admin port should be listening on 45871')

Once running, clients interact with the service system via the HTTP API:

# Upload a file via the HTTP API (requires registration key → access key)

# 1️⃣ Get a registration key (admin account can request one)

curl -s -X GET "https://127.0.0.1:45871/auto_create_registration_key?account_type_key=admin" \
    --insecure | python -c "import sys, json; print(json.load(sys.stdin)['registration_key'])"

# → prints a 64‑byte registration key (hex)

# 2️⃣ Exchange the registration key for an access key

REG_KEY=<registration_key_from_above>
curl -s -X GET "https://127.0.0.1:45871/access_key?registration_key=$REG_KEY" \
    --insecure | python -c "import sys, json; print(json.load(sys.stdin)['access_key'])"

# → prints an access key

# 3️⃣ Obtain a session cookie (required for uploading)

ACCESS_KEY=<access_key_from_above>
curl -i -X GET "https://127.0.0.1:45871/session_key?access_key=$ACCESS_KEY" \
    --insecure | grep Set-Cookie

# → Set-Cookie: session_key=...

# 4️⃣ Upload a file to the file repository service (assume its port is 45872)

FILE_REPO_PORT=45872
curl -X POST "https://127.0.0.1:$FILE_REPO_PORT/file" \
    -H "Cookie: session_key=$(echo $COOKIE | cut -d'=' -f2)" \
    -F "hash=$(sha256sum myimage.jpg | cut -d' ' -f1)" \
    -F "size=$(stat -c%s myimage.jpg)" \
    -F "mime=image/jpeg" \
    -F "file=@myimage.jpg" \
    --insecure

Summary

• The Hydrus service system architecture relies on a Twisted-based HTTP/HTTPS server that hosts multiple concurrent services on separate ports.
• ServerController.py acts as the central orchestrator, managing the SQLite database (ServerDB.py), session state, and service lifecycle.
• Three service types—SERVER_ADMIN, FILE_REPOSITORY, and TAG_REPOSITORY—expose distinct REST endpoints registered via ServerServer.py.
• Physical storage is abstracted by ServerFiles.py, which maps hashes to paths under server_files/, while ServerGlobals.py provides global access to the running controller.
• Requests flow from Twisted resources in ServerServerResources.py through permission checks to synchronous database reads/writes.

Frequently Asked Questions

What is the role of the ServerController in Hydrus?

The ServerController class in hydrus/server/ServerController.py serves as the central orchestrator for the entire service system. It initializes the SQLite database via ServerDB, manages the session manager, and drives the main event loop. The controller also handles graceful shutdown by stopping Twisted listeners and flushing pending database writes.

How does Hydrus handle file storage in the service system?

File storage uses a dual-layer approach: hydrus/server/ServerDB.py tracks file metadata, hashes, and ownership in SQLite, while hydrus/server/ServerFiles.py maps each file hash to a deterministic on-disk path under server_files/. This separation allows the database to manage logical relationships while the storage layer handles physical byte persistence.

What types of services can run in a Hydrus repository?

The architecture supports three core service types defined in the HC constants module: SERVER_ADMIN for maintenance and user management (default port 45871), FILE_REPOSITORY for raw file and thumbnail storage, and TAG_REPOSITORY for tag-to-hash mappings. Each service runs as an isolated Twisted SSL listener with its own resource tree.

How does the boot process initialize the Twisted reactor?

The hydrus/hydrus_server_boot.py script parses command-line arguments to determine the run mode (start, stop, or restart), then creates a dedicated thread for the Twisted reactor. This bootstrapping component instantiates the ServerController, which subsequently calls SetRunningTwistedServices to register each service's SSL endpoint with the reactor before entering the main event loop.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}