# How Hydrus Client-Server Networking Handles HTTP/HTTPS Requests

> Discover how Hydrus client-server networking manages HTTP/HTTPS requests using Python requests and Twisted. Understand its dual-layer architecture for efficient API handling.

- Repository: [Hydrus Network Developer/hydrus](https://github.com/hydrusnetwork/hydrus)
- Tags: internals
- Published: 2026-03-03

---

**Hydrus implements a dual-layer architecture where the client uses Python's `requests` library with custom retry logic in [`ClientNetworkingJobs.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingJobs.py) for outbound HTTP/HTTPS calls, while the server utilizes Twisted's asynchronous framework in [`HydrusServer.py`](https://github.com/hydrusnetwork/hydrus/blob/main/HydrusServer.py) to handle incoming API requests with TLS support.**

The hydrusnetwork/hydrus repository organizes its networking stack into distinct client and server components to manage media downloads and API services. Understanding how Hydrus handles HTTP/HTTPS requests reveals a sophisticated system that balances robust error recovery on the client side with efficient asynchronous request processing on the server side.

## Client-Side HTTP/HTTPS Request Flow

The client networking layer manages outbound connections to remote image hosts and the local Hydrus API through the `NetworkJob` class defined in [`hydrus/client/networking/ClientNetworkingJobs.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/networking/ClientNetworkingJobs.py).

### Creating Network Jobs

Callers initialize requests by instantiating `NetworkJob` with the HTTP method, URL, and optional body parameters. This object encapsulates the entire request lifecycle including headers, cookies, and bandwidth tracking.

```python
from hydrus.client.networking import ClientNetworkingJobs

job = ClientNetworkingJobs.NetworkJob('GET', 'https://example.com/api/data')

```

### Domain and Session Management

Before transmission, the job constructs **network contexts** (global, domain, and second-level domain) via [`hydrus/client/networking/ClientNetworkingFunctions.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/networking/ClientNetworkingFunctions.py). These contexts drive session selection, cookie policies, and bandwidth throttling managed by [`ClientNetworkingSessions.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingSessions.py) and the domain manager.

### Sending Requests with Stream Support

The actual HTTP/HTTPS transmission occurs in `_SendRequestAndGetResponse`, which retrieves a `requests.Session` from the session manager and invokes `session.request()` with `stream=True` for memory-efficient large file handling:

```python
response = session.request(method, url, data=data, headers=headers,
                         stream=True, timeout=(connect, read), verify=session.verify)

```

### Range Requests and Resumable Downloads

For large file downloads, the client automatically manages HTTP Range headers. The `_ReadResponse` method parses `Content-Range` responses and updates the `Range: bytes=<already-read>-` header for subsequent requests until the full content is retrieved, writing chunks to a temporary `SpooledTemporaryFile`.

### Error Handling and Retry Logic

The client maps HTTP status codes to Hydrus-specific exceptions via `ConvertStatusCodeAndDataIntoExceptionInfo`. When encountering connection errors, server-side throttling (429/509/529), or non-2xx responses, the job implements exponential backoff within configured retry limits. Developers can override wait periods using `OverrideConnectionErrorWait` and `OverrideServersideBandwidthWait`.

## Server-Side HTTP/HTTPS Request Handling

The Hydrus server exposes services through an asynchronous Twisted web stack defined in [`hydrus/core/networking/HydrusServer.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/core/networking/HydrusServer.py).

### Twisted Web Server Architecture

The `HydrusService` class extends Twisted's `Site` to build a resource tree in `_InitRoot`, serving endpoints for the API, welcome pages, and static resources like `favicon.ico` and [`robots.txt`](https://github.com/hydrusnetwork/hydrus/blob/main/robots.txt).

### Request Wrapping and Header Injection

Incoming requests are wrapped in `HydrusRequest` or `HydrusRequestLogging` (when logging is enabled) from [`hydrus/core/networking/HydrusServerRequest.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/core/networking/HydrusServerRequest.py). These subclasses add `parsed_request_args` for variable handling and timing information. The `getResourceFor` method injects `Server` and `Hydrus-Server` headers containing version metadata before routing.

### Resource Endpoints and Rendering

Each API endpoint inherits from `HydrusServerResources.Resource` in [`hydrus/server/networking/ServerServerResources.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/server/networking/ServerServerResources.py), implementing `render_GET` or `render_POST` to parse arguments and return `ResponseContext` objects containing status codes, bodies, and cookies:

```python
class HydrusResourceBusyCheck(HydrusServerResources.Resource):
    def render_GET(self, request):
        return HydrusServerResources.ResponseContext(200, body=b'0')

```

### Authentication and Access Control

The `HydrusServiceRestricted` base class enforces session key validation, account permissions, and bandwidth limits before executing resource methods.

### HTTPS and TLS Configuration

TLS termination utilizes Twisted's `SSLContextFactory` via `HydrusServerContextFactory`. Clients connect using `https://` URLs, with certificate verification controlled by the `verify` flag in the underlying `requests` session.

## Practical Code Examples

### Executing a Simple GET Request

```python
from hydrus.client.networking import ClientNetworkingJobs

job = ClientNetworkingJobs.NetworkJob('GET',
                                      'https://example.com/api/info')
job.Start()                     # runs in a background thread

job._is_done_event.wait()       # block until finished

if job.HasError():
    raise job.GetErrorException()
data = job.GetContentBytes()    # raw response body

text = job.GetContentText()     # auto-decoded Unicode string

print('Received', len(data), 'bytes')

```

### Adding Custom Headers and Referrals

```python
job = ClientNetworkingJobs.NetworkJob('GET',
                                      'https://api.example.com/data',
                                      referral_url='https://myapp.local')
job.AddAdditionalHeader('X-My-Header', 'value')
job.Start()

```

### Resumable Large File Downloads

```python
job = ClientNetworkingJobs.NetworkJob('GET',
                                      'https://bigfile.example.com/file.bin',
                                      temp_path='/tmp/file.bin')
job.Start()

# Automatically uses Range headers until complete

```

### Starting the HTTPS Server

```bash
python3 hydrus_server_boot.py start --port 45871

```

## Summary

- **Hydrus** separates networking into a **client layer** ([`ClientNetworkingJobs.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingJobs.py)) using Python's `requests` library and a **server layer** ([`HydrusServer.py`](https://github.com/hydrusnetwork/hydrus/blob/main/HydrusServer.py)) using Twisted.
- The client handles **resumable downloads** via HTTP Range headers, **exponential backoff retries**, and **bandwidth throttling** through network contexts.
- The server wraps requests in `HydrusRequest` objects, injects version headers, and validates **session keys** before serving resources.
- **HTTPS support** is implemented via Twisted's `SSLContextFactory` with configurable certificate verification.

## Frequently Asked Questions

### What Python libraries does Hydrus use for HTTP/HTTPS networking?

The Hydrus client utilizes the standard `requests` library for synchronous HTTP/HTTPS calls with streaming support, while the server employs **Twisted**'s asynchronous web framework (`twisted.web.server`) to handle concurrent API requests.

### How does Hydrus handle partial content downloads?

The client automatically manages **HTTP Range** headers in [`ClientNetworkingJobs.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingJobs.py). When downloading large files, it iterates over `response.iter_content` and updates the `Range: bytes=<already-read>-` header for subsequent requests until the full content is retrieved, storing chunks in a `SpooledTemporaryFile`.

### Where is the retry logic implemented for failed client requests?

Retry logic resides in [`hydrus/client/networking/ClientNetworkingJobs.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/networking/ClientNetworkingJobs.py) within the `NetworkJob` class. It detects connection errors and server-side throttling (status codes 429, 509, 529), implements exponential backoff delays, and provides override methods like `OverrideConnectionErrorWait` for manual control.

### Does the Hydrus server require HTTPS or support HTTP?

The Hydrus server supports both protocols. HTTPS is configured through Twisted's `SSLContextFactory` (`HydrusServerContextFactory`), while HTTP operates over standard TCP. Client verification of server certificates is controlled by the `verify` parameter in the `requests.Session` configuration.