# How Hydrus Throttles Network Requests: A Deep Dive into Its Bandwidth Management System

> Discover how Hydrus throttles network requests using its two-layer bandwidth control system combining usage tracking and policy enforcement for efficient bandwidth management across five time granularities.

- Repository: [Hydrus Network Developer/hydrus](https://github.com/hydrusnetwork/hydrus)
- Tags: deep-dive
- Published: 2026-03-03

---

**Hydrus implements a two-layer bandwidth control system that combines granular usage tracking via `BandwidthTracker` with policy enforcement via `BandwidthRules` to decide whether network jobs may start, continue, or must wait across five time granularities.**

The hydrusnetwork/hydrus client manages thousands of concurrent download jobs across multiple domains, subscriptions, and gallery crawlers. To prevent overwhelming servers or hitting ISP caps, the application employs a sophisticated **bandwidth management system** that throttles network requests through per-context accounting and configurable rate limits.

## BandwidthTracker: Granular Usage Accounting

At the foundation of the throttling mechanism lies the `BandwidthTracker` class, implemented in [[`hydrus/core/networking/HydrusNetworking.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/core/networking/HydrusNetworking.py)](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/core/networking/HydrusNetworking.py#L305-L400). Each network context—whether global, domain-specific, or tied to a specific subscription—maintains its own tracker instance.

### Five Time-Granularity Counters

The tracker maintains **five distinct time-granularity counters** for both data volume and request frequency:

- `_seconds_*` for real-time second-level tracking
- `_minutes_*` for minute-level aggregation  
- `_hours_*` for hourly windows
- `_days_*` for daily caps
- `_months_*` for monthly quotas

When a job completes a transfer, it reports usage through `ReportDataUsed(num_bytes)` or `ReportRequestUsed()`. The tracker automatically buckets this usage into the appropriate counter based on the current UTC timestamp. Querying usage via `GetUsage(bandwidth_type, time_delta)` allows the system to check consumption against any look-back window, where `bandwidth_type` is either `HC.BANDWIDTH_TYPE_DATA` (bytes) or `HC.BANDWIDTH_TYPE_REQUESTS`.

```python

# Record 1 KB of data transferred

bandwidth_tracker.ReportDataUsed(1024)

# Record a single HTTP request

bandwidth_tracker.ReportRequestUsed()

```

## BandwidthRules: Policy Enforcement

While `BandwidthTracker` records history, the `BandwidthRules` class—found in [[`hydrus/core/networking/HydrusNetworking.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/core/networking/HydrusNetworking.py)](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/core/networking/HydrusNetworking.py#L14-L85)—defines the actual throttling policies. Rules are stored as tuples of **(bandwidth_type, time_delta, max_allowed)** in a thread-safe set.

### Rule Evaluation Methods

The class provides three core methods for throttling decisions:

- **`CanStartRequest(tracker, threshold=5)`**: Returns **False** if any data rule would be exceeded, preventing new connections when bandwidth is scarce. Request-type rules are ignored for start checks to avoid deadlocks.
- **`CanDoWork(tracker, expected_requests, expected_bytes, threshold=30)`**: Used by batch operations like subscriptions; discounts expected usage from limits before checking if the entire batch should proceed.
- **`CanContinueDownload(tracker, threshold=15)`**: Called during active transfers to determine if a large file download should pause mid-stream when approaching hourly or monthly caps.

```python

# Allow 10 MiB per hour

rules.AddRule(HC.BANDWIDTH_TYPE_DATA, 3600, 10 * 1024 * 1024)

# Allow 100 requests per minute  

rules.AddRule(HC.BANDWIDTH_TYPE_REQUESTS, 60, 100)

```

## NetworkBandwidthManager: Per-Context Coordination

The `NetworkBandwidthManager` in [[`hydrus/client/networking/ClientNetworkingBandwidth.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/networking/ClientNetworkingBandwidth.py)](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/networking/ClientNetworkingBandwidth.py#L58-L150) orchestrates trackers and rules across different network contexts. It maintains a **default tracker** (`_my_bandwidth_tracker`) for the client itself and a dictionary of **tracker containers** keyed by `NetworkContext` objects (defined in [`ClientNetworkingContexts.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingContexts.py)).

When the networking engine evaluates a job, it calls `_CanStartRequest(network_contexts)`, which iterates through all relevant contexts—global, domain, subscription—and verifies that each satisfies its respective rules:

```python
def _CanStartRequest(self, network_contexts):
    for network_context in network_contexts:
        rules = self._GetRules(network_context)
        tracker = self._GetTracker(network_context)
        if not rules.CanStartRequest(tracker):
            return False
    return True

```

The manager creates fresh `BandwidthTracker` instances lazily via `_GetTracker`, wrapping them in `NetworkBandwidthManagerTrackerContainer` objects. Upon request completion, `_ReportDataUsed` and `_ReportRequestUsed` propagate usage statistics to **all relevant trackers** simultaneously, including the global bandwidth tracker.

## Throttling in Action: From Job Admission to Mid-Download Checks

The bandwidth management system integrates into the request lifecycle at three critical points, as implemented in the main networking engine at [[`hydrus/client/networking/ClientNetworking.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/networking/ClientNetworking.py)](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/networking/ClientNetworking.py#L293-L300).

### Job Admission Control

Before moving a job from *awaiting* to *running*, the engine invokes `_CanStartRequest`. If any context reports insufficient quota, the job remains queued until the time window rolls over or competing jobs release bandwidth.

### Mid-Download Interruption

For large file downloads, `CanContinueDownload` checks whether completing the transfer would breach data caps. This prevents overshooting monthly limits when downloading multi-gigabyte archives.

### Batch Work Regulation

Import subscriptions and gallery crawlers call `CanDoWork` with estimates of their planned byte and request consumption. The manager may delay the entire batch until sufficient quota accumulates, preventing partial job executions that would waste server resources.

## Practical Implementation Examples

### Adding a Custom Domain Rule

To limit a specific domain to 5 MiB per hour:

```python
from hydrus.core import HydrusConstants as HC
from hydrus.client.networking import ClientNetworkingBandwidth, ClientNetworkingContexts

bw_manager = ClientNetworkingBandwidth.NetworkBandwidthManager()

domain_ctx = ClientNetworkingContexts.NetworkContext(
    context_type=CC.NETWORK_CONTEXT_DOMAIN,
    identifier='example.com'
)

rules = bw_manager._GetRules(domain_ctx)
rules.AddRule(HC.BANDWIDTH_TYPE_DATA, 3600, 5 * 1024 * 1024)

```

### Simulating Throttle Conditions

When the hourly quota is exhausted, subsequent requests block:

```python
tracker = bw_manager._GetTracker(domain_ctx)
tracker.ReportDataUsed(5 * 1024 * 1024)  # Consume full quota

if not bw_manager._CanStartRequest([domain_ctx]):
    print('Request throttled – bandwidth limit reached')

```

### Querying Current Usage

Display bandwidth statistics in the UI by querying specific time windows:

```python
from hydrus.core import HydrusNumbers, HydrusData

global_tracker = bw_manager._my_bandwidth_tracker

# Bytes used this month

month_bytes = global_tracker.GetUsage(HC.BANDWIDTH_TYPE_DATA, None)
print('Month usage:', HydrusData.ConvertValueRangeToBytes(month_bytes, None))

# Requests in last minute

recent_reqs = global_tracker.GetUsage(HC.BANDWIDTH_TYPE_REQUESTS, 60)
print('Requests last minute:', HydrusNumbers.ValueRangeToPrettyString(recent_reqs, None))

```

## Summary

- **Two-layer architecture**: `BandwidthTracker` handles usage accounting while `BandwidthRules` enforces policies across five time granularities (seconds to months).
- **Per-context isolation**: Each domain, subscription, and the global client maintain separate trackers and rule sets via `NetworkBandwidthManager`.
- **Three-phase throttling**: The system controls job admission (`CanStartRequest`), batch approval (`CanDoWork`), and mid-download continuation (`CanContinueDownload`) with configurable thresholds of 5, 30, and 15 respectively.
- **Thread-safe reporting**: Usage propagates to all relevant context trackers simultaneously when requests complete.
- **Implementation locations**: Core logic resides in [`HydrusNetworking.py`](https://github.com/hydrusnetwork/hydrus/blob/main/HydrusNetworking.py) (tracker/rules) and [`ClientNetworkingBandwidth.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingBandwidth.py) (manager coordination).

## Frequently Asked Questions

### What are the five time granularities tracked by the bandwidth management system?

The `BandwidthTracker` class maintains counters for **seconds, minutes, hours, days, and months**. This allows the system to enforce both burst-rate limiting (per-second) and long-term quotas (per-month) simultaneously using the same accounting infrastructure.

### How does Hydrus decide whether to start a new network request?

Before starting a job, the networking engine calls `_CanStartRequest`, which checks every relevant `NetworkContext` (global, domain, subscription). For each context, it retrieves the `BandwidthRules` and queries `CanStartRequest(tracker, threshold=5)`. If any rule would be exceeded, the job remains queued. Notably, this check only evaluates **data** rules, not request-count rules, to prevent starvation scenarios.

### What is the difference between `CanStartRequest` and `CanContinueDownload`?

**`CanStartRequest`** (threshold=5) evaluates whether a new connection should be allowed to initiate, while **`CanContinueDownload`** (threshold=15) determines if an already-active large file transfer should proceed or pause. The higher threshold for continuation provides slack for in-progress transfers while still protecting against quota overruns.

### Where are bandwidth rules stored and how are they accessed?

Rules are stored as tuples in thread-safe `set` objects within `BandwidthRules` instances. The `NetworkBandwidthManager` organizes these by `NetworkContext` in [`ClientNetworkingBandwidth.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientNetworkingBandwidth.py), creating rule sets lazily via `_GetRules` when specific domains or contexts are first encountered. Users interact with these through the manager's interface rather than manipulating rule sets directly.