How Hydrus Throttles Network Requests: A Deep Dive into Its Bandwidth Management System

Hydrus implements a two-layer bandwidth control system that combines granular usage tracking via BandwidthTracker with policy enforcement via BandwidthRules to decide whether network jobs may start, continue, or must wait across five time granularities.

The hydrusnetwork/hydrus client manages thousands of concurrent download jobs across multiple domains, subscriptions, and gallery crawlers. To prevent overwhelming servers or hitting ISP caps, the application employs a sophisticated bandwidth management system that throttles network requests through per-context accounting and configurable rate limits.

BandwidthTracker: Granular Usage Accounting

At the foundation of the throttling mechanism lies the BandwidthTracker class, implemented in [hydrus/core/networking/HydrusNetworking.py](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/core/networking/HydrusNetworking.py#L305-L400). Each network context—whether global, domain-specific, or tied to a specific subscription—maintains its own tracker instance.

Five Time-Granularity Counters

The tracker maintains five distinct time-granularity counters for both data volume and request frequency:

• _seconds_* for real-time second-level tracking
• _minutes_* for minute-level aggregation
• _hours_* for hourly windows
• _days_* for daily caps
• _months_* for monthly quotas

When a job completes a transfer, it reports usage through ReportDataUsed(num_bytes) or ReportRequestUsed(). The tracker automatically buckets this usage into the appropriate counter based on the current UTC timestamp. Querying usage via GetUsage(bandwidth_type, time_delta) allows the system to check consumption against any look-back window, where bandwidth_type is either HC.BANDWIDTH_TYPE_DATA (bytes) or HC.BANDWIDTH_TYPE_REQUESTS.

# Record 1 KB of data transferred

bandwidth_tracker.ReportDataUsed(1024)

# Record a single HTTP request

bandwidth_tracker.ReportRequestUsed()

BandwidthRules: Policy Enforcement

While BandwidthTracker records history, the BandwidthRules class—found in [hydrus/core/networking/HydrusNetworking.py](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/core/networking/HydrusNetworking.py#L14-L85)—defines the actual throttling policies. Rules are stored as tuples of (bandwidth_type, time_delta, max_allowed) in a thread-safe set.

Rule Evaluation Methods

The class provides three core methods for throttling decisions:

• CanStartRequest(tracker, threshold=5): Returns False if any data rule would be exceeded, preventing new connections when bandwidth is scarce. Request-type rules are ignored for start checks to avoid deadlocks.
• CanDoWork(tracker, expected_requests, expected_bytes, threshold=30): Used by batch operations like subscriptions; discounts expected usage from limits before checking if the entire batch should proceed.
• CanContinueDownload(tracker, threshold=15): Called during active transfers to determine if a large file download should pause mid-stream when approaching hourly or monthly caps.

# Allow 10 MiB per hour

rules.AddRule(HC.BANDWIDTH_TYPE_DATA, 3600, 10 * 1024 * 1024)

# Allow 100 requests per minute  

rules.AddRule(HC.BANDWIDTH_TYPE_REQUESTS, 60, 100)

NetworkBandwidthManager: Per-Context Coordination

The NetworkBandwidthManager in [hydrus/client/networking/ClientNetworkingBandwidth.py](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/networking/ClientNetworkingBandwidth.py#L58-L150) orchestrates trackers and rules across different network contexts. It maintains a default tracker (_my_bandwidth_tracker) for the client itself and a dictionary of tracker containers keyed by NetworkContext objects (defined in ClientNetworkingContexts.py).

When the networking engine evaluates a job, it calls _CanStartRequest(network_contexts), which iterates through all relevant contexts—global, domain, subscription—and verifies that each satisfies its respective rules:

def _CanStartRequest(self, network_contexts):
    for network_context in network_contexts:
        rules = self._GetRules(network_context)
        tracker = self._GetTracker(network_context)
        if not rules.CanStartRequest(tracker):
            return False
    return True

The manager creates fresh BandwidthTracker instances lazily via _GetTracker, wrapping them in NetworkBandwidthManagerTrackerContainer objects. Upon request completion, _ReportDataUsed and _ReportRequestUsed propagate usage statistics to all relevant trackers simultaneously, including the global bandwidth tracker.

Throttling in Action: From Job Admission to Mid-Download Checks

The bandwidth management system integrates into the request lifecycle at three critical points, as implemented in the main networking engine at [hydrus/client/networking/ClientNetworking.py](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/networking/ClientNetworking.py#L293-L300).

Job Admission Control

Before moving a job from awaiting to running, the engine invokes _CanStartRequest. If any context reports insufficient quota, the job remains queued until the time window rolls over or competing jobs release bandwidth.

Mid-Download Interruption

For large file downloads, CanContinueDownload checks whether completing the transfer would breach data caps. This prevents overshooting monthly limits when downloading multi-gigabyte archives.

Batch Work Regulation

Import subscriptions and gallery crawlers call CanDoWork with estimates of their planned byte and request consumption. The manager may delay the entire batch until sufficient quota accumulates, preventing partial job executions that would waste server resources.

Practical Implementation Examples

Adding a Custom Domain Rule

To limit a specific domain to 5 MiB per hour:

from hydrus.core import HydrusConstants as HC
from hydrus.client.networking import ClientNetworkingBandwidth, ClientNetworkingContexts

bw_manager = ClientNetworkingBandwidth.NetworkBandwidthManager()

domain_ctx = ClientNetworkingContexts.NetworkContext(
    context_type=CC.NETWORK_CONTEXT_DOMAIN,
    identifier='example.com'
)

rules = bw_manager._GetRules(domain_ctx)
rules.AddRule(HC.BANDWIDTH_TYPE_DATA, 3600, 5 * 1024 * 1024)

Simulating Throttle Conditions

When the hourly quota is exhausted, subsequent requests block:

tracker = bw_manager._GetTracker(domain_ctx)
tracker.ReportDataUsed(5 * 1024 * 1024)  # Consume full quota

if not bw_manager._CanStartRequest([domain_ctx]):
    print('Request throttled – bandwidth limit reached')

Querying Current Usage

Display bandwidth statistics in the UI by querying specific time windows:

from hydrus.core import HydrusNumbers, HydrusData

global_tracker = bw_manager._my_bandwidth_tracker

# Bytes used this month

month_bytes = global_tracker.GetUsage(HC.BANDWIDTH_TYPE_DATA, None)
print('Month usage:', HydrusData.ConvertValueRangeToBytes(month_bytes, None))

# Requests in last minute

recent_reqs = global_tracker.GetUsage(HC.BANDWIDTH_TYPE_REQUESTS, 60)
print('Requests last minute:', HydrusNumbers.ValueRangeToPrettyString(recent_reqs, None))

Summary

• Two-layer architecture: BandwidthTracker handles usage accounting while BandwidthRules enforces policies across five time granularities (seconds to months).
• Per-context isolation: Each domain, subscription, and the global client maintain separate trackers and rule sets via NetworkBandwidthManager.
• Three-phase throttling: The system controls job admission (CanStartRequest), batch approval (CanDoWork), and mid-download continuation (CanContinueDownload) with configurable thresholds of 5, 30, and 15 respectively.
• Thread-safe reporting: Usage propagates to all relevant context trackers simultaneously when requests complete.
• Implementation locations: Core logic resides in HydrusNetworking.py (tracker/rules) and ClientNetworkingBandwidth.py (manager coordination).

Frequently Asked Questions

What are the five time granularities tracked by the bandwidth management system?

The BandwidthTracker class maintains counters for seconds, minutes, hours, days, and months. This allows the system to enforce both burst-rate limiting (per-second) and long-term quotas (per-month) simultaneously using the same accounting infrastructure.

How does Hydrus decide whether to start a new network request?

Before starting a job, the networking engine calls _CanStartRequest, which checks every relevant NetworkContext (global, domain, subscription). For each context, it retrieves the BandwidthRules and queries CanStartRequest(tracker, threshold=5). If any rule would be exceeded, the job remains queued. Notably, this check only evaluates data rules, not request-count rules, to prevent starvation scenarios.

What is the difference between CanStartRequest and CanContinueDownload?

CanStartRequest (threshold=5) evaluates whether a new connection should be allowed to initiate, while CanContinueDownload (threshold=15) determines if an already-active large file transfer should proceed or pause. The higher threshold for continuation provides slack for in-progress transfers while still protecting against quota overruns.

Where are bandwidth rules stored and how are they accessed?

Rules are stored as tuples in thread-safe set objects within BandwidthRules instances. The NetworkBandwidthManager organizes these by NetworkContext in ClientNetworkingBandwidth.py, creating rule sets lazily via _GetRules when specific domains or contexts are first encountered. Users interact with these through the manager's interface rather than manipulating rule sets directly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}