How Hydrus Manages Physical File Storage Locations: A Deep Dive into Hash-Based Directory Trees

Hydrus stores every imported file in a hierarchical, prefix-based directory tree derived from SHA-256 hashes, distributing files across weighted base locations using deterministic folder generation.

The hydrusnetwork/hydrus client manages massive media collections through a sophisticated physical file storage system that translates cryptographic hashes into predictable folder paths. This architecture balances storage across multiple disks while enabling seamless reorganization and retrieval. Understanding how Hydrus manages file locations reveals a design optimized for scalability, deterministic access, and dynamic capacity management.

Base Locations and Weighted Storage Distribution

Hydrus organizes storage into base locations—root directories configured with specific capacity limits and relative weights. The system selects where to write new files based on which location is furthest from its ideal storage weight.

FilesStorageBaseLocation Architecture

The FilesStorageBaseLocation class in hydrus/client/files/ClientFilesPhysical.py encapsulates each storage root. According to the source code (lines 77‑86), each location maintains:

• A real path for absolute file system access
• A portable path for cross-platform compatibility
• An ideal weight defining the target percentage of total storage
• An optional max-bytes limit to prevent disk overflow

Capacity checks (lines 41‑57) determine whether a location can accept additional sub-folders or requires clearing. When the client evaluates write targets, it calculates current weight distributions and selects the location with the greatest deficit relative to its ideal allocation.

Weight Distribution Algorithm

The static method STATICGetIdealWeights (referenced in lines 27‑110) implements the distribution logic. When multiple base locations are defined in the client database under service HYDRUS_LOCAL_FILE_STORAGE, Hydrus computes the optimal prefix allocation across locations. The algorithm prioritizes locations with available capacity and the largest negative deviation from their target weight, ensuring balanced utilization across heterogeneous storage configurations.

Prefix Generation from SHA-256 Hashes

All file placement derives from the content’s SHA-256 hash, transformed into a hexadecimal string and segmented into folder names.

GetPrefix Implementation and Granularity

The core prefix generation resides in hydrus/core/files/HydrusFilesPhysicalStorage.py. The function GetPrefix(hash, prefix_type, prefix_length) (lines 31‑34) extracts segments from the hexadecimal hash representation:

from hydrus.core.files import HydrusFilesPhysicalStorage

hash_bytes = b'\x3a\x5f\xf9...'  # 32-byte SHA-256 digest

prefix = HydrusFilesPhysicalStorage.GetPrefix(
    hash_bytes, 
    'f',  # 'f' for raw files, 't' for thumbnails

    HydrusFilesPhysicalStorage.DEFAULT_PREFIX_LENGTH  # defaults to 2

)

# Returns: "f3a5"

The default granularity is defined as DEFAULT_PREFIX_LENGTH = 2, creating 256 top-level folders per location (16² combinations). Administrators may increase this to 3 or higher to reduce per-folder file counts, generating up to 4,096 folders (16³).

Prefix Types and Folder Hierarchy

The prefix system distinguishes content types through the first character:

• 'f' prefixes denote raw imported files
• 't' prefixes denote generated thumbnails

The remaining characters form the directory hierarchy. For a prefix length of 2, the structure creates folders like f3/a5/ beneath the base location root.

Subfolder Resolution and Path Construction

Once Hydrus determines the appropriate base location and calculates the hash prefix, it constructs the nested directory structure through the FilesStorageSubfolder class.

FilesStorageSubfolder Mechanics

Located in hydrus/client/files/ClientFilesPhysical.py (lines 48‑56), this class splits the hexadecimal portion of the prefix into two-character chunks. A prefix of f3a5 becomes the array ['f3', 'a5'], which the system joins onto the base location’s root path to form the absolute folder path.

The class exposes GetFilePath(filename) to append the actual filename—typically the full hexadecimal hash with extension—to the constructed directory path. This ensures every file rests at a predictable location: <base>/<chunk1>/<chunk2>/<hash>.ext.

Building Absolute File Paths

When the client retrieves a file, it follows this resolution chain:

from hydrus.core.files import HydrusFilesPhysicalStorage
from hydrus.client.files.ClientFilesPhysical import FilesStorageBaseLocation, FilesStorageSubfolder

# Base location normally loaded from database configuration

base = FilesStorageBaseLocation('/home/user/hydrus/storage', ideal_weight=1)

hash_bytes = b'\x3a\x5f\xf9...'
prefix = HydrusFilesPhysicalStorage.GetPrefix(
    hash_bytes, 'f', 
    HydrusFilesPhysicalStorage.DEFAULT_PREFIX_LENGTH
)

subfolder = FilesStorageSubfolder(prefix, base)
filename = f'{hash_bytes.hex()}.ext'
full_path = subfolder.GetFilePath(filename)

# Result: /home/user/hydrus/storage/f3/a5/3a5ff9....ext

This mirrors the lookup performed in hydrus/client/files/ClientFilesManager.py (line 508), where the client calls HydrusFilesPhysicalStorage.GetPrefix to resolve file requests.

Rebalancing Storage with RegranulariseFileStorage

Hydrus supports dynamic reorganization through the RegranulariseFileStorage function in hydrus/client/files/ClientFilesPhysical.py, enabling administrators to change folder granularity or redistribute files across base locations without breaking references.

Changing Granularity Levels

The re-granulation process (lines 91‑124) accepts a starting prefix length (e.g., 2) and a target length (e.g., 3). It enumerates all existing source subfolders using IteratePrefixes (lines 36‑46 in the core storage module), then computes destination prefixes for the new granularity level.

Moving Files Between Locations

The rebalancing workflow (lines 150‑210) scans each source subfolder and matches files to their destination prefixes based on hash analysis. It builds rename jobs mapping source_path → destination_path, executes them in batches (lines 222‑242), and cleans up empty source directories (lines 245‑251).

from hydrus.client.files.ClientFilesPhysical import RegranulariseFileStorage
from hydrus.client import ClientThreading

job_status = ClientThreading.JobStatus('Regranularising', total_work=1)
base_paths = ['/home/user/hydrus/storage']

new_map, moved, weird_dirs, weird_files = RegranulariseFileStorage(
    base_location_paths=base_paths,
    prefix_types=['f', 't'],
    starting_prefix_length=2,
    ending_prefix_length=3,
    job_status=job_status
)

The function returns a mapping of new prefixes to canonical base-location paths, along with statistics on moved files and any non-conforming directories requiring manual intervention.

Summary

• Base locations (FilesStorageBaseLocation) define weighted root directories with capacity limits and portable path handling
• Prefix generation (HydrusFilesPhysicalStorage.GetPrefix) creates deterministic folder names from SHA-256 hashes, using 2-character granularity by default
• Subfolder construction (FilesStorageSubfolder) splits prefixes into two-character chunks to build nested directory hierarchies
• File resolution combines base locations, prefixes, and filenames to produce absolute paths without database lookups during retrieval
• Rebalancing (RegranulariseFileStorage) enables live migration between granularity levels and across storage locations while maintaining hash-based organization

Frequently Asked Questions

How does Hydrus determine which hard drive to store a file on?

Hydrus evaluates all configured base locations and selects the one whose current storage weight is furthest below its ideal weight while still having available capacity. The FilesStorageBaseLocation class tracks ideal weights and max-bytes limits, and the selection algorithm prioritizes locations with the largest negative deviation from their target allocation to maintain balanced distribution across multiple disks.

What is the default folder structure depth in Hydrus storage?

The default configuration uses a prefix length of 2 (DEFAULT_PREFIX_LENGTH = 2), creating 256 top-level folders per base location. Each folder contains subfolders named from subsequent two-character pairs of the hexadecimal hash, typically resulting in paths like f3/a5/ beneath the storage root. This provides 65,536 possible leaf directories (256 × 256) for file distribution.

Can I change the folder granularity after importing files?

Yes. The RegranulariseFileStorage function in hydrus/client/files/ClientFilesPhysical.py supports live re-granulation from any source prefix length to a different target length. The system enumerates existing prefixes, computes new destinations based on hash analysis, and performs batched file moves while updating internal mappings, allowing migration from 2-character to 3-character granularity without data loss.

How does Hydrus handle thumbnail storage differently from raw files?

Thumbnails use the prefix type 't' rather than 'f', causing them to reside in parallel directory trees (e.g., t3/a5/ instead of f3/a5/). The GetPrefix function accepts a prefix_type parameter that directs files into separate hierarchies within the same base locations, enabling independent granularity management and storage weighting for thumbnails versus raw media files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}