How the Hydrus URL Mapping System Associates URLs with Files

Hydrus stores a many-to-many relationship between media files (identified by hash IDs) and their source URLs using a dedicated SQLite table (url_map) that links a file’s hash_id to normalized URL records in the master database.

The hydrusnetwork/hydrus media library tracks the origin of every imported file through a robust URL mapping system. This subsystem maintains a persistent many-to-many relationship between media hashes and the URLs where they were discovered, enabling reverse lookups and provenance tracking. Understanding this architecture is essential for developers extending the client or troubleshooting database integrity issues.

Core Database Schema

The URL mapping system relies on three interconnected SQLite tables that normalize URL storage and eliminate redundancy.

Master URL Tables (defined in hydrus/client/db/ClientDBMaster.py):

• external_master.url_domains (lines 822-823): Stores unique domain strings with a domain_id primary key and a domain TEXT column.
• external_master.urls (lines 822-823): Stores each full URL once with url_id, domain_id, and url TEXT columns.

Mapping Table (defined in hydrus/client/db/ClientDBURLMap.py):

• main.url_map (line 37): Contains a composite primary key (hash_id, url_id) that creates the many-to-many relationship between files and URLs.

This schema ensures that a single URL record can associate with multiple files, and a single file can originate from multiple URLs.

Adding URL Mappings During Import

When a file is imported with a known source URL, Hydrus calls ClientDBURLMap.AddMapping to persist the relationship.

def AddMapping(self, hash_id: int, url: str):
    url_id = self.modules_urls.GetURLId(url)          # look up or create URL row

    self._Execute('INSERT OR IGNORE INTO url_map (hash_id, url_id) VALUES (?, ?);',
                  (hash_id, url_id))

The method uses INSERT OR IGNORE to guarantee idempotency—a file can only be linked to a particular URL once.

Resolving URLs to IDs

The self.modules_urls object is an instance of ClientDBMaster.ClientDBMasterURLs, which implements GetURLId (see hydrus/client/db/ClientDBMaster.py, lines 871-894). This method normalizes the URL string and returns the existing url_id or creates a new entry in the master tables if the URL has never been seen.

Removing URL Associations

To disassociate a file from a URL, ClientDBURLMap.DeleteMapping performs the inverse operation:

def DeleteMapping(self, hash_id: int, url: str):
    url_id = self.modules_urls.GetURLId(url)
    self._Execute('DELETE FROM url_map WHERE hash_id = ? AND url_id = ?;',
                  (hash_id, url_id))

The same GetURLId lookup is used to resolve the url_id before deletion, ensuring consistency with the master tables.

Querying Files by URL

To find all files that reference a given URL, ClientDBURLMap.GetHashIds performs a natural join between url_map and the master urls table:

def GetHashIds(self, search_url: str):
    # SELECT hash_id FROM url_map NATURAL JOIN urls WHERE url = ?

    hash_ids = self._STS(self._Execute('SELECT hash_id FROM url_map '
                                      'NATURAL JOIN urls WHERE url = ?;',
                                      (search_url,)))
    return hash_ids

The result is a set of hash_id integers that can be resolved into FileInfo objects elsewhere in the client.

Advanced URL Class Queries

Hydrus supports richer "URL class" rules involving domain masks and regex patterns. ClientDBURLMap analyzes these rules in GetHashIdsFromCountTests (lines 125-190) and uses the master URL tables to resolve domain masks (ClientNetworkingURLClass.URLDomainMask) to matching domain_ids via GetURLDomainAndSubdomainIds in ClientDBMaster.py (lines 852-866).

Maintenance and Orphan Cleanup

Because the mapping table can survive a corrupted client.master.db, Hydrus provides a maintenance command to purge entries that reference deleted files.

The GUI entry is defined in hydrus/client/gui/ClientGUI.py (line 3372), while the implementation lives in hydrus/client/db/ClientDB.py as the method _ClearOrphanURLMappings (line 1151). This routine identifies hash_ids that no longer exist in the files table and deletes their corresponding rows from url_map, preserving database integrity.

Import Safety Checks

During the import process, Hydrus checks for "untrustworthy neighbour URLs" to prevent spam or mass-linking attacks. This logic lives in hydrus/client/importing/ClientImportFileSeeds.py within the function FileURLMappingHasUntrustworthyNeighbours (line 75). If a candidate URL fails the neighbour-spam test, the import skips adding the mapping entirely.

Practical Code Examples

Adding a URL mapping after importing a file

from hydrus.client.db.clientdb import ClientDB
from hydrus.client.db.module import ClientDBMaster

# Assume we already have a DB connection (client_db) and a hash_id for the file

url = 'https://example.com/image.jpg'

client_db = ClientDB()
hash_id = client_db.modules_files.GetHashIdFromHash(file_hash)   # get hash_id

client_db.modules_url_map.AddMapping(hash_id, url)

Behind the scenes: AddMapping resolves the URL in the master tables (creating rows if necessary) and inserts the (hash_id, url_id) pair into url_map.

Finding all files that reference a URL

hash_ids = client_db.modules_url_map.GetHashIds('https://example.com/image.jpg')
for h_id in hash_ids:
    file_info = client_db.modules_files.GetFileInfoFromHashId(h_id)
    print(file_info.GetFilePath())

The call returns the set of hash_ids that are linked to the supplied URL.

Removing a mapping manually

client_db.modules_url_map.DeleteMapping(hash_id, 'https://example.com/image.jpg')

Clearing orphan mappings

client_db._ClearOrphanURLMappings()

All entries in url_map that reference a non-existent hash_id are purged, keeping the database tidy.

Summary

• Hydrus uses a normalized three-table schema where url_map creates many-to-many relationships between hash_id and url_id.
• The ClientDBURLMap class in hydrus/client/db/ClientDBURLMap.py provides the primary API for creating, deleting, and querying these associations.
• URL normalization occurs in ClientDBMaster, ensuring duplicate URLs share the same url_id across the database.
• Maintenance routines like _ClearOrphanURLMappings prevent dangling references when files are deleted.
• Safety checks during import prevent the URL mapping system from being exploited by spam or malicious neighbour URLs.

Frequently Asked Questions

What database tables are involved in the URL mapping system?

The system uses external_master.url_domains and external_master.urls (defined in ClientDBMaster.py) to store normalized URL data, and main.url_map (defined in ClientDBURLMap.py) to store the actual many-to-many relationships between hash IDs and URL IDs.

How does Hydrus prevent duplicate URL mappings for the same file?

The AddMapping method in ClientDBURLMap.py uses an INSERT OR IGNORE SQL statement with a composite primary key (hash_id, url_id), ensuring that a specific file-URL pair can only exist once in the url_map table.

What happens to URL mappings when a file is deleted?

The url_map entries persist until explicitly cleaned. Hydrus provides the _ClearOrphanURLMappings method in ClientDB.py (accessible via the GUI) to purge mappings where the hash_id no longer exists in the files table, maintaining referential integrity.

How does Hydrus handle URL normalization before creating mappings?

Before inserting into url_map, the system calls ClientDBMaster.ClientDBMasterURLs.GetURLId (lines 871-894), which normalizes the URL string and either returns an existing url_id or creates a new normalized entry in the master URL tables, ensuring consistent storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}