# How the Hydrus URL Mapping System Associates URLs with Files

> Discover how the Hydrus URL mapping system associates URLs with files using hash IDs and an SQLite table. Learn about this core feature of the Hydrus network repository.

- Repository: [Hydrus Network Developer/hydrus](https://github.com/hydrusnetwork/hydrus)
- Tags: internals
- Published: 2026-03-03

---

**Hydrus stores a many-to-many relationship between media files (identified by hash IDs) and their source URLs using a dedicated SQLite table (`url_map`) that links a file’s `hash_id` to normalized URL records in the master database.**

The hydrusnetwork/hydrus media library tracks the origin of every imported file through a robust URL mapping system. This subsystem maintains a persistent many-to-many relationship between media hashes and the URLs where they were discovered, enabling reverse lookups and provenance tracking. Understanding this architecture is essential for developers extending the client or troubleshooting database integrity issues.

## Core Database Schema

The URL mapping system relies on three interconnected SQLite tables that normalize URL storage and eliminate redundancy.

**Master URL Tables** (defined in [`hydrus/client/db/ClientDBMaster.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBMaster.py)):

- `external_master.url_domains` (lines 822-823): Stores unique domain strings with a `domain_id` primary key and a `domain` TEXT column.
- `external_master.urls` (lines 822-823): Stores each full URL once with `url_id`, `domain_id`, and `url` TEXT columns.

**Mapping Table** (defined in [`hydrus/client/db/ClientDBURLMap.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBURLMap.py)):

- `main.url_map` (line 37): Contains a composite primary key `(hash_id, url_id)` that creates the many-to-many relationship between files and URLs.

This schema ensures that a single URL record can associate with multiple files, and a single file can originate from multiple URLs.

## Adding URL Mappings During Import

When a file is imported with a known source URL, Hydrus calls `ClientDBURLMap.AddMapping` to persist the relationship.

```python
def AddMapping(self, hash_id: int, url: str):
    url_id = self.modules_urls.GetURLId(url)          # look up or create URL row

    self._Execute('INSERT OR IGNORE INTO url_map (hash_id, url_id) VALUES (?, ?);',
                  (hash_id, url_id))

```

The method uses `INSERT OR IGNORE` to guarantee idempotency—a file can only be linked to a particular URL once.

### Resolving URLs to IDs

The `self.modules_urls` object is an instance of `ClientDBMaster.ClientDBMasterURLs`, which implements `GetURLId` (see [`hydrus/client/db/ClientDBMaster.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBMaster.py), lines 871-894). This method normalizes the URL string and returns the existing `url_id` or creates a new entry in the master tables if the URL has never been seen.

## Removing URL Associations

To disassociate a file from a URL, `ClientDBURLMap.DeleteMapping` performs the inverse operation:

```python
def DeleteMapping(self, hash_id: int, url: str):
    url_id = self.modules_urls.GetURLId(url)
    self._Execute('DELETE FROM url_map WHERE hash_id = ? AND url_id = ?;',
                  (hash_id, url_id))

```

The same `GetURLId` lookup is used to resolve the `url_id` before deletion, ensuring consistency with the master tables.

## Querying Files by URL

To find all files that reference a given URL, `ClientDBURLMap.GetHashIds` performs a natural join between `url_map` and the master `urls` table:

```python
def GetHashIds(self, search_url: str):
    # SELECT hash_id FROM url_map NATURAL JOIN urls WHERE url = ?

    hash_ids = self._STS(self._Execute('SELECT hash_id FROM url_map '
                                      'NATURAL JOIN urls WHERE url = ?;',
                                      (search_url,)))
    return hash_ids

```

The result is a set of `hash_id` integers that can be resolved into `FileInfo` objects elsewhere in the client.

### Advanced URL Class Queries

Hydrus supports richer "URL class" rules involving domain masks and regex patterns. `ClientDBURLMap` analyzes these rules in `GetHashIdsFromCountTests` (lines 125-190) and uses the master URL tables to resolve domain masks (`ClientNetworkingURLClass.URLDomainMask`) to matching `domain_id`s via `GetURLDomainAndSubdomainIds` in [`ClientDBMaster.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDBMaster.py) (lines 852-866).

## Maintenance and Orphan Cleanup

Because the mapping table can survive a corrupted `client.master.db`, Hydrus provides a maintenance command to purge entries that reference deleted files.

The GUI entry is defined in [`hydrus/client/gui/ClientGUI.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/gui/ClientGUI.py) (line 3372), while the implementation lives in [`hydrus/client/db/ClientDB.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDB.py) as the method `_ClearOrphanURLMappings` (line 1151). This routine identifies `hash_id`s that no longer exist in the `files` table and deletes their corresponding rows from `url_map`, preserving database integrity.

## Import Safety Checks

During the import process, Hydrus checks for "untrustworthy neighbour URLs" to prevent spam or mass-linking attacks. This logic lives in [`hydrus/client/importing/ClientImportFileSeeds.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/importing/ClientImportFileSeeds.py) within the function `FileURLMappingHasUntrustworthyNeighbours` (line 75). If a candidate URL fails the neighbour-spam test, the import skips adding the mapping entirely.

## Practical Code Examples

### Adding a URL mapping after importing a file

```python
from hydrus.client.db.clientdb import ClientDB
from hydrus.client.db.module import ClientDBMaster

# Assume we already have a DB connection (client_db) and a hash_id for the file

url = 'https://example.com/image.jpg'

client_db = ClientDB()
hash_id = client_db.modules_files.GetHashIdFromHash(file_hash)   # get hash_id

client_db.modules_url_map.AddMapping(hash_id, url)

```

*Behind the scenes:* `AddMapping` resolves the URL in the master tables (creating rows if necessary) and inserts the `(hash_id, url_id)` pair into `url_map`.

### Finding all files that reference a URL

```python
hash_ids = client_db.modules_url_map.GetHashIds('https://example.com/image.jpg')
for h_id in hash_ids:
    file_info = client_db.modules_files.GetFileInfoFromHashId(h_id)
    print(file_info.GetFilePath())

```

The call returns the set of `hash_id`s that are linked to the supplied URL.

### Removing a mapping manually

```python
client_db.modules_url_map.DeleteMapping(hash_id, 'https://example.com/image.jpg')

```

### Clearing orphan mappings

```python
client_db._ClearOrphanURLMappings()

```

All entries in `url_map` that reference a non-existent `hash_id` are purged, keeping the database tidy.

## Summary

- **Hydrus uses a normalized three-table schema** where `url_map` creates many-to-many relationships between `hash_id` and `url_id`.
- **The `ClientDBURLMap` class** in [`hydrus/client/db/ClientDBURLMap.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBURLMap.py) provides the primary API for creating, deleting, and querying these associations.
- **URL normalization occurs in `ClientDBMaster`**, ensuring duplicate URLs share the same `url_id` across the database.
- **Maintenance routines** like `_ClearOrphanURLMappings` prevent dangling references when files are deleted.
- **Safety checks** during import prevent the URL mapping system from being exploited by spam or malicious neighbour URLs.

## Frequently Asked Questions

### What database tables are involved in the URL mapping system?

The system uses `external_master.url_domains` and `external_master.urls` (defined in [`ClientDBMaster.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDBMaster.py)) to store normalized URL data, and `main.url_map` (defined in [`ClientDBURLMap.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDBURLMap.py)) to store the actual many-to-many relationships between hash IDs and URL IDs.

### How does Hydrus prevent duplicate URL mappings for the same file?

The `AddMapping` method in [`ClientDBURLMap.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDBURLMap.py) uses an `INSERT OR IGNORE` SQL statement with a composite primary key `(hash_id, url_id)`, ensuring that a specific file-URL pair can only exist once in the `url_map` table.

### What happens to URL mappings when a file is deleted?

The `url_map` entries persist until explicitly cleaned. Hydrus provides the `_ClearOrphanURLMappings` method in [`ClientDB.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDB.py) (accessible via the GUI) to purge mappings where the `hash_id` no longer exists in the files table, maintaining referential integrity.

### How does Hydrus handle URL normalization before creating mappings?

Before inserting into `url_map`, the system calls `ClientDBMaster.ClientDBMasterURLs.GetURLId` (lines 871-894), which normalizes the URL string and either returns an existing `url_id` or creates a new normalized entry in the master URL tables, ensuring consistent storage.