# How the Hydrus Content Update System Batches and Processes Tag Changes

> Learn how Hydrus batches and processes tag changes efficiently. Discover how ContentUpdate objects and packages minimize database trips for faster synchronization.

- Repository: [Hydrus Network Developer/hydrus](https://github.com/hydrusnetwork/hydrus)
- Tags: internals
- Published: 2026-03-03

---

**Hydrus aggregates individual tag operations into ContentUpdate objects, groups them by service in a ContentUpdatePackage, and processes them in bulk through ProcessContentUpdatePackage to minimize SQLite round-trips and synchronize mapping caches atomically.**

Hydrus is a personal media tagging application designed to handle massive collections with high-performance metadata operations. The content update system batch processes tag changes through a three-stage pipeline—conversion, batching, and bulk processing—to ensure that adding or removing thousands of tags triggers only a handful of database transactions.

## From Client Action to ContentUpdate

When the client initiates a tag change—such as adding the tag **cat** to a file—Hydrus first wraps the operation in a `ContentUpdate` object. In [`hydrus/client/metadata/ClientContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/metadata/ClientContentUpdates.py), the function `ConvertClientToServerUpdateToContentUpdates` translates high-level client actions into concrete update objects:

```python
def ConvertClientToServerUpdateToContentUpdates(client_to_server_update):
    content_updates = []
    for (action, content, reason) in client_to_server_update.IterateAllActionsAndContentsAndReasons():
        # translate pending/petition into add/delete for the server

        if action == HC.CONTENT_UPDATE_PEND:
            content_update_action = HC.CONTENT_UPDATE_ADD
        elif action == HC.CONTENT_UPDATE_PETITION:
            content_update_action = HC.CONTENT_UPDATE_DELETE
        else:
            continue
        row = content.GetContentData()
        content_update = ContentUpdate(content.GetContentType(),
                                      content_update_action,
                                      row,
                                      reason=reason)
        content_updates.append(content_update)
    return content_updates

```

*Source:* [ClientContentUpdates.py, lines 10–38](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/metadata/ClientContentUpdates.py#L10-L38)

Each `ContentUpdate` stores four critical fields:
- **data_type** – The content category (e.g., `HC.CONTENT_TYPE_MAPPINGS` for tag-to-file mappings)
- **action** – The operation type (add, delete, pend, petition, rescind-pend, rescind-petition)
- **row** – A tuple containing the tag and affected hashes
- **reason** – Optional text for petition actions

## Grouping Updates into a ContentUpdatePackage

To prepare for batch execution, Hydrus aggregates multiple `ContentUpdate` objects by their target **service key** (the unique identifier for a tag service). The `ContentUpdatePackage` class in [`hydrus/client/metadata/ClientContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/metadata/ClientContentUpdates.py) implements this grouping using a `defaultdict` structure:

```python
class ContentUpdatePackage(object):
    def __init__(self):
        self._service_keys_to_content_updates = collections.defaultdict(list)

    def AddContentUpdate(self, service_key, content_update):
        self.AddContentUpdates(service_key, (content_update,))

```

*Source:* [ClientContentUpdates.py, lines 13–24](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/metadata/ClientContentUpdates.py#L13-L24)

When the client is ready to persist changes—whether from UI actions or server synchronization—it passes the package to the database layer:

```python
client_controller.client_db.ProcessContentUpdatePackage(content_update_package)

```

This single call triggers the entire bulk processing pipeline, ensuring that all updates for a given service are handled in one atomic transaction.

## Bulk Processing in ProcessContentUpdatePackage

The core batching logic resides in `ClientDBContentUpdates.ProcessContentUpdatePackage` within [`hydrus/client/db/ClientDBContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBContentUpdates.py). This method extracts tag changes, validates them, and executes bulk SQL operations.

### Initial Validation and Normalization

The processor first filters out invalid services. As shown in lines 58–78 of [`ClientDBContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDBContentUpdates.py), the method iterates over the package and validates service keys:

```python
for (service_key, content_updates) in content_update_package.IterateContentUpdates():
    try:
        service_id = self.modules_services.GetServiceId(service_key)
    except HydrusExceptions.DataMissing:
        continue   # skip unknown services

    valid_content_update_package.AddContentUpdates(service_key, content_updates)
    service = self.modules_services.GetService(service_id)
    service_type = service.GetServiceType()

```

Only updates belonging to recognized services proceed to the processing stage.

### Tag-Service Branch and Aggregation

For real tag services (`HC.REAL_TAG_SERVICES`), the method examines each `ContentUpdate` with `data_type == HC.CONTENT_TYPE_MAPPINGS`. Between lines 78–165, it extracts the tag and hashes, normalizes the tag via `HydrusTags.CleanTag`, and converts identifiers to internal IDs (`tag_id` and `hash_ids`):

```python
elif data_type == HC.CONTENT_TYPE_MAPPINGS:
    (tag, hashes) = row
    # clean and validate tag

    tag = HydrusTags.CleanTag(tag)
    tag_id = self.modules_tags.GetTagId(tag)
    hash_ids = self.modules_hashes_local_cache.GetHashIds(hashes)

    # decide which aggregate list to push to

    if action == HC.CONTENT_UPDATE_ADD:
        ultimate_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_DELETE:
        ultimate_deleted_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_PEND:
        ultimate_pending_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_RESCIND_PEND:
        ultimate_pending_rescinded_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_PETITION:
        reason_id = self.modules_texts.GetTextId(content_update.GetReason())
        ultimate_petitioned_mappings_ids.append((tag_id, hash_ids, reason_id))
    elif action == HC.CONTENT_UPDATE_RESCIND_PETITION:
        ultimate_petitioned_rescinded_mappings_ids.append((tag_id, hash_ids))

```

*Source:* [ClientDBContentUpdates.py, lines 784–861](https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/client/db/ClientDBContentUpdates.py#L784-L861)

The method maintains six "ultimate" lists—`ultimate_mappings_ids`, `ultimate_deleted_mappings_ids`, `ultimate_pending_mappings_ids`, `ultimate_pending_rescinded_mappings_ids`, `ultimate_petitioned_mappings_ids`, and `ultimate_petitioned_rescinded_mappings_ids`—to accumulate all changes of each type before hitting the database.

### Database Writes and Cache Coordination

After aggregation, Hydrus executes bulk SQL operations using `executemany` for maximum throughput. The calls in lines 1060–1100 dispatch to [`ClientDBMappingsStorage.py`](https://github.com/hydrusnetwork/hydrus/blob/main/ClientDBMappingsStorage.py):

```python

# Add mappings

self.modules_mappings_storage.AddMappings(service_id, ultimate_mappings_ids)

# Delete mappings

self.modules_mappings_storage.DeleteMappings(service_id, ultimate_deleted_mappings_ids)

# Pend/Rescind pend

self.modules_mappings_storage.PendMappings(service_id, ultimate_pending_mappings_ids)
self.modules_mappings_storage.RescindPending(service_id, ultimate_pending_rescinded_mappings_ids)

# Petition/Rescind petition

self.modules_mappings_storage.PetitionMappings(service_id, ultimate_petitioned_mappings_ids)
self.modules_mappings_storage.RescindPetition(service_id, ultimate_petitioned_rescinded_mappings_ids)

```

Simultaneously, the same aggregated lists update the mapping caches (`ClientDBMappingsCacheSpecificStorage` and `ClientDBMappingsCacheSpecificDisplay`) to ensure tag display managers refresh efficiently. If changes affect visible tags, the method collects affected `hash_ids` into `self._regen_tags_managers_hash_ids` (lines 108–114), triggering a single UI refresh after the transaction commits rather than updating individual files incrementally.

## Practical Example: Batching Tag Additions

The following snippet demonstrates how a plugin or script would batch-add the tag **dog** to three files using the content update system:

```python
from hydrus.client import ClientController
from hydrus.client.metadata import ClientContentUpdates
from hydrus.core import HydrusConstants as HC

# 1️⃣ Gather service key for the default tag service

client = ClientController()
tag_service_key = client.services_manager.GetDefaultTagService().GetServiceKey()

# 2️⃣ Build a ContentUpdate for the mapping

hashes = ['a1b2c3', 'd4e5f6', '112233']          # human-readable hashes

content_update = ClientContentUpdates.ContentUpdate(
    data_type=HC.CONTENT_TYPE_MAPPINGS,
    action=HC.CONTENT_UPDATE_ADD,
    row=('dog', hashes)                         # (tag, [hashes])

)

# 3️⃣ Package it per service

package = ClientContentUpdates.ContentUpdatePackage()
package.AddContentUpdate(tag_service_key, content_update)

# 4️⃣ Hand it to the DB – this is where the whole batching logic runs

client.client_db.ProcessContentUpdatePackage(package)

```

When executed, this code resolves the tag to a `tag_id`, converts the three hashes to `hash_id`s, appends the tuple to `ultimate_mappings_ids`, and performs a **single bulk INSERT** into the tag-mapping table. The specific mapping caches update immediately, and the UI regenerates tag displays for only the affected files.

## Summary

Hydrus delivers high-performance tag modifications through a carefully designed batching architecture:

- **Conversion** – Raw client actions become structured `ContentUpdate` objects containing typed data, actions, and metadata.
- **Aggregation** – `ContentUpdatePackage` groups updates by service key, preparing them for atomic processing.
- **Bulk Execution** – `ProcessContentUpdatePackage` validates services, aggregates changes into "ultimate" lists by action type, and issues multi-row SQL statements via `executemany`.
- **Cache Synchronization** – The same batched data updates mapping caches and triggers targeted UI refreshes without per-file overhead.

This pipeline ensures that importing or modifying thousands of tags generates minimal database churn while maintaining consistent, real-time cache states across the application.

## Frequently Asked Questions

### What is a ContentUpdatePackage in Hydrus?

A **ContentUpdatePackage** is a container class defined in [`hydrus/client/metadata/ClientContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/metadata/ClientContentUpdates.py) that aggregates multiple `ContentUpdate` objects by their target service key. It acts as the primary data structure for batching tag operations before they enter the database processing pipeline, ensuring that all changes targeting the same tag service can be committed in a single atomic transaction.

### How does Hydrus handle different tag actions like pending or petitioning?

Hydrus processes **pend** and **petition** actions by routing them to dedicated "ultimate" lists—`ultimate_pending_mappings_ids` and `ultimate_petitioned_mappings_ids`—inside `ProcessContentUpdatePackage`. According to the logic in [`hydrus/client/db/ClientDBContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBContentUpdates.py), pending actions insert rows into pending tables, while petitions store the request along with a `reason_id`. Rescind actions remove entries from these pending or petition tables rather than the main mapping storage.

### Why does Hydrus use "ultimate" lists during tag processing?

The **"ultimate" lists** (such as `ultimate_mappings_ids` and `ultimate_deleted_mappings_ids`) serve as aggregation buffers that collect all tag changes of a specific type before database insertion. This design allows Hydrus to execute a single `INSERT OR REPLACE` statement for hundreds or thousands of tag mappings using SQLite's `executemany`, dramatically reducing transaction overhead compared to processing individual rows one at a time.

### Which source files control the tag content update pipeline?

The pipeline spans four critical files: [`hydrus/client/metadata/ClientContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/metadata/ClientContentUpdates.py) defines the `ContentUpdate` and `ContentUpdatePackage` classes; [`hydrus/client/db/ClientDBContentUpdates.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBContentUpdates.py) contains the `ProcessContentUpdatePackage` batch processor; [`hydrus/client/db/ClientDBMappingsStorage.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/client/db/ClientDBMappingsStorage.py) handles the low-level SQL inserts and deletes; and [`hydrus/core/HydrusTags.py`](https://github.com/hydrusnetwork/hydrus/blob/main/hydrus/core/HydrusTags.py) provides tag normalization via `CleanTag` before database storage.