How the Hydrus Content Update System Batches and Processes Tag Changes

Hydrus aggregates individual tag operations into ContentUpdate objects, groups them by service in a ContentUpdatePackage, and processes them in bulk through ProcessContentUpdatePackage to minimize SQLite round-trips and synchronize mapping caches atomically.

Hydrus is a personal media tagging application designed to handle massive collections with high-performance metadata operations. The content update system batch processes tag changes through a three-stage pipeline—conversion, batching, and bulk processing—to ensure that adding or removing thousands of tags triggers only a handful of database transactions.

From Client Action to ContentUpdate

When the client initiates a tag change—such as adding the tag cat to a file—Hydrus first wraps the operation in a ContentUpdate object. In hydrus/client/metadata/ClientContentUpdates.py, the function ConvertClientToServerUpdateToContentUpdates translates high-level client actions into concrete update objects:

def ConvertClientToServerUpdateToContentUpdates(client_to_server_update):
    content_updates = []
    for (action, content, reason) in client_to_server_update.IterateAllActionsAndContentsAndReasons():
        # translate pending/petition into add/delete for the server

        if action == HC.CONTENT_UPDATE_PEND:
            content_update_action = HC.CONTENT_UPDATE_ADD
        elif action == HC.CONTENT_UPDATE_PETITION:
            content_update_action = HC.CONTENT_UPDATE_DELETE
        else:
            continue
        row = content.GetContentData()
        content_update = ContentUpdate(content.GetContentType(),
                                      content_update_action,
                                      row,
                                      reason=reason)
        content_updates.append(content_update)
    return content_updates

Source: ClientContentUpdates.py, lines 10–38

Each ContentUpdate stores four critical fields:

• data_type – The content category (e.g., HC.CONTENT_TYPE_MAPPINGS for tag-to-file mappings)
• action – The operation type (add, delete, pend, petition, rescind-pend, rescind-petition)
• row – A tuple containing the tag and affected hashes
• reason – Optional text for petition actions

Grouping Updates into a ContentUpdatePackage

To prepare for batch execution, Hydrus aggregates multiple ContentUpdate objects by their target service key (the unique identifier for a tag service). The ContentUpdatePackage class in hydrus/client/metadata/ClientContentUpdates.py implements this grouping using a defaultdict structure:

class ContentUpdatePackage(object):
    def __init__(self):
        self._service_keys_to_content_updates = collections.defaultdict(list)

    def AddContentUpdate(self, service_key, content_update):
        self.AddContentUpdates(service_key, (content_update,))

Source: ClientContentUpdates.py, lines 13–24

When the client is ready to persist changes—whether from UI actions or server synchronization—it passes the package to the database layer:

client_controller.client_db.ProcessContentUpdatePackage(content_update_package)

This single call triggers the entire bulk processing pipeline, ensuring that all updates for a given service are handled in one atomic transaction.

Bulk Processing in ProcessContentUpdatePackage

The core batching logic resides in ClientDBContentUpdates.ProcessContentUpdatePackage within hydrus/client/db/ClientDBContentUpdates.py. This method extracts tag changes, validates them, and executes bulk SQL operations.

Initial Validation and Normalization

The processor first filters out invalid services. As shown in lines 58–78 of ClientDBContentUpdates.py, the method iterates over the package and validates service keys:

for (service_key, content_updates) in content_update_package.IterateContentUpdates():
    try:
        service_id = self.modules_services.GetServiceId(service_key)
    except HydrusExceptions.DataMissing:
        continue   # skip unknown services

    valid_content_update_package.AddContentUpdates(service_key, content_updates)
    service = self.modules_services.GetService(service_id)
    service_type = service.GetServiceType()

Only updates belonging to recognized services proceed to the processing stage.

Tag-Service Branch and Aggregation

For real tag services (HC.REAL_TAG_SERVICES), the method examines each ContentUpdate with data_type == HC.CONTENT_TYPE_MAPPINGS. Between lines 78–165, it extracts the tag and hashes, normalizes the tag via HydrusTags.CleanTag, and converts identifiers to internal IDs (tag_id and hash_ids):

elif data_type == HC.CONTENT_TYPE_MAPPINGS:
    (tag, hashes) = row
    # clean and validate tag

    tag = HydrusTags.CleanTag(tag)
    tag_id = self.modules_tags.GetTagId(tag)
    hash_ids = self.modules_hashes_local_cache.GetHashIds(hashes)

    # decide which aggregate list to push to

    if action == HC.CONTENT_UPDATE_ADD:
        ultimate_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_DELETE:
        ultimate_deleted_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_PEND:
        ultimate_pending_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_RESCIND_PEND:
        ultimate_pending_rescinded_mappings_ids.append((tag_id, hash_ids))
    elif action == HC.CONTENT_UPDATE_PETITION:
        reason_id = self.modules_texts.GetTextId(content_update.GetReason())
        ultimate_petitioned_mappings_ids.append((tag_id, hash_ids, reason_id))
    elif action == HC.CONTENT_UPDATE_RESCIND_PETITION:
        ultimate_petitioned_rescinded_mappings_ids.append((tag_id, hash_ids))

Source: ClientDBContentUpdates.py, lines 784–861

The method maintains six "ultimate" lists—ultimate_mappings_ids, ultimate_deleted_mappings_ids, ultimate_pending_mappings_ids, ultimate_pending_rescinded_mappings_ids, ultimate_petitioned_mappings_ids, and ultimate_petitioned_rescinded_mappings_ids—to accumulate all changes of each type before hitting the database.

Database Writes and Cache Coordination

After aggregation, Hydrus executes bulk SQL operations using executemany for maximum throughput. The calls in lines 1060–1100 dispatch to ClientDBMappingsStorage.py:

# Add mappings

self.modules_mappings_storage.AddMappings(service_id, ultimate_mappings_ids)

# Delete mappings

self.modules_mappings_storage.DeleteMappings(service_id, ultimate_deleted_mappings_ids)

# Pend/Rescind pend

self.modules_mappings_storage.PendMappings(service_id, ultimate_pending_mappings_ids)
self.modules_mappings_storage.RescindPending(service_id, ultimate_pending_rescinded_mappings_ids)

# Petition/Rescind petition

self.modules_mappings_storage.PetitionMappings(service_id, ultimate_petitioned_mappings_ids)
self.modules_mappings_storage.RescindPetition(service_id, ultimate_petitioned_rescinded_mappings_ids)

Simultaneously, the same aggregated lists update the mapping caches (ClientDBMappingsCacheSpecificStorage and ClientDBMappingsCacheSpecificDisplay) to ensure tag display managers refresh efficiently. If changes affect visible tags, the method collects affected hash_ids into self._regen_tags_managers_hash_ids (lines 108–114), triggering a single UI refresh after the transaction commits rather than updating individual files incrementally.

Practical Example: Batching Tag Additions

The following snippet demonstrates how a plugin or script would batch-add the tag dog to three files using the content update system:

from hydrus.client import ClientController
from hydrus.client.metadata import ClientContentUpdates
from hydrus.core import HydrusConstants as HC

# 1️⃣ Gather service key for the default tag service

client = ClientController()
tag_service_key = client.services_manager.GetDefaultTagService().GetServiceKey()

# 2️⃣ Build a ContentUpdate for the mapping

hashes = ['a1b2c3', 'd4e5f6', '112233']          # human-readable hashes

content_update = ClientContentUpdates.ContentUpdate(
    data_type=HC.CONTENT_TYPE_MAPPINGS,
    action=HC.CONTENT_UPDATE_ADD,
    row=('dog', hashes)                         # (tag, [hashes])

)

# 3️⃣ Package it per service

package = ClientContentUpdates.ContentUpdatePackage()
package.AddContentUpdate(tag_service_key, content_update)

# 4️⃣ Hand it to the DB – this is where the whole batching logic runs

client.client_db.ProcessContentUpdatePackage(package)

When executed, this code resolves the tag to a tag_id, converts the three hashes to hash_ids, appends the tuple to ultimate_mappings_ids, and performs a single bulk INSERT into the tag-mapping table. The specific mapping caches update immediately, and the UI regenerates tag displays for only the affected files.

Summary

Hydrus delivers high-performance tag modifications through a carefully designed batching architecture:

• Conversion – Raw client actions become structured ContentUpdate objects containing typed data, actions, and metadata.
• Aggregation – ContentUpdatePackage groups updates by service key, preparing them for atomic processing.
• Bulk Execution – ProcessContentUpdatePackage validates services, aggregates changes into "ultimate" lists by action type, and issues multi-row SQL statements via executemany.
• Cache Synchronization – The same batched data updates mapping caches and triggers targeted UI refreshes without per-file overhead.

This pipeline ensures that importing or modifying thousands of tags generates minimal database churn while maintaining consistent, real-time cache states across the application.

Frequently Asked Questions

What is a ContentUpdatePackage in Hydrus?

A ContentUpdatePackage is a container class defined in hydrus/client/metadata/ClientContentUpdates.py that aggregates multiple ContentUpdate objects by their target service key. It acts as the primary data structure for batching tag operations before they enter the database processing pipeline, ensuring that all changes targeting the same tag service can be committed in a single atomic transaction.

How does Hydrus handle different tag actions like pending or petitioning?

Hydrus processes pend and petition actions by routing them to dedicated "ultimate" lists—ultimate_pending_mappings_ids and ultimate_petitioned_mappings_ids—inside ProcessContentUpdatePackage. According to the logic in hydrus/client/db/ClientDBContentUpdates.py, pending actions insert rows into pending tables, while petitions store the request along with a reason_id. Rescind actions remove entries from these pending or petition tables rather than the main mapping storage.

Why does Hydrus use "ultimate" lists during tag processing?

The "ultimate" lists (such as ultimate_mappings_ids and ultimate_deleted_mappings_ids) serve as aggregation buffers that collect all tag changes of a specific type before database insertion. This design allows Hydrus to execute a single INSERT OR REPLACE statement for hundreds or thousands of tag mappings using SQLite's executemany, dramatically reducing transaction overhead compared to processing individual rows one at a time.

Which source files control the tag content update pipeline?

The pipeline spans four critical files: hydrus/client/metadata/ClientContentUpdates.py defines the ContentUpdate and ContentUpdatePackage classes; hydrus/client/db/ClientDBContentUpdates.py contains the ProcessContentUpdatePackage batch processor; hydrus/client/db/ClientDBMappingsStorage.py handles the low-level SQL inserts and deletes; and hydrus/core/HydrusTags.py provides tag normalization via CleanTag before database storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}