How Does Bi-Directional Sync Work in kcmd? A Technical Deep Dive

Bi-directional sync in kcmd reconciles local JSON snapshots with remote Google Cloud catalogs through a three-phase process: loading states, computing checksum-based differences, and applying changes via the CatalogSync class.

The kcmd command-line tool from the GoogleCloudPlatform/knowledge-catalog repository enables developers to manage Knowledge Catalog metadata as code. Understanding how bi-directional synchronization works requires examining the CatalogSync class and its reconciliation engine that bridges local file systems with BigQuery, Dataplex, and Knowledge Base APIs.

The Three-Phase Synchronization Architecture

The CatalogSync class implements bi-directional sync through a deterministic reconciliation workflow. According to the source code in src/libts/sync.ts, the process treats both local snapshots and remote catalogs as hierarchical trees of Entry objects, then computes and applies differences.

Phase 1: Loading Local and Remote States

The sync process begins by materializing both data sources into memory.

Local State: The CatalogSnapshot.fromPath('.') method reads on-disk metadata from JSON files in the current directory. Implemented in src/libts/snapshot.ts, this creates a hierarchy of Entry objects representing the local catalog state.

Remote State: The CatalogManifest class loads the remote catalog via GCP client libraries. Depending on the target service, it uses initWithEntryGroup(), initWithKnowledgeBase(), or initWithBigQuery() to fetch entries and construct an in-memory representation. This logic resides in src/libts/manifest.ts.

Phase 2: Computing Checksum-Based Differences

Once both states are loaded, CatalogSync.diff() compares the two trees. The algorithm:

1. Builds lookup maps keyed by entry IDs for both local and remote entries
2. Invokes CatalogSync._compareEntries() to evaluate each pair
3. Generates CatalogDiff objects categorizing changes into three sets:
   • Added: Entries present only in the remote catalog (for pull) or local snapshot (for push)
   • Deleted: Entries missing from the opposite side
   • Modified: Entries with identical IDs but differing content

The comparison relies on SHA-256 checksums calculated from each entry's JSON representation. By comparing entry.checksum values rather than performing deep object equality checks, the system efficiently detects any metadata drift.

Phase 3: Applying Changes with Pull and Push

The final phase applies the computed diff in the direction specified by the command:

Pull Operations (kcmd pull): The CatalogSync.applyPull() method writes new or changed entries to the local snapshot root as JSON files and deletes local files corresponding to remote deletions. This synchronizes the remote state to the local filesystem.

Push Operations (kcmd push): The CatalogSync.applyPush() method invokes GCP APIs to mutate the remote catalog. It calls catalogEntries.create, catalogEntries.update, and catalogEntries.delete RPCs to make the remote catalog match the local snapshot.

Both commands support --dry-run to preview changes without applying them, and --force to override checksum conflicts.

Conflict Resolution and Safety Mechanisms

When executing kcmd push, the system performs strict conflict detection. If a remote entry's checksum differs from the local version, CatalogSync flags this as a conflict and aborts the operation with a clear error message unless --force is specified.

To inspect pending changes without applying them, use kcmd status. This command runs the same diff logic as the sync operations but outputs a concise table of added, modified, and deleted items, providing visibility into drift before reconciliation.

Command-Line Workflow

The typical bi-directional sync workflow follows this pattern:

# Initialize a local snapshot from a BigQuery dataset

kcmd init --bigquery-dataset my-project.my_dataset

# Preview changes before applying

kcmd status          # Display diff between local and remote

kcmd pull --dry-run  # Preview remote-to-local updates

kcmd push --dry-run  # Preview local-to-remote updates

# Apply changes

kcmd pull            # Update local files from remote

kcmd push            # Push local edits to Google Cloud

This workflow enables GitOps-style catalog management where metadata changes undergo code review before deployment to production environments.

Programmatic Sync with TypeScript

Beyond the CLI, kcmd exposes the synchronization engine as a TypeScript library. The following example demonstrates programmatic bi-directional sync:

import * as kcmd from 'kcmd';

// Initialize API context
const ctx = kcmd.gcp.ApiContext.default();

// Load local snapshot from current directory
const snapshot = await kcmd.CatalogSnapshot.fromPath('.', ctx);

// Load remote catalog (BigQuery example)
const remote = await kcmd.CatalogManifest.initWithBigQuery(
  ['my-project.my_dataset'], ctx);

// Instantiate sync manager
const sync = new kcmd.CatalogSync(remote, snapshot);

// Compute and display differences
const diff = sync.diff();
console.log('Added:', diff.added.length);
console.log('Modified:', diff.modified.length);
console.log('Deleted:', diff.deleted.length);

// Apply changes
await sync.applyPush();   // Push local to remote
// await sync.applyPull(); // Or pull remote to local

The CatalogSync class handles all checksum validation and API communication, allowing developers to integrate catalog synchronization into custom pipelines or CI/CD systems.

Summary

• Bi-directional sync in kcmd uses the CatalogSync class to reconcile local JSON snapshots with remote Google Cloud catalogs.
• The process follows three phases: loading states via CatalogSnapshot and CatalogManifest, computing SHA-256 checksum differences, and applying changes through applyPull() or applyPush().
• Key implementation files include src/libts/sync.ts for the core logic, src/libts/snapshot.ts for local state, and src/libts/manifest.ts for remote API integration.
• Safety features include --dry-run for previewing changes, kcmd status for diff inspection, and checksum-based conflict detection with --force override support.

Frequently Asked Questions

What is the difference between kcmd pull and kcmd push?

kcmd pull executes CatalogSync.applyPull(), which updates local JSON files to match the remote catalog state, effectively importing changes from Google Cloud. kcmd push executes CatalogSync.applyPush(), which updates the remote catalog via GCP APIs to match the local snapshot, exporting your local edits to BigQuery, Dataplex, or Knowledge Base.

How does kcmd detect conflicts during synchronization?

The system calculates a SHA-256 checksum for each entry's JSON representation and stores it in entry.checksum. When pushing, if the remote entry's checksum differs from the local version (indicating someone else modified the remote catalog), kcmd detects a conflict and aborts the operation unless you provide the --force flag to overwrite the remote version.

Can I preview changes before applying them in kcmd?

Yes. The kcmd status command runs the diff logic without applying changes, displaying a table of added, modified, and deleted entries. Additionally, both kcmd pull and kcmd push accept the --dry-run flag, which executes the full synchronization logic but stops before calling write operations, allowing you to verify the proposed changes safely.

How do I resolve checksum conflicts when pushing changes?

When kcmd push encounters a checksum mismatch, it aborts with an error indicating which entry has diverged. You can resolve this by either pulling the remote changes first with kcmd pull to merge updates, or by forcing the local version to overwrite the remote entry using kcmd push --force. The --force flag bypasses the checksum validation and applies your local state regardless of remote modifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}