How to Configure Bi-Directional Metadata Sync Between a Local Workspace and the Knowledge Catalog Service

The Knowledge Catalog repository implements Metadata-as-Code (MaC) through the CatalogSync class, enabling bi-directional synchronization between your local filesystem and the remote service using kcmd pull and kcmd push commands.

Bi-directional metadata sync allows teams to manage Google Cloud Data Catalog resources using version-controlled YAML files while keeping the local workspace and remote service in perfect alignment. The GoogleCloudPlatform/knowledge-catalog repository provides the toolbox/mdcode package, which orchestrates this two-way flow through a specialized sync engine and command-line interface.

Understanding the Metadata-as-Code Architecture

The synchronization logic resides in toolbox/mdcode/src/libts/sync.ts, where the CatalogSync class manages the bidirectional flow between the CatalogClient (GCP service client) and the CatalogSnapshot (local filesystem representation).

Pull Operations

The CatalogSync.pull method iterates over entries defined in the snapshot's manifest, fetching each resource from the Knowledge Catalog service via CatalogClient.lookupEntry, and persists the results locally using the private _storeEntry method. According to the source code in toolbox/mdcode/src/libts/sync.ts (lines 31-55), this builds a local directory hierarchy that mirrors your remote BigQuery datasets, Pub/Sub topics, and other cataloged assets as YAML files.

Push Operations

Conversely, CatalogSync.push walks the local snapshot using listEntries and _fetchEntry, compares the state against the remote service, and applies changes with granular update masks. As implemented in toolbox/mdcode/src/libts/sync.ts (lines 63-118), the system creates missing entries or updates existing ones using specific field masks (aspects, entry_source, parent_entry), reporting any conflicts back to the caller for resolution.

Prerequisites and Authentication

Both the CLI and TypeScript library require Application Default Credentials to communicate with the Knowledge Catalog API.

Run the following command to authenticate:

gcloud auth application-default login

The CLI README at toolbox/mdcode/README.md (lines 68-70) documents this requirement as mandatory for all sync operations. Ensure your account has appropriate Data Catalog permissions (e.g., datacatalog.entries.get and datacatalog.entries.create) for the target project.

Implementing the Bi-Directional Sync Workflow

The kcmd CLI tool in toolbox/mdcode/src/tool/commands.ts (lines 61-69) exposes intuitive commands that wrap the underlying CatalogSync methods. Follow this workflow to establish continuous synchronization:

1. Initialize the Local Snapshot

Create a metadata manifest for your target resource. For example, to track a BigQuery dataset:

kcmd init --scope bq-dataset.my-project.sales --entries sales

This generates a catalog.yaml manifest and an empty catalog/ directory structure in your workspace.

2. Pull Remote State

Fetch the current metadata from the Knowledge Catalog service to establish your baseline:

kcmd pull

This command invokes CatalogSync.pull, writing each entry as a separate YAML file under catalog/ with full aspect specifications.

3. Modify Local Metadata

Edit the generated YAML files or sidecar markdown documentation. Changes to aspects, descriptions, or business metadata remain purely local until you push.

4. Review Changes

Preview the differences between your local workspace and the remote service:

kcmd status

The status command computes a diff without transmitting modifications, allowing you to verify updates before application.

5. Push to Remote

Synchronize your local changes back to the Knowledge Catalog service:

kcmd push

The CLI calculates the minimal update mask, creates new entries if missing, and modifies existing resources to match your local definitions.

Programmatic Synchronization with TypeScript

For CI/CD pipelines or custom tooling, import the library directly to execute sync operations programmatically.

Pulling metadata via the library:

import * as kcmd from 'kcmd';

// Initialize manifest for BigQuery dataset
const manifest = new kcmd.CatalogManifest({
  scope: 'bq-dataset.my-project.sales',
  snapshot: { entries: ['sales'] },
});
await manifest.save('/tmp/kc-demo');

// Load local snapshot
const snapshot = kcmd.CatalogSnapshot.fromPath('/tmp/kc-demo');

// Execute bi-directional pull
const pullResult = await snapshot.pull();
if (!pullResult.success) {
  console.error('Pull failed:', pullResult.error);
}

Pushing with conflict override:

const pushResult = await snapshot.push({ force: true });
if (!pushResult.success) {
  console.error('Push error:', pushResult.error);
}

The force option bypasses certain conflict checks, useful when your local workspace must become the authoritative source of truth.

Summary

• Metadata-as-Code (MaC) enables treating catalog definitions as version-controlled artifacts in toolbox/mdcode/src/libts/sync.ts.
• Bi-directional sync consists of pull (remote-to-local) and push (local-to-remote) operations orchestrated by the CatalogSync class.
• CLI workflow follows init → pull → edit → status → push, with --dry-run flags available for safe previews.
• Authentication requires gcloud auth application-default login for both CLI and library usage.
• Conflict handling occurs during CatalogSync.push, utilizing update masks to apply granular changes without overwriting entire entries.

Frequently Asked Questions

What is Metadata-as-Code (MaC) in the Knowledge Catalog context?

Metadata-as-Code is the practice of defining Data Catalog entries, aspects, and metadata as YAML files stored in version control. According to the toolbox/mdcode specification in docs/spec.md, this approach enables GitOps workflows, code reviews for metadata changes, and reproducible deployments across environments while maintaining synchronization with the live Knowledge Catalog service.

How does the sync handle conflicts between local and remote changes?

The CatalogSync.push method implements optimistic concurrency control. When pushing modifications from toolbox/mdcode/src/libts/sync.ts (lines 63-118), the system compares local checksums against remote state. If changes are detected server-side that conflict with local edits, the operation reports specific conflicts to the caller without overwriting data. Use kcmd status to identify these discrepancies before pushing, or pass { force: true } in the library to override remote changes with local definitions.

Can I preview changes before pushing them to the Knowledge Catalog service?

Yes. The kcmd CLI supports dry-run capabilities via flags (--dry-run) that simulate the push operation without modifying the service. Additionally, the kcmd status command displays the computed diff between your local catalog/ directory and the remote entries, showing exactly which aspects and fields will be created, updated, or deleted during the next push operation.

Which authentication methods does the bi-directional sync support?

The system exclusively uses Application Default Credentials (ADC) as documented in toolbox/mdcode/README.md (lines 68-70). Configure ADC by running gcloud auth application-default login or by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to a service account key file. Both the kcmd CLI and the TypeScript CatalogSync class automatically retrieve credentials from the ADC chain when initializing the CatalogClient.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}