How to Use the kcmd TypeScript Library to Manage Metadata in Google Cloud

The kcmd TypeScript library provides a programmatic SDK for managing Google Cloud Knowledge Catalog metadata through CatalogSnapshot, CatalogManifest, and CatalogSync classes.

The kcmd library, housed in the GoogleCloudPlatform/knowledge-catalog repository under the toolbox/mdcode package, enables developers to inspect, create, update, and synchronize catalog metadata using TypeScript. This library abstracts the underlying Data Catalog and BigQuery APIs into a cohesive workflow that handles entries, entry groups, and datasets through an in-memory snapshot pattern.

Core Architecture Components

The library implements a three-stage pipeline for metadata management: snapshot (reading source state), manifest (defining target state), and sync (applying differences).

CatalogSnapshot

The CatalogSnapshot class reads metadata from a local directory or remote source and builds an in-memory view of all catalog objects. According to the implementation in toolbox/mdcode/src/libts/snapshot.ts, the static method fromPath parses markdown files and extracts metadata into a structured map.

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const snapshot = await kcmd.CatalogSnapshot.fromPath('./catalog', ctx);

CatalogManifest

The CatalogManifest class, defined in toolbox/mdcode/src/libts/manifest.ts, represents the destination for your metadata. It provides factory methods to initialize manifests from different Google Cloud resources:

• initWithEntryGroup(entryGroup, ctx) – Targets a specific Data Catalog entry group
• initWithKnowledgeBase(kb, ctx) – Connects to a Knowledge Base resource
• initWithBigQuery(datasets, ctx) – Targets BigQuery datasets

CatalogSync

The CatalogSync class in toolbox/mdcode/src/libts/sync.ts orchestrates the reconciliation between a source snapshot and a target manifest. The constructor takes (target, source) arguments, and the apply() method executes the necessary API calls to create, update, or delete entries.

Entry Types and Metadata

Individual metadata objects are represented as kcmd.Entry types, defined in toolbox/mdcode/src/libts/metadata.ts. These entries encapsulate tables, files, datasets, and custom metadata fields that flow between snapshots and manifests.

Authenticating with Google Cloud

All kcmd operations require a GCP API context that encapsulates authentication and project configuration. The ApiContext.default() method in toolbox/mdcode/src/libts/gcp/context.ts uses Application Default Credentials (ADC) to resolve service account or user credentials.

const ctx = kcmd.gcp.ApiContext.default();

Ensure the authenticated principal has appropriate IAM roles, such as roles/datacatalog.admin or roles/bigquery.dataEditor, depending on your target manifest type.

Practical Implementation Examples

Loading a Local Catalog Snapshot

This pattern initializes a snapshot from a local markdown directory, parsing frontmatter and content into Entry objects:

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const snapshot = await kcmd.CatalogSnapshot.fromPath('./source-catalog', ctx);

Synchronizing with a Knowledge Base

To mirror a local catalog to an existing Knowledge Base entry group:

const ctx = kcmd.gcp.ApiContext.default();
const source = await kcmd.CatalogSnapshot.fromPath('./source-catalog', ctx);

const kbId = 'projects/my-project/locations/us-central1/entryGroups/my-kb';
const manifest = await kcmd.CatalogManifest.initWithKnowledgeBase(kbId, ctx);

const sync = new kcmd.CatalogSync(manifest, source);
await sync.apply();

The CatalogSync instance compares the source snapshot against the remote Knowledge Base state and applies only the necessary changes.

Publishing Metadata to BigQuery

To materialize catalog entries as BigQuery tables or views:

const ctx = kcmd.gcp.ApiContext.default();
const snapshot = await kcmd.CatalogSnapshot.fromPath('./my-markdown-catalog', ctx);

const datasets = [
  'projects/my-project/datasets/dataset_a',
  'projects/my-project/datasets/dataset_b',
];
const manifest = await kcmd.CatalogManifest.initWithBigQuery(datasets, ctx);

const sync = new kcmd.CatalogSync(manifest, snapshot);
await sync.apply();

Summary

• Import: Use import * as kcmd from 'kcmd' to access the SDK.
• Authenticate: Create a context with kcmd.gcp.ApiContext.default() before any operations.
• Snapshot: Load source metadata using CatalogSnapshot.fromPath() from toolbox/mdcode/src/libts/snapshot.ts.
• Manifest: Choose your target via CatalogManifest factory methods in toolbox/mdcode/src/libts/manifest.ts.
• Sync: Reconcile differences using CatalogSync from toolbox/mdcode/src/libts/sync.ts, which handles creates, updates, and deletes idempotently.
• CLI Alternative: The kcmd CLI in toolbox/mdcode/src/tool/commands.ts wraps these same SDK functions for command-line usage.

Frequently Asked Questions

How does kcmd handle authentication in CI/CD environments?

The kcmd library relies on Application Default Credentials via kcmd.gcp.ApiContext.default() in toolbox/mdcode/src/libts/gcp/context.ts. In CI/CD pipelines, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to a service account key file, or use workload identity federation to provide credentials without storing keys.

What is the difference between CatalogSnapshot and CatalogManifest?

CatalogSnapshot represents a read-only view of metadata loaded from a source (typically local files), while CatalogManifest represents a writable target destination (such as a Knowledge Base or BigQuery dataset). The CatalogSync class bridges these two states to determine what changes are required.

Can I use kcmd to sync between two Knowledge Bases?

Yes. You can load a source Knowledge Base into a CatalogSnapshot using fromPath (if exported locally) or by initializing a snapshot from a remote source, then create a target manifest using CatalogManifest.initWithKnowledgeBase() with the destination Knowledge Base ID. Running CatalogSync.apply() will migrate entries between them.

Is the kcmd sync operation idempotent?

Yes. The CatalogSync implementation in toolbox/mdcode/src/libts/sync.ts calculates a diff between the source snapshot and target manifest before making any API calls. Running the sync multiple times with the same inputs will only apply changes if the underlying metadata has actually changed, making it safe to run in scheduled jobs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}