How kcmd Manages Metadata Using Source Code Artifacts: A Complete Guide to the Knowledge Catalog CLI

kcmd treats catalog metadata as first-class source-code artifacts by modeling entries, aspects, and manifests as plain TypeScript objects that are persisted to the filesystem using well-defined layouts, then synchronized with Google Cloud Data Catalog.

The kcmd CLI from the GoogleCloudPlatform/knowledge-catalog repository enables teams to manage metadata using source code artifacts through a declarative, version-controlled workflow. Unlike traditional database-centric approaches, this tool materializes BigQuery datasets, Dataplex entry groups, and Knowledge Bases as ordinary YAML and Markdown files. This architecture allows developers to apply standard IDE tooling, linting, and Git workflows to metadata management while maintaining bidirectional synchronization with Google Cloud Data Catalog.

Understanding the kcmd Architecture

The kcmd architecture consists of a small set of TypeScript libraries that model catalog entities as code representations, orchestrated by a CLI that dispatches to these core components.

CatalogManifest: Declaring the Source

The CatalogManifest class describes the source of a catalog—whether a BigQuery dataset, Dataplex entry group, or Knowledge Base—and optionally limits which entries and aspects are captured. According to the source code in src/libts/manifest.ts, this manifest is persisted as a YAML file named catalog.yaml that serves as the source-code artifact declaring what metadata will be materialized locally.

Entry and Aspect Models

At the heart of the system are the Entry and Aspect interfaces defined in src/libts/metadata.ts. These simple TypeScript interfaces model the shape of catalog entries and their attached aspects, representing the code-level abstraction of the metadata. When loaded into memory, these plain objects enable deterministic checksums and validation before persistence.

CatalogSnapshot: The Persistence Layer

The CatalogSnapshot class in src/libts/snapshot.ts provides the high-level API for reading, writing, listing, and validating entries on disk. Rather than handling file formats directly, it delegates to a layout strategy—such as YAML side-cars or Markdown front-matter—allowing the same metadata to be stored in different textual formats depending on team preferences.

CatalogSync: The Synchronization Engine

The CatalogSync class in src/libts/sync.ts implements the pull and push operations that compare local snapshots with the remote Google Cloud Data Catalog service. This component handles checksum computation, dry-run capabilities, and conflict resolution, ensuring that only explicitly declared artifacts (respecting the manifest's snapshot and publishing whitelists) are propagated to the cloud service.

The Metadata-as-Code Workflow

The kcmd tool implements a five-stage workflow that treats metadata as version-controlled source code.

1. Manifest Creation with kcmd init

When initializing a new catalog, the CatalogManifest.initWithBigQuery method constructs a CatalogSource object based on user-provided parameters like --bigquery-dataset PROJECT.DATASET. This method, implemented in src/libts/manifest.ts, saves the configuration to catalog.yaml, creating the source-code artifact that declares the metadata scope.

2. Snapshot Generation with kcmd pull

The CatalogSnapshot.fromPath method reads the manifest from the repository root, resolves the source connection, and walks the chosen layout strategy to produce in-memory Entry objects. The snapshot maintains a map of entries and aspects, enabling deterministic checksums for change detection.

3. Local Modification

Users edit the generated files directly—modifying my_table.yaml or my_table.md using standard editors. Because these are ordinary source files, teams can apply linting, code review, and version control practices that are impossible with API-only metadata management.

4. Synchronization with kcmd push

The CatalogSync class compares the local snapshot with the remote Data Catalog, detecting changes via checksums and performing create, update, or delete operations via the gcp.CatalogClient. The sync respects the manifest's whitelists, ensuring only declared artifacts are propagated.

5. MCP Server Exposure

When invoked as kcmd mcp, the binary runs a lightweight HTTP server defined in src/tool/mcp.ts that exposes sync primitives as tools for external agents. This allows automated systems to treat the catalog as "metadata-as-code" and invoke the same operations programmatically.

CLI Implementation and MCP Server

The command-line interface wires user-facing commands to the library classes through a structured entry point and command implementations.

Command Routing in main.ts

The CLI entry point in src/tool/main.ts uses the cac argument parser to dispatch subcommands (init, pull, push, status, mcp) to their respective handlers. This binary (dist/kcmd) serves as the unified interface for both human operators and automated systems.

Command Logic in commands.ts

The concrete CLI logic resides in src/tool/commands.ts, where implementations create a CatalogManifest, load a CatalogSnapshot, and invoke CatalogSync methods. This separation ensures that the CLI layer remains thin, delegating all business logic to the reusable library classes.

MCP Server for External Agents

The MCP server implementation in src/tool/mcp.ts exposes the same CatalogSnapshot and CatalogSync logic via HTTP RPC endpoints such as list-entries, lookup-entry, pull, and push. This architecture allows external agents to interact with metadata artifacts without shelling out to the CLI.

Working with Source Code Artifacts Programmatically

The following examples demonstrate how to interact with kcmd's source-code artifact system using the TypeScript API.

Create a manifest for a BigQuery dataset:

import * as kcmd from 'kcmd';
import { gcp } from 'kcmd';

const ctx = gcp.ApiContext.default();
const manifest = await kcmd.CatalogManifest.initWithBigQuery(
  'my-project.my_dataset', ctx);
manifest.save('catalog.yaml');

Load a local snapshot from the filesystem:

const snapshot = await kcmd.CatalogSnapshot.fromPath(
  '.',  // repo root containing catalog.yaml + entry files
  ctx);
console.log(snapshot.entries.map(e => e.name));

Push local changes back to Cloud Data Catalog:

import { CatalogSync } from 'kcmd/libts/sync';

const catalog = new kcmd.gcp.CatalogClient(ctx);
const sync = new kcmd.CatalogSync(catalog, snapshot);
await sync.push();  // Handles checksums, dry-run, etc.

Run the MCP server for external tool integration:

import * as http from 'node:http';
import { startMcp } from 'kcmd/tool/mcp';

http.createServer(startMcp({ ctx })).listen(8080);

Summary

• kcmd manages metadata through source-code artifacts by treating BigQuery datasets and other catalog sources as declarative YAML configurations and entry files.
• The CatalogManifest in src/libts/manifest.ts defines the source and scope, persisted as catalog.yaml.
• CatalogSnapshot from src/libts/snapshot.ts materializes metadata into the filesystem using pluggable layouts (YAML side-cars or Markdown front-matter).
• CatalogSync in src/libts/sync.ts handles bidirectional synchronization with checksum validation and whitelist filtering.
• The CLI in src/tool/main.ts and command implementations in src/tool/commands.ts provide the user interface, while src/tool/mcp.ts exposes these capabilities as HTTP tools for external agents.

Frequently Asked Questions

How does kcmd store metadata on the local filesystem?

kcmd stores metadata using layout strategies delegated by CatalogSnapshot. Depending on configuration, entries are saved as YAML side-car files or Markdown documents with front-matter, allowing the metadata to exist as ordinary source files alongside application code.

What is the purpose of the catalog.yaml file?

The catalog.yaml file is the CatalogManifest that declares the source of the catalog (such as a BigQuery dataset or Dataplex entry group) and defines which entries and aspects should be captured. It serves as the source-code artifact that configures the synchronization scope.

How does kcmd detect changes between local files and the remote Data Catalog?

CatalogSync computes checksums of the local Entry and Aspect objects managed by CatalogSnapshot, then compares these against the remote state via the gcp.CatalogClient. This allows precise detection of create, update, and delete operations before applying them.

Can external tools or agents interact with kcmd programmatically?

Yes. When started with kcmd mcp, the tool runs an HTTP server defined in src/tool/mcp.ts that exposes the snapshot and sync operations as RPC endpoints. External agents can invoke list-entries, lookup-entry, pull, and push operations directly without using the CLI interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}