How CatalogManifest Is Used in the kcmd CLI: Initialization, Sync, and Source Control

CatalogManifest is the core TypeScript abstraction in the kcmd CLI that represents Knowledge Catalog entries and exposes factory methods to initialize manifests from BigQuery, Dataplex, or Knowledge Base sources.

The kcmd command-line tool in the GoogleCloudPlatform/knowledge-catalog repository manages Knowledge Catalog metadata through a declarative manifest system. At the heart of this system lies CatalogManifest, a class defined in toolbox/mdcode/src/libts/manifest.ts that bridges external data sources with local catalog snapshots. This guide examines how CatalogManifest functions within the kcmd architecture, covering its initialization patterns, synchronization workflows, and integration with the broader catalog management pipeline.

Initializing CatalogManifest from Data Sources

The CatalogManifest class provides three static factory methods to construct manifests from different Google Cloud data sources. These methods handle API authentication, data transformation, and entry creation.

From BigQuery Datasets

Use CatalogManifest.initWithBigQuery() to ingest metadata from one or more BigQuery datasets. This method queries the BigQuery API, converts each dataset and table into a kcmd.Entry, and assembles them into a manifest.

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const datasets = ['my-project.my_dataset1', 'my-project.my_dataset2'];

const manifest = await kcmd.CatalogManifest.initWithBigQuery(datasets, ctx);
await manifest.saveToPath('./.catalog');

From Dataplex Entry Groups

For existing Dataplex metadata, use CatalogManifest.initWithEntryGroup(). This retrieves the entry group's entries via the Dataplex API and builds a manifest representation.

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const entryGroup = 'my-project.us-central1.myEntryGroup';

const manifest = await kcmd.CatalogManifest.initWithEntryGroup(entryGroup, ctx);
await manifest.saveToPath('./.catalog');

From Knowledge Base Definitions

To load pre-defined knowledge-base specifications from YAML or JSON configuration files, use CatalogManifest.initWithKnowledgeBase().

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const manifest = await kcmd.CatalogManifest.initWithKnowledgeBase('my-kb', ctx);
await manifest.saveToPath('./.catalog');

Core Workflow in the kcmd CLI

The kcmd CLI commands orchestrate CatalogManifest operations through a consistent workflow implemented in toolbox/mdcode/src/tool/commands.ts and orchestrated from toolbox/mdcode/src/tool/main.ts.

The init Command

The kcmd init command executes the appropriate factory method based on user-supplied flags (--bigquery-dataset, --entry-group, or --kb). The resulting CatalogManifest is written to a local snapshot directory (.catalog/) using the saveToPath() method.

Synchronization with pull and push

The kcmd pull and kcmd push commands rely on the manifest to drive state changes. The workflow involves:

1. Loading the manifest from the local .catalog/ directory
2. Creating a snapshot using CatalogSnapshot.fromPath('.', ctx) to represent the current state
3. Computing differences via the CatalogSync class
4. Applying updates to synchronize the remote catalog

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const manifest = await kcmd.CatalogManifest.initWithKnowledgeBase('my-kb', ctx);
const snapshot = await kcmd.CatalogSnapshot.fromPath('.', ctx);

const sync = new kcmd.CatalogSync(manifest, snapshot);
await sync.apply(); // Pushes local changes to the remote catalog

Integration with CatalogSnapshot and CatalogSync

CatalogSnapshot provides a point-in-time representation of the catalog state persisted to disk. The CatalogSync class accepts both the desired state (CatalogManifest) and the current state (CatalogSnapshot) to calculate deltas and execute synchronization operations.

Key Implementation Files

The CatalogManifest functionality is distributed across the following source files:

• toolbox/mdcode/src/libts/manifest.ts – Defines the CatalogManifest class and its static factory methods (initWithBigQuery, initWithEntryGroup, initWithKnowledgeBase).
• toolbox/mdcode/src/tool/commands.ts – Implements CLI command handlers that invoke manifest factories and synchronization logic.
• toolbox/mdcode/src/tool/main.ts – Entry point for the unified binary that routes sub-commands to the appropriate handlers.

Summary

• CatalogManifest serves as the source of truth for Knowledge Catalog metadata in kcmd, representing the desired state of entries.
• Three factory methods support initialization from BigQuery datasets, Dataplex entry groups, and Knowledge Base IDs, each handling the respective API interactions.
• The kcmd init command persists manifests to the .catalog/ directory using saveToPath().
• CatalogSync uses the manifest alongside CatalogSnapshot to compute and apply state changes during pull and push operations.
• All implementations reside in the toolbox/mdcode/src/ directory, with the core class definition in libts/manifest.ts.

Frequently Asked Questions

What is the difference between CatalogManifest and CatalogSnapshot?

CatalogManifest represents the desired state of the catalog derived from external sources like BigQuery or Dataplex, while CatalogSnapshot represents the current state persisted to the local .catalog/ directory. The CatalogSync class compares these two structures to determine what changes to apply to the remote Knowledge Catalog.

How do I initialize a manifest from multiple BigQuery datasets?

Pass an array of fully-qualified dataset strings to CatalogManifest.initWithBigQuery(), such as ['project.dataset1', 'project.dataset2'], along with a valid API context obtained from kcmd.gcp.ApiContext.default().

Where does kcmd store local manifest files?

By default, kcmd init writes manifest data to a hidden .catalog/ directory in the current working path using the saveToPath('./.catalog') method. This directory serves as the local state store for subsequent pull, push, and status operations.

Can I use CatalogManifest programmatically outside the kcmd CLI?

Yes. Import the kcmd library as a TypeScript module, create an API context via kcmd.gcp.ApiContext.default(), and instantiate CatalogManifest using any of the static factory methods. This allows you to build custom synchronization workflows or integrate catalog management into existing data pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}