How to Pull the Latest Catalog Entries Using kcmd: A Complete Guide

Run kcmd pull in your initialized directory to synchronize your local metadata-as-code snapshot with the remote Dataplex catalog, downloading any new or updated entries.

The kcmd command-line interface is part of the Knowledge Catalog toolbox in the GoogleCloudPlatform/knowledge-catalog repository. It enables teams to manage metadata-as-code workflows by treating Dataplex catalog entries as version-controlled files. Running kcmd pull performs a read-only, idempotent operation that updates your local repository to match the authoritative state stored in Dataplex.

Prerequisites: Initialize Your Local Snapshot

Before pulling entries, you must initialize a local catalog snapshot. This creates the catalog.yaml configuration file and directory structure.

kcmd init --bigquery-dataset my-project.my_dataset

This command sets up the metadata-as-code foundation required for subsequent sync operations.

How kcmd pull Works Under the Hood

When you execute kcmd pull, the CLI performs a six-step synchronization process defined in toolbox/mdcode/src/tool/commands.ts (lines 62-78).

1. Create an API Context

The process begins by resolving Google Cloud credentials and project configuration:

const apiCtx = kcmd.gcp.context.ApiContext.default();

This method, called at line 62-64 in commands.ts, establishes the authenticated session required for Dataplex API calls.

2. Load the Local Snapshot

The CLI reads your local metadata files into memory:

const snapshot = await kcmd.CatalogSnapshot.fromPath('.', apiCtx);

As implemented at line 63, fromPath() scans the current directory for YAML/JSON catalog entries and builds an in-memory representation of your local state.

3. Instantiate the Dataplex Client

A thin wrapper around the Dataplex REST API is created:

const client = new dataplex.CatalogClient(apiCtx);

This occurs at line 65 in commands.ts, using the API context for authentication.

4. Create the Sync Helper

The system wires together the remote client and local snapshot:

const sync = new kcmd.CatalogSync(client, snapshot);

Line 66 initializes the CatalogSync class, which implements the differential algorithm comparing remote and local states.

5. Execute the Pull Operation

The actual synchronization happens here:

const result = await sync.pull();

At lines 68-70, sync.pull() contacts Dataplex, compares remote entries with your local snapshot, and writes new or changed entries to the filesystem. The method returns a Result object indicating success or failure.

6. Report the Outcome

Finally, the CLI reports results (lines 71-78):

• Success: Prints "Successfully updated local snapshot."
• Failure: Logs error details and returns a non-zero exit code

Pulling Catalog Entries from the Command Line

For most users, the CLI workflow involves two simple commands:

# Initialize once per project

kcmd init --bigquery-dataset my-project.my_dataset

# Pull latest entries

kcmd pull

Expected output:

Pulling catalog entries...
Successfully updated local snapshot.

Programmatic Usage with TypeScript

You can also trigger pulls programmatically using the kcmd library. This is useful for automation scripts or custom tooling:

import * as kcmd from 'kcmd';
import * as dataplex from 'kcmd/libts/gcp/dataplex';
import * as ctx from 'kcmd/libts/gcp/context';

async function pullCatalog() {
  // Step 1: Create API context
  const apiCtx = ctx.ApiContext.default();
  
  // Step 2: Load local snapshot
  const snapshot = await kcmd.CatalogSnapshot.fromPath('.', apiCtx);
  
  // Step 3: Create Dataplex client
  const client = new dataplex.CatalogClient(apiCtx);
  
  // Step 4: Initialize sync helper
  const sync = new kcmd.CatalogSync(client, snapshot);
  
  // Step 5: Execute pull
  const result = await sync.pull();
  
  if (result.success) {
    console.log('Local snapshot refreshed');
  } else {
    console.error('Pull failed:', result.details);
  }
}

This implementation mirrors the CLI logic found in src/tool/commands.ts but allows custom error handling and integration into larger applications.

Key Source Files

The kcmd pull command relies on several core modules in the GoogleCloudPlatform/knowledge-catalog repository:

• toolbox/mdcode/src/tool/commands.ts: Contains the CLI entry points for init, pull, and push operations, including the main pull implementation at lines 62-78
• toolbox/mdcode/src/libts/snapshot.ts: Handles reading and writing local catalog snapshots via CatalogSnapshot
• toolbox/mdcode/src/libts/gcp/dataplex.ts: Implements the CatalogClient wrapper around Dataplex REST APIs
• toolbox/mdcode/src/libts/sync.ts: Defines the CatalogSync class and its pull() method for differential synchronization

Summary

• kcmd pull synchronizes your local metadata-as-code repository with the remote Dataplex catalog
• The operation is read-only and idempotent, making it safe to run repeatedly without side effects
• The process involves six steps: authentication, loading local state, creating the API client, initializing the sync helper, executing the differential pull, and reporting results
• Source code resides primarily in toolbox/mdcode/src/tool/commands.ts and toolbox/mdcode/src/libts/sync.ts
• You can use kcmd pull via CLI or programmatically through the TypeScript SDK

Frequently Asked Questions

What happens if I run kcmd pull without initializing first?

The command will fail because CatalogSnapshot.fromPath() expects a valid catalog.yaml configuration file and directory structure created by kcmd init. Always initialize your local snapshot before attempting to pull remote entries.

Is kcmd pull safe to run in a CI/CD pipeline?

Yes. The pull operation is read-only and idempotent, meaning it only downloads changes and never modifies remote state. However, ensure your CI environment has valid Google Cloud credentials configured so ApiContext.default() can authenticate with Dataplex.

How does kcmd handle merge conflicts between local and remote entries?

The CatalogSync.pull() method implemented in src/libts/sync.ts performs a differential comparison favoring the remote state. If entries exist remotely that differ from your local files, the remote versions overwrite local copies. To preserve local changes, commit them to version control before pulling, or use kcmd push to upload your modifications first.

Can I pull entries from a specific directory rather than the current one?

Yes. Both the CLI and programmatic API support specifying paths. In the TypeScript SDK, pass your target directory to CatalogSnapshot.fromPath('/path/to/dir', apiCtx) instead of '.'. The CLI will respect the directory context from which you run the command.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}