How to Initialize BigQuery Dataset Metadata Snapshots Using kcmd init with Multiple Entry Types

You can initialize snapshots for multiple BigQuery datasets by passing the --bigquery-dataset flag multiple times or as a comma-separated list, which creates a catalog.yaml manifest that records the scope for the Knowledge Catalog snapshot.

The kcmd CLI in the GoogleCloudPlatform/knowledge-catalog repository implements a Metadata as Code workflow that mirrors remote metadata sources into version-controlled local files. When working with BigQuery, the init sub-command bootstraps a local snapshot directory containing YAML for structured metadata and Markdown sidecars for unstructured content, enabling you to capture one or many datasets in a single operation.

Understanding the kcmd init Command

The kcmd init command serves as the entry point for creating CatalogManifest files that define your snapshot scope. Located in toolbox/mdcode/src/tool/commands.ts, this command accepts mutually exclusive flags for different source types: --entry-group for Dataplex EntryGroups, --kb for Knowledge Bases, or --bigquery-dataset for BigQuery datasets.

When you specify BigQuery datasets, the CLI invokes CatalogManifest.initWithBigQuery() from toolbox/mdcode/src/libts/manifest.ts. This method constructs a manifest that records dataset identifiers and sync configuration, then serializes it to catalog.yaml in your current working directory.

Handling Multiple Entry Types

For multiple entry types of the same source kind—specifically multiple BigQuery datasets—the CLI accepts dataset identifiers in two formats. Both approaches result in a comma-joined string passed to the manifest initialization logic.

Using Comma-Separated Values

Pass multiple datasets as a single comma-delimited string:

kcmd init --bigquery-dataset my-project.ds1,my-project.ds2,my-project.ds3

Using Repeated Flags

Alternatively, repeat the flag for each dataset:

kcmd init \
  --bigquery-dataset my-project.ds1 \
  --bigquery-dataset my-project.ds2 \
  --bigquery-dataset my-project.ds3

Both methods trigger the same underlying logic in commands.ts that handles arrays:

const datasets = Array.isArray(options.bigqueryDataset)
  ? options.bigqueryDataset.join(',')
  : options.bigqueryDataset;
manifest = await kcmd.CatalogManifest.initWithBigQuery(datasets, ctx);

The Snapshot Layout and Immediate Sync

After creating the manifest, kcmd init writes the Standard Layout to disk. This layout organizes metadata into YAML files for structured aspects (schemas, labels) and sidecar Markdown files for unstructured content like overviews.

To populate these files immediately rather than just creating the manifest, append the --pull flag:

kcmd init --bigquery-dataset my-project.ds1,my-project.ds2 --pull

This combination creates the catalog.yaml manifest and immediately executes the pull sub-command, downloading current metadata from BigQuery into the snapshot directory.

Complete Implementation Details

The initialization workflow in toolbox/mdcode/src/tool/commands.ts (lines 25-51) performs these specific steps:

1. Source Type Detection – Validates that exactly one source flag is provided among --entry-group, --kb, or --bigquery-dataset.
2. Manifest Construction – Calls the appropriate CatalogManifest factory method based on the source type.
3. Persistence – Saves the manifest as catalog.yaml in the current directory.
4. Optional Pull – If --pull is specified, chains to the pull() function to hydrate the snapshot.

The CatalogManifest class persists sync configuration and dataset scope, enabling subsequent kcmd pull and kcmd push operations to synchronize changes between your local files and the Knowledge Catalog service.

Summary

• Multiple datasets are supported via comma-separated strings or repeated --bigquery-dataset flags when initializing BigQuery snapshots.
• The source of truth for initialization logic resides in toolbox/mdcode/src/tool/commands.ts, while manifest definitions live in toolbox/mdcode/src/libts/manifest.ts.
• CatalogManifest factory methods like initWithBigQuery() convert CLI arguments into persistent configuration stored in catalog.yaml.
• The Standard Layout produces YAML/Markdown files that enable version-controlled metadata editing.
• Use the --pull flag to immediately populate the snapshot after manifest creation.

Frequently Asked Questions

Can I initialize snapshots for multiple BigQuery datasets in one command?

Yes. You can pass multiple dataset identifiers either as a comma-separated list (--bigquery-dataset project.ds1,project.ds2) or by repeating the flag multiple times. Both approaches generate a single catalog.yaml that encompasses all specified datasets.

Where is the snapshot configuration stored after running kcmd init?

The command creates a catalog.yaml file in your current working directory. This file contains the CatalogManifest that records the source type, dataset identifiers, and sync configuration required for subsequent pull and push operations.

Can I mix BigQuery datasets with Dataplex EntryGroups in a single init command?

No. The CLI enforces mutually exclusive source types through conditional logic in commands.ts. You must choose exactly one source type per initialization: either --bigquery-dataset for BigQuery, --entry-group for Dataplex, or --kb for Knowledge Base.

Does kcmd init immediately download the metadata files?

By default, kcmd init only creates the manifest. To immediately populate the snapshot directory with actual metadata files, include the --pull flag. This triggers the pull workflow immediately after manifest creation, downloading YAML and Markdown representations of your datasets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}