How to Configure the catalog.yaml Manifest File in Google Cloud Knowledge Catalog

The catalog.yaml file acts as the central manifest that defines the synchronization scope, publishing directives, and resource aliases for Google Cloud Knowledge Catalog metadata snapshots.

The catalog.yaml manifest file lives in the root of your snapshot directory and controls how the toolbox/mdcode tooling synchronizes local metadata with the remote Google Cloud Knowledge Catalog (Dataplex) service. This configuration file determines which entries are pulled, which aspects can be pushed back, and how resource identifiers are aliased throughout your project.

Core Configuration Sections

The manifest uses YAML structure with four primary properties that govern metadata synchronization behavior.

Scope Definition

The scope property is required and serves as a unified identifier in the format <type>.<name>. It tells the tooling which remote resource the local snapshot represents.

Supported scope types include:

• bq-dataset – for BigQuery dataset metadata
• entryGroup – for Dataplex entry groups
• kb – for Knowledge Base documents

scope: entryGroup.test.us.g1

In the source code at toolbox/mdcode/src/libts/catalogManifest.ts, this scope string is parsed to determine the target resource type and identity during synchronization operations.

Publishing Directives

The optional publishing section lists entry-type and aspect combinations that should be written back to the catalog. Only entries explicitly listed here are mutable during push operations; everything else remains read-only.

publishing:
  - entry: table
    aspect: schema
  - entry: table
    aspect: profile

When configured as above, only the schema and profile aspects of tables will be published, while other aspects remain protected from modification.

Resource Aliases

The optional aliases section maps long resource identifiers to short, friendly names. Once defined, these aliases must be used consistently across all metadata files in the snapshot.

aliases:
  myDataset: projects/myproject/locations/us/datasets/dataset1

According to the specification in toolbox/mdcode/docs/spec.md, the validation logic ensures that every file reference matches the alias defined in the manifest before push operations complete.

Layout Inference

The optional layout field determines on-disk organization but is typically auto-detected based on the scope type:

• Standard Layout – Used for bq-dataset and entryGroup scopes (YAML per entry + sidecar Markdown)
• Documents Layout – Used for kb scopes (single Markdown with frontmatter)

You normally omit this field; the tooling infers the correct layout from the scope type.

How the Manifest Is Used

The catalog.yaml file drives three primary operations in the synchronization lifecycle.

Initialization

The kcmd init command creates a catalog.yaml with a scope matching your target resource. This establishes the connection between your local directory and the remote catalog instance.

Pull Operations

During pull, the tool reads the manifest to determine which entries and aspects to download from the catalog service. The scope dictates the API endpoints queried, while the implied layout determines the filesystem structure created.

Push Operations

Before pushing metadata back to the service, the tooling validates that:

1. All file references use defined aliases (if configured)
2. Only aspects listed in the publishing section are included in the write payload
3. Resource identifiers match the scope constraints

This validation logic is implemented in toolbox/mdcode/src/libts/catalogManifest.ts, which handles checksum computation and conflict detection.

Configuration Examples

Minimal Manifest

For a basic entry group synchronization, only the scope is required:

scope: entryGroup.test.us.g1

This example matches the test fixture found at toolbox/mdcode/tests/scenarios/manifest_simple.yaml, which verifies that the tool correctly parses single-line scope declarations.

Production Manifest with Publishing Rules

Restrict write operations to specific aspects using the publishing array:

scope: bq-dataset.myproject.us.dataset1
publishing:
  - entry: table
    aspect: schema
  - entry: table
    aspect: profile

Alias-Enabled Configuration

Simplify resource references while maintaining strict validation:

scope: bq-dataset.myproject.us.dataset1
aliases:
  myDataset: projects/myproject/locations/us/datasets/dataset1
publishing:
  - entry: table
    aspect: schema

Programmatic Generation

Create manifests dynamically using TypeScript, as demonstrated in toolbox/mdcode/demo/kb/setup.ts:

import { join } from 'path';
import { writeFileSync } from 'fs';
import YAML from 'yaml';

const manifest = {
  scope: 'kb.myproject.us.mykb',
  publishing: [{ entry: 'article', aspect: 'overview' }],
};

writeFileSync(
  join(process.cwd(), 'catalog.yaml'),
  YAML.stringify(manifest)
);

Summary

• The catalog.yaml file is the mandatory root configuration that defines your snapshot's relationship to Google Cloud Knowledge Catalog resources.
• The scope field is required and uses the format <type>.<name> to identify remote resources such as BigQuery datasets or entry groups.
• The publishing section controls mutability by explicitly listing which entry/aspect combinations can be pushed back to the service.
• The aliases section enables friendly names for long resource paths but requires consistent usage across all metadata files.
• Implementation files including toolbox/mdcode/src/libts/catalogManifest.ts and specifications in toolbox/mdcode/docs/spec.md handle parsing, validation, and synchronization logic.

Frequently Asked Questions

What happens if I omit the publishing section in catalog.yaml?

If you omit the publishing section, all metadata remains read-only. The tooling will pull remote data to your local snapshot, but push operations will not write any changes back to the catalog service, effectively creating a one-way synchronization.

Can I use multiple aliases for the same resource path?

No, the aliases section requires unique mapping. Each alias name must map to a single resource identifier, and the validation logic enforces consistent usage. If you need to reference the same resource differently in separate contexts, you must use separate snapshot directories with individual manifest files.

How does the tool validate the scope format?

The validation logic in toolbox/mdcode/src/libts/catalogManifest.ts parses the scope string to ensure it follows the <type>.<name> pattern with supported types (bq-dataset, entryGroup, or kb). Invalid scope formats will cause initialization or synchronization to fail with a validation error before any API calls are made.

Where can I find the formal specification for catalog.yaml?

The formal specification resides in toolbox/mdcode/docs/spec.md within the repository. This document defines the complete schema, validation rules, and behavioral contracts for the manifest file, while toolbox/mdcode/docs/design.md provides the architectural rationale behind the CatalogManifest implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}