Understanding catalog.yaml Manifest Structure: Scope, Snapshots, and Publishing in Knowledge Catalog

The catalog.yaml manifest file uses three primary sections—scope, snapshot, and publishing—to declaratively define target resources, metadata capture criteria, and output destinations for Knowledge Catalog CLI operations.

The GoogleCloudPlatform/knowledge-catalog repository provides a CLI-driven workflow for managing Data Catalog metadata through declarative configuration files. Understanding the catalog.yaml manifest structure is essential for defining which entry groups to target, what aspects to capture, and where to publish the resulting snapshots.

Scope Configuration

The scope section serves as the entry point for every operation, identifying the specific Entry Group or resource hierarchy that subsequent commands will target. This field follows a dot-separated naming convention that maps directly to Google Cloud resource paths parsed by the CLI tooling.

Resource Naming Patterns

The scope string uses predefined patterns that translate into API resource names:

• entryGroup.<project>.<location>.<group> — Targets a specific entry group
• bq-dataset.<project>.<dataset> — Targets a BigQuery dataset
• kb.<project>.<location>.<knowledge_base> — Targets a Knowledge Base

For example, the scope entryGroup.test.us.g1 translates to the fully qualified API resource name projects/test/locations/us/entryGroups/g1 when processed by the mdcode binary.

Minimal Scope Example

A manifest containing only scope configuration appears in toolbox/mdcode/tests/scenarios/manifest_simple.yaml:

scope: entryGroup.test.us.g1

Execute this minimal configuration using:

mdcode apply -f toolbox/mdcode/tests/scenarios/manifest_simple.yaml

The CLI parses this scope and lists the entries within the specified group.

Snapshot Section Definition

The optional snapshot section declares read-only captures of catalog state, enumerating specific entry types and aspect types to include in the export. This section transforms the manifest from a simple scope definition into a metadata extraction configuration.

Entry and Aspect Types

The snapshot configuration requires two lists:

• entries — Entry types to capture (e.g., BigQuery datasets, Cloud Storage buckets)
• aspects — Aspect types representing metadata facets (e.g., overview, schema)

Both lists use the canonical dataplex-type notation: <domain>.<location>.<type-name>.

Complete Snapshot Configuration

Referencing toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml, a typical snapshot block captures BigQuery dataset entries with their overview metadata:

scope: entryGroup.test.us.g1
snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
  aspects:
    - dataplex-types.global.overview

Generate this snapshot using:

mdcode snapshot -f toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml

During execution, the CLI queries the specified entries and aspects, producing a portable representation of the current catalog state.

Publishing Configuration Options

While the current test scenarios do not provide concrete examples, the optional publishing section controls output destinations and post-processing for generated artifacts. This section typically configures:

• destination — Cloud Storage URI (gs://), BigQuery table reference, or file path
• format — Serialization format such as JSON, YAML, or Avro
• hooks — Optional post-publish commands or callbacks

Hypothetical Publishing Block

A complete manifest including publishing configuration might structure the section as follows:

scope: entryGroup.test.us.g1
snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
  aspects:
    - dataplex-types.global.overview
publishing:
  destination: gs://my-bucket/catalog-snapshots/
  format: JSON
  hooks:
    - gcloud functions deploy my-function --trigger-http

When processed with mdcode apply, the CLI would upload the snapshot to the specified Cloud Storage bucket in JSON format and invoke any defined hooks.

Practical Implementation Examples

Building complete workflows requires combining all three sections or using minimal configurations for specific tasks.

Basic Scope-Only Operation

For listing entries without capturing metadata:

scope: bq-dataset.my-project.my-dataset

Run with:

mdcode apply -f manifest.yaml

Full Metadata Export

For extracting specific entry types with multiple aspects:

scope: entryGroup.test.us.g1
snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
    - dataplex-types.global.cloud-storage-bucket
  aspects:
    - dataplex-types.global.overview
    - dataplex-types.global.schema
publishing:
  destination: gs://my-bucket/snapshots/
  format: JSON

This configuration captures both BigQuery datasets and Cloud Storage buckets along with their overview and schema metadata, publishing results to Cloud Storage.

Summary

• The catalog.yaml manifest structure centers on three optional sections: scope (required), snapshot, and publishing.
• Scope uses dot-separated patterns like entryGroup.<project>.<location>.<group> to define target resources.
• Snapshot enumerates entry types and aspect types using dataplex-type notation to determine what metadata to capture.
• Publishing configures output destinations, serialization formats, and post-processing hooks for generated artifacts.
• Reference implementations exist in toolbox/mdcode/tests/scenarios/manifest_simple.yaml and toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml.

Frequently Asked Questions

What is the required format for the scope field in catalog.yaml?

The scope field follows a dot-separated pattern that mirrors Google Cloud resource hierarchies. Valid formats include entryGroup.<project>.<location>.<group> for entry groups, bq-dataset.<project>.<dataset> for BigQuery datasets, and kb.<project>.<location>.<knowledge_base> for Knowledge Bases. The CLI translates these shortcuts into full API resource names like projects/test/locations/us/entryGroups/g1.

How do I specify which metadata fields to capture in a snapshot?

Define the aspects list within the snapshot section using canonical dataplex-type notation. For example, include dataplex-types.global.overview to capture overview metadata or dataplex-types.global.schema to capture schema definitions. Each aspect type corresponds to a specific metadata facet supported by the Data Catalog API.

Can I publish snapshots directly to BigQuery instead of Cloud Storage?

While the current test manifests in the repository focus on Cloud Storage destinations, the publishing section supports any valid destination URI including BigQuery table references. The format field determines the serialization, and you should specify the destination using the appropriate BigQuery table path format recognized by the Knowledge Catalog CLI.

Where can I find authoritative examples of catalog.yaml manifests?

The repository provides reference implementations in the test scenarios directory. Examine toolbox/mdcode/tests/scenarios/manifest_simple.yaml for scope-only configurations and toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml for complete examples including both scope and snapshot sections with entries and aspects defined.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}