# Understanding catalog.yaml Manifest Structure: Scope, Snapshots, and Publishing in Knowledge Catalog

> Learn the catalog.yaml manifest structure for Knowledge Catalog CLI. Understand scope, snapshots, and publishing to define resources and metadata capture.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-14

---

**The catalog.yaml manifest file uses three primary sections—`scope`, `snapshot`, and `publishing`—to declaratively define target resources, metadata capture criteria, and output destinations for Knowledge Catalog CLI operations.**

The GoogleCloudPlatform/knowledge-catalog repository provides a CLI-driven workflow for managing Data Catalog metadata through declarative configuration files. Understanding the **catalog.yaml manifest structure** is essential for defining which entry groups to target, what aspects to capture, and where to publish the resulting snapshots.

## Scope Configuration

The **scope** section serves as the entry point for every operation, identifying the specific Entry Group or resource hierarchy that subsequent commands will target. This field follows a dot-separated naming convention that maps directly to Google Cloud resource paths parsed by the CLI tooling.

### Resource Naming Patterns

The scope string uses predefined patterns that translate into API resource names:

- `entryGroup.<project>.<location>.<group>` — Targets a specific entry group
- `bq-dataset.<project>.<dataset>` — Targets a BigQuery dataset
- `kb.<project>.<location>.<knowledge_base>` — Targets a Knowledge Base

For example, the scope `entryGroup.test.us.g1` translates to the fully qualified API resource name `projects/test/locations/us/entryGroups/g1` when processed by the `mdcode` binary.

### Minimal Scope Example

A manifest containing only scope configuration appears in [`toolbox/mdcode/tests/scenarios/manifest_simple.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/manifest_simple.yaml):

```yaml
scope: entryGroup.test.us.g1

```

Execute this minimal configuration using:

```bash
mdcode apply -f toolbox/mdcode/tests/scenarios/manifest_simple.yaml

```

The CLI parses this scope and lists the entries within the specified group.

## Snapshot Section Definition

The optional **snapshot** section declares read-only captures of catalog state, enumerating specific entry types and aspect types to include in the export. This section transforms the manifest from a simple scope definition into a metadata extraction configuration.

### Entry and Aspect Types

The snapshot configuration requires two lists:

- **entries** — Entry types to capture (e.g., BigQuery datasets, Cloud Storage buckets)
- **aspects** — Aspect types representing metadata facets (e.g., overview, schema)

Both lists use the canonical **dataplex-type** notation: `<domain>.<location>.<type-name>`.

### Complete Snapshot Configuration

Referencing [`toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml), a typical snapshot block captures BigQuery dataset entries with their overview metadata:

```yaml
scope: entryGroup.test.us.g1
snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
  aspects:
    - dataplex-types.global.overview

```

Generate this snapshot using:

```bash
mdcode snapshot -f toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml

```

During execution, the CLI queries the specified entries and aspects, producing a portable representation of the current catalog state.

## Publishing Configuration Options

While the current test scenarios do not provide concrete examples, the optional **publishing** section controls output destinations and post-processing for generated artifacts. This section typically configures:

- **destination** — Cloud Storage URI (`gs://`), BigQuery table reference, or file path
- **format** — Serialization format such as JSON, YAML, or Avro
- **hooks** — Optional post-publish commands or callbacks

### Hypothetical Publishing Block

A complete manifest including publishing configuration might structure the section as follows:

```yaml
scope: entryGroup.test.us.g1
snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
  aspects:
    - dataplex-types.global.overview
publishing:
  destination: gs://my-bucket/catalog-snapshots/
  format: JSON
  hooks:
    - gcloud functions deploy my-function --trigger-http

```

When processed with `mdcode apply`, the CLI would upload the snapshot to the specified Cloud Storage bucket in JSON format and invoke any defined hooks.

## Practical Implementation Examples

Building complete workflows requires combining all three sections or using minimal configurations for specific tasks.

### Basic Scope-Only Operation

For listing entries without capturing metadata:

```yaml
scope: bq-dataset.my-project.my-dataset

```

Run with:

```bash
mdcode apply -f manifest.yaml

```

### Full Metadata Export

For extracting specific entry types with multiple aspects:

```yaml
scope: entryGroup.test.us.g1
snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
    - dataplex-types.global.cloud-storage-bucket
  aspects:
    - dataplex-types.global.overview
    - dataplex-types.global.schema
publishing:
  destination: gs://my-bucket/snapshots/
  format: JSON

```

This configuration captures both BigQuery datasets and Cloud Storage buckets along with their overview and schema metadata, publishing results to Cloud Storage.

## Summary

- The **catalog.yaml manifest structure** centers on three optional sections: `scope` (required), `snapshot`, and `publishing`.
- **Scope** uses dot-separated patterns like `entryGroup.<project>.<location>.<group>` to define target resources.
- **Snapshot** enumerates entry types and aspect types using dataplex-type notation to determine what metadata to capture.
- **Publishing** configures output destinations, serialization formats, and post-processing hooks for generated artifacts.
- Reference implementations exist in [`toolbox/mdcode/tests/scenarios/manifest_simple.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/manifest_simple.yaml) and [`toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml).

## Frequently Asked Questions

### What is the required format for the scope field in catalog.yaml?

The scope field follows a dot-separated pattern that mirrors Google Cloud resource hierarchies. Valid formats include `entryGroup.<project>.<location>.<group>` for entry groups, `bq-dataset.<project>.<dataset>` for BigQuery datasets, and `kb.<project>.<location>.<knowledge_base>` for Knowledge Bases. The CLI translates these shortcuts into full API resource names like `projects/test/locations/us/entryGroups/g1`.

### How do I specify which metadata fields to capture in a snapshot?

Define the **aspects** list within the snapshot section using canonical dataplex-type notation. For example, include `dataplex-types.global.overview` to capture overview metadata or `dataplex-types.global.schema` to capture schema definitions. Each aspect type corresponds to a specific metadata facet supported by the Data Catalog API.

### Can I publish snapshots directly to BigQuery instead of Cloud Storage?

While the current test manifests in the repository focus on Cloud Storage destinations, the publishing section supports any valid destination URI including BigQuery table references. The `format` field determines the serialization, and you should specify the destination using the appropriate BigQuery table path format recognized by the Knowledge Catalog CLI.

### Where can I find authoritative examples of catalog.yaml manifests?

The repository provides reference implementations in the test scenarios directory. Examine [`toolbox/mdcode/tests/scenarios/manifest_simple.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/manifest_simple.yaml) for scope-only configurations and [`toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/manifest_snapshot.yaml) for complete examples including both scope and snapshot sections with entries and aspects defined.