What Is Metadata as Code (kcmd) in Google Cloud Knowledge Catalog?

Metadata as Code (kcmd) treats your data catalog metadata as version-controlled source files, enabling CI/CD workflows, code reviews, and programmatic updates through YAML and Markdown representations.

In the GoogleCloudPlatform/knowledge-catalog repository, metadata as code (kcmd) represents a paradigm shift in how organizations manage their data assets. Instead of interacting with metadata solely through graphical interfaces or API calls, this approach stores catalog information as plain-text files within standard version control systems. This methodology bridges the gap between data governance and software engineering practices, allowing teams to apply the same rigor of code review and automated testing to their metadata management.

The Core Concept: Metadata as Version-Controlled Code

The fundamental principle behind kcmd is treating metadata with the same lifecycle management as application source code. Metadata lives in plain-text files—specifically YAML for structured data and Markdown side-cars for unstructured text—stored directly in your repository. Because these files exist within version control, they can be edited, diffed, reviewed, and deployed through existing CI/CD pipelines that development teams already use daily. This alignment with standard developer workflows eliminates the friction typically associated with catalog maintenance and governance tasks.

Alignment with the Knowledge Catalog Service Model

The file layout in kcmd directly mirrors the Catalog data model, supporting Entry, EntryLink, and Aspect types. A catalog.yaml manifest file sits at the root of your repository to describe the sync scope and processing directives for the kcmd toolchain. The actual metadata artifacts reside in a catalog/ directory, maintaining a clear separation between configuration and data.

Repository Layout Standards

According to the design documentation in toolbox/mdcode/docs/concept.md, kcmd supports two distinct file layouts to accommodate different organizational preferences.

Standard Layout

The standard layout creates one YAML file per Entry, supplemented by optional Markdown side-cars for aspects like Overview. This separation of concerns keeps structured schema definitions distinct from narrative documentation.

path/to/root/
├── catalog.yaml
└── catalog/
    └── <dir>/<entry-id>.yaml          # structured metadata

    └── <entry-id>.<aspect>.md         # markdown side‑car

Documents Layout

The documents layout consolidates each Entry into a single Markdown file where YAML front-matter contains the structured fields and the body contains the Overview aspect. This approach appeals to teams that prefer document-centric workflows.

path/to/root/
├── catalog.yaml
└── catalog/
    └── <dir>/<entry-id>.md            # front-matter + body

Bi-Directional Sync Operations

The toolbox/mdcode/src/tool/commands.ts file implements the core CLI commands that enable bi-directional synchronization between your repository and the Knowledge Catalog service. The kcmd toolkit can download a snapshot of existing catalog metadata into your local file hierarchy and publish local changes back to the service, preserving fidelity for both first-party and third-party metadata.

To pull the latest catalog state into your repository:

kcmd sync --project=my-gcp-project --scope=entryGroup

After modifying files in your catalog/ directory, publish changes back to the service:

kcmd publish --project=my-gcp-project --scope=entryGroup

Agent-Ready Architecture and Extensibility

Because metadata exists as code, autonomous agents can read the YAML and Markdown representations, reason about the content, and issue programmatic updates. This enables enrichment agents that automatically add glossary terms, lineage information, or data quality annotations. Human oversight remains available through standard pull-request workflows, creating a hybrid governance model that combines automation with human judgment.

The architecture remains deliberately layout-agnostic, as documented in toolbox/mdcode/docs/design.md, allowing future extensions such as embedding kcmd artifacts within DBT or Dataform projects while maintaining consistent representation across use cases.

Configuration Examples

A catalog.yaml manifest defines the synchronization parameters for your project:

# catalog.yaml – manifest that tells kcmd how to treat the directory

scope: entryGroup
project: my-gcp-project
output_dir: catalog

Individual Entry files contain the structured metadata definitions:

# catalog/dataset/my_table.yaml – structured entry representation

name: projects/my-gcp-project/locations/us/entryGroups/my_dataset/entries/my_table
type: TABLE
aspect:
  schema:
    fields:
      - name: user_id
        type: STRING
      - name: event_ts
        type: TIMESTAMP

Unstructured aspect documentation uses Markdown side-cars:

# catalog/dataset/my_table.overview.md – side‑car for the Overview aspect

# Overview

This table captures user interaction events used for analytics and model training.

Getting Started with kcmd

The toolbox/mdcode/README.md provides the primary entry point for understanding the Metadata-as-Code toolbox, while toolbox/mdcode/demo/README.md offers a quick-start guide demonstrating typical kcmd usage flows. For architectural details regarding the sync workflow, refer to toolbox/mdcode/docs/design.md.

Summary

• Metadata as Code (kcmd) transforms Google Cloud Knowledge Catalog metadata into version-controlled YAML and Markdown files stored in the GoogleCloudPlatform/knowledge-catalog toolbox.
• The repository supports two layout patterns: standard layout (separate YAML and Markdown files) and documents layout (consolidated Markdown with front-matter).
• Bi-directional sync via the kcmd CLI (implemented in toolbox/mdcode/src/tool/commands.ts) enables downloading snapshots from the service and publishing local changes back.
• The agent-ready architecture allows both automated enrichment and human review through standard Git workflows and CI/CD pipelines.

Frequently Asked Questions

What file formats does kcmd use for metadata storage?

kcmd uses YAML for structured metadata fields and Markdown for unstructured text aspects. The standard layout keeps these in separate files, while the documents layout combines them into single Markdown files with YAML front-matter.

How does kcmd handle synchronization with the Knowledge Catalog service?

The kcmd CLI provides bi-directional sync capabilities. The sync command downloads the current catalog state into your repository, and the publish command pushes your local file changes back to the service, as implemented in toolbox/mdcode/src/tool/commands.ts.

Can metadata changes be reviewed through pull requests?

Yes. Because metadata exists as plain-text files in your repository, all changes follow standard Git workflows. Teams can create branches, open pull requests, conduct code reviews, and merge updates through CI/CD pipelines before the kcmd publish command deploys them to the catalog.

What is the difference between standard layout and documents layout?

The standard layout stores each catalog Entry as a separate YAML file with optional Markdown side-cars for documentation aspects. The documents layout consolidates each Entry into a single Markdown file where the YAML front-matter contains structured fields and the document body contains the Overview aspect.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}