Implementing CI/CD Pipelines for Metadata with Version Control: A Metadata-as-Code Guide

The GoogleCloudPlatform/knowledge-catalog repository enables implementing CI/CD pipelines for metadata with version control by treating YAML and Markdown metadata files as source code artifacts, using the kcmd CLI to validate, snapshot test, and idempotently push changes to Google Cloud Knowledge Catalog.

The Knowledge Catalog repository adopts a Metadata-as-Code philosophy that positions metadata definitions as first-class source artifacts. By storing catalog definitions in Git alongside application code, data teams can apply standard DevOps practices—automated testing, peer review, and continuous deployment—to manage data assets at scale. This approach centers on the kcmd library and CLI tools found in the toolbox/mdcode directory, allowing you to version, validate, and deploy metadata with the same rigor as application code.

Metadata-as-Code Repository Structure

The architecture mirrors the resource hierarchy of underlying data assets to ensure both human readability and automation compatibility.

Catalog Manifest and Entry Files

At the repository root, the catalog.yaml manifest defines the scope of the catalog—such as a BigQuery dataset or user-managed EntryGroup—and enumerates the entry types, aspects, and links to be captured. Beneath the catalog/ directory, each entry is represented by a YAML file (<entry-id>.yaml) alongside optional Markdown side-car files (<entry-id>.overview.md) for unstructured aspects like overviews.

This structure is documented in toolbox/mdcode/docs/concept.md and exemplified in toolbox/mdcode/README.md, ensuring that changes to the scope or included aspects are tracked atomically alongside the entry files.

CI/CD Pipeline Components

A robust pipeline for metadata requires three core stages to ensure safety and consistency before deployment to production.

Static Validation

The lint step verifies YAML syntax integrity, validates required fields against the schema, and ensures that Markdown side-car files contain properly formatted front-matter. This prevents malformed metadata from entering the catalog and failing at runtime.

Snapshot Testing and Drift Detection

Using the kcmd library, the pipeline executes kcmd pull --dry-run to capture the current state from Knowledge Catalog and compare it against the repository version. If unexpected drift is detected—indicating out-of-band changes in the catalog—the CI build fails, triggering human review before any write operations occur.

The expected pull behavior is illustrated in toolbox/mdcode/tests/scenarios/pull_basic.yaml.

Idempotent Push Deployment

Upon successful validation, the pipeline invokes kcmd push to publish metadata changes. This operation is idempotent and respects the --dry-run flag for safety testing before actual deployment. The push command publishes new entries and updates existing ones atomically, ensuring the catalog state converges to the repository definition.

The push operation schema is demonstrated in toolbox/mdcode/tests/scenarios/push_new_entry.yaml.

Implementing the CI/CD Workflow

A production-ready pipeline orchestrates validation and deployment through environment-specific stages using standard CI/CD platforms.

GitHub Actions Configuration

The following workflow file (.github/workflows/metadata-ci.yml) demonstrates the complete lifecycle from validation to deployment:

name: Metadata CI/CD
on:
  push:
    paths:
      - 'toolbox/mdcode/**'
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      - name: Install kcmd
        run: npm install kcmd
      - name: Lint YAML/MD
        run: |
          npx yaml-lint toolbox/mdcode/**/**/*.yaml
          markdownlint toolbox/mdcode/**/**/*.md
      - name: Pull latest catalog snapshot
        run: |
          npx kcmd pull --dry-run
      - name: Run unit tests
        run: npm test
  deploy:
    needs: validate
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install kcmd
        run: npm install kcmd
      - name: Push metadata
        env:
          GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
        run: |
          npx kcmd push

Environment-Specific Deployment

The deploy job triggers only on the main branch after successful validation, using GOOGLE_APPLICATION_CREDENTIALS stored as repository secrets to authenticate push operations against the Google Cloud API.

Conflict Detection and Resolution

When concurrent edits occur in Knowledge Catalog outside the repository workflow, kcmd pull surfaces conflicts during the snapshot phase. The CI step automatically aborts the push operation, prompting developers to reconcile changes via a pull request. This mechanism prevents accidental overwrites of metadata modified outside the version-controlled pipeline, ensuring that the repository remains the single source of truth.

Programmatic Metadata Management

Beyond CLI usage, the kcmd library supports TypeScript-based automation for complex transformations and bulk updates within CI jobs:

import * as kcmd from 'kcmd';

// Load an existing snapshot
const snapshot = kcmd.CatalogSnapshot.fromPath('/path/to/root');

// Modify an entry programmatically
const entry = snapshot.getEntry('my_dataset.my_table');
entry.resource.description = 'Updated description by CI pipeline';
await snapshot.push(); // Publishes the change

This programmatic approach enables sophisticated CI logic, such as automatically updating metadata based on schema changes detected in upstream data sources.

Summary

• Store metadata as code in a Git repository to enable version control and audit trails through standard Git operations like branch, merge, and revert.
• Validate metadata using static linting of YAML and Markdown files before allowing deployment to catch syntax errors and schema violations early.
• Detect drift using kcmd pull --dry-run to ensure the repository state matches the actual Knowledge Catalog state before pushing changes.
• Deploy safely with the idempotent kcmd push command, which supports --dry-run testing and atomic updates to prevent partial state corruption.
• Handle conflicts by aborting pushes when drift is detected, forcing reconciliation through pull requests rather than silent overwrites.

Frequently Asked Questions

How does the kcmd push command handle partial failures during deployment?

The kcmd push operation is designed to be atomic and idempotent. If any part of the metadata update fails validation or encounters a permissions error, the entire transaction is aborted, preventing partial updates that could leave the catalog in an inconsistent state. You can verify behavior before execution by running kcmd push --dry-run to simulate the changes.

Can I use kcmd with CI/CD platforms other than GitHub Actions?

Yes, the kcmd library and CLI are platform-agnostic Node.js tools that execute in any environment supporting Node.js 20 or higher. You can integrate the same kcmd pull and kcmd push commands into GitLab CI, CircleCI, Jenkins, or Azure DevOps pipelines by installing the package via npm install kcmd and configuring the appropriate authentication credentials.

What authentication method does kcmd require for Google Cloud deployment?

kcmd uses standard Google Cloud Application Default Credentials. In CI/CD environments, you must provide a service account key via the GOOGLE_APPLICATION_CREDENTIALS environment variable, as shown in the workflow example. Ensure the service account has the necessary Knowledge Catalog roles (such as datacatalog.entries.update) to perform push operations.

How do I initialize a new metadata repository from an existing BigQuery dataset?

Use the kcmd init command with the --bigquery-dataset flag to scaffold the initial catalog structure from an existing dataset. After initialization, commit the generated files to your repository and proceed with the standard CI/CD validation workflow to manage future changes through version control.

# Initialize a new snapshot for a BigQuery dataset

kcmd init --bigquery-dataset my-project.sales_data

# Pull the latest state from the service (dry-run first)

kcmd pull --dry-run

# After review, push changes to Knowledge Catalog

kcmd push

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}