# Implementing CI/CD Pipelines for Metadata with Version Control: A Metadata-as-Code Guide

> Implement CI/CD pipelines for metadata with version control using the kcmd CLI. Treat metadata as code for automated validation and push to Google Cloud Knowledge Catalog.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: how-to-guide
- Published: 2026-07-14

---

**The GoogleCloudPlatform/knowledge-catalog repository enables implementing CI/CD pipelines for metadata with version control by treating YAML and Markdown metadata files as source code artifacts, using the kcmd CLI to validate, snapshot test, and idempotently push changes to Google Cloud Knowledge Catalog.**

The Knowledge Catalog repository adopts a **Metadata-as-Code** philosophy that positions metadata definitions as first-class source artifacts. By storing catalog definitions in Git alongside application code, data teams can apply standard DevOps practices—automated testing, peer review, and continuous deployment—to manage data assets at scale. This approach centers on the kcmd library and CLI tools found in the `toolbox/mdcode` directory, allowing you to version, validate, and deploy metadata with the same rigor as application code.

## Metadata-as-Code Repository Structure

The architecture mirrors the resource hierarchy of underlying data assets to ensure both human readability and automation compatibility.

### Catalog Manifest and Entry Files

At the repository root, the **catalog.yaml** manifest defines the scope of the catalog—such as a BigQuery dataset or user-managed EntryGroup—and enumerates the entry types, aspects, and links to be captured. Beneath the `catalog/` directory, each entry is represented by a YAML file (`<entry-id>.yaml`) alongside optional Markdown side-car files (`<entry-id>.overview.md`) for unstructured aspects like overviews.

This structure is documented in [`toolbox/mdcode/docs/concept.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/docs/concept.md) and exemplified in [`toolbox/mdcode/README.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/README.md), ensuring that changes to the scope or included aspects are tracked atomically alongside the entry files.

## CI/CD Pipeline Components

A robust pipeline for metadata requires three core stages to ensure safety and consistency before deployment to production.

### Static Validation

The lint step verifies YAML syntax integrity, validates required fields against the schema, and ensures that Markdown side-car files contain properly formatted front-matter. This prevents malformed metadata from entering the catalog and failing at runtime.

### Snapshot Testing and Drift Detection

Using the **kcmd** library, the pipeline executes `kcmd pull --dry-run` to capture the current state from Knowledge Catalog and compare it against the repository version. If unexpected drift is detected—indicating out-of-band changes in the catalog—the CI build fails, triggering human review before any write operations occur.

The expected pull behavior is illustrated in [`toolbox/mdcode/tests/scenarios/pull_basic.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/pull_basic.yaml).

### Idempotent Push Deployment

Upon successful validation, the pipeline invokes `kcmd push` to publish metadata changes. This operation is **idempotent** and respects the `--dry-run` flag for safety testing before actual deployment. The push command publishes new entries and updates existing ones atomically, ensuring the catalog state converges to the repository definition.

The push operation schema is demonstrated in [`toolbox/mdcode/tests/scenarios/push_new_entry.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/tests/scenarios/push_new_entry.yaml).

## Implementing the CI/CD Workflow

A production-ready pipeline orchestrates validation and deployment through environment-specific stages using standard CI/CD platforms.

### GitHub Actions Configuration

The following workflow file ([`.github/workflows/metadata-ci.yml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/.github/workflows/metadata-ci.yml)) demonstrates the complete lifecycle from validation to deployment:

```yaml
name: Metadata CI/CD
on:
  push:
    paths:
      - 'toolbox/mdcode/**'
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      - name: Install kcmd
        run: npm install kcmd
      - name: Lint YAML/MD
        run: |
          npx yaml-lint toolbox/mdcode/**/**/*.yaml
          markdownlint toolbox/mdcode/**/**/*.md
      - name: Pull latest catalog snapshot
        run: |
          npx kcmd pull --dry-run
      - name: Run unit tests
        run: npm test
  deploy:
    needs: validate
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install kcmd
        run: npm install kcmd
      - name: Push metadata
        env:
          GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
        run: |
          npx kcmd push

```

### Environment-Specific Deployment

The deploy job triggers only on the main branch after successful validation, using `GOOGLE_APPLICATION_CREDENTIALS` stored as repository secrets to authenticate push operations against the Google Cloud API.

## Conflict Detection and Resolution

When concurrent edits occur in Knowledge Catalog outside the repository workflow, `kcmd pull` surfaces conflicts during the snapshot phase. The CI step automatically aborts the push operation, prompting developers to reconcile changes via a pull request. This mechanism prevents accidental overwrites of metadata modified outside the version-controlled pipeline, ensuring that the repository remains the single source of truth.

## Programmatic Metadata Management

Beyond CLI usage, the kcmd library supports TypeScript-based automation for complex transformations and bulk updates within CI jobs:

```typescript
import * as kcmd from 'kcmd';

// Load an existing snapshot
const snapshot = kcmd.CatalogSnapshot.fromPath('/path/to/root');

// Modify an entry programmatically
const entry = snapshot.getEntry('my_dataset.my_table');
entry.resource.description = 'Updated description by CI pipeline';
await snapshot.push(); // Publishes the change

```

This programmatic approach enables sophisticated CI logic, such as automatically updating metadata based on schema changes detected in upstream data sources.

## Summary

- **Store metadata as code** in a Git repository to enable version control and audit trails through standard Git operations like branch, merge, and revert.
- **Validate metadata** using static linting of YAML and Markdown files before allowing deployment to catch syntax errors and schema violations early.
- **Detect drift** using `kcmd pull --dry-run` to ensure the repository state matches the actual Knowledge Catalog state before pushing changes.
- **Deploy safely** with the idempotent `kcmd push` command, which supports `--dry-run` testing and atomic updates to prevent partial state corruption.
- **Handle conflicts** by aborting pushes when drift is detected, forcing reconciliation through pull requests rather than silent overwrites.

## Frequently Asked Questions

### How does the kcmd push command handle partial failures during deployment?

The `kcmd push` operation is designed to be atomic and idempotent. If any part of the metadata update fails validation or encounters a permissions error, the entire transaction is aborted, preventing partial updates that could leave the catalog in an inconsistent state. You can verify behavior before execution by running `kcmd push --dry-run` to simulate the changes.

### Can I use kcmd with CI/CD platforms other than GitHub Actions?

Yes, the kcmd library and CLI are platform-agnostic Node.js tools that execute in any environment supporting Node.js 20 or higher. You can integrate the same `kcmd pull` and `kcmd push` commands into GitLab CI, CircleCI, Jenkins, or Azure DevOps pipelines by installing the package via `npm install kcmd` and configuring the appropriate authentication credentials.

### What authentication method does kcmd require for Google Cloud deployment?

kcmd uses standard Google Cloud Application Default Credentials. In CI/CD environments, you must provide a service account key via the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, as shown in the workflow example. Ensure the service account has the necessary Knowledge Catalog roles (such as `datacatalog.entries.update`) to perform push operations.

### How do I initialize a new metadata repository from an existing BigQuery dataset?

Use the `kcmd init` command with the `--bigquery-dataset` flag to scaffold the initial catalog structure from an existing dataset. After initialization, commit the generated files to your repository and proceed with the standard CI/CD validation workflow to manage future changes through version control.

```bash

# Initialize a new snapshot for a BigQuery dataset

kcmd init --bigquery-dataset my-project.sales_data

# Pull the latest state from the service (dry-run first)

kcmd pull --dry-run

# After review, push changes to Knowledge Catalog

kcmd push

```