Implementing Metadata Versioning Strategies Using Git Branches in Knowledge Catalog

You can implement robust metadata versioning strategies using Git branches by treating Knowledge Catalog entries as plain-text Markdown files, enabling traceability, isolation, and rollback through standard Git workflows.

The GoogleCloudPlatform/knowledge-catalog repository stores metadata as plain-text Markdown files that are version-controlled with Git. This design makes it straightforward to apply classic Git branching techniques to manage different metadata versions (e.g., development, staging, production) and to record the evolution of catalog entries over time. By leveraging Git’s native capabilities, teams can treat infrastructure-as-code principles for their data catalog metadata.

Why Version Metadata with Git?

Storing metadata as Markdown in Git provides several architectural advantages over traditional database-only approaches. Traceability is automatic because every change becomes a Git commit, creating a complete audit trail. Isolation allows separate branches to hold independent versions of the same entry, such as a new schema that is not yet released. Collaboration enables multiple contributors to work on different branches and merge changes later, with Git automatically handling conflicts. Rollback becomes trivial—reverting to a previous commit or branch instantly restores an older metadata snapshot.

The repository provides utilities that read and write these Markdown files, such as snapshot.py in the enrichment samples, which can run against any Git checkout without modification.

Branch-Based Workflow for Knowledge Catalog

Implementing a version strategy follows a standard Git branching model. Each environment or release can exist as a separate branch, with changes flowing from development through staging to production.

1. Create a version branch from your main line of metadata:

   git checkout -b metadata/v1.2

2. Make changes by editing the Markdown files (e.g., adding a new column description):

   vi datasets/my_dataset/table_a.md
   git add datasets/my_dataset/table_a.md
   git commit -m "Add column new_field description"

3. Validate the changes using the provided enrichment helpers to ensure files are well-formed:

   python -m samples.enrichment.src.enrichment.metadata.snapshot \
          --dir datasets/ my_dataset

4. Push the branch to share the version with teammates or CI systems:

   git push origin metadata/v1.2

5. Review and merge through a Pull Request. When approved, merge into the target branch:

   git checkout main
   git merge --no-ff metadata/v1.2 -m "Release metadata v1.2"

6. Tag the release for downstream tools that need stable references:

   git tag -a v1.2 -m "Metadata version 1.2"
   git push origin v1.2

Handling Merge Conflicts in Metadata Files

When two branches modify the same entry, Git flags a conflict as a standard line-based diff. Because each entry is a single Markdown document, resolution is straightforward using manual editing or visual merge tools. After fixing the conflict markers, complete the merge:

git add <conflicted_file>
git commit

This simplicity exists because the Knowledge Catalog uses flat Markdown files rather than binary formats or complex database schemas, making line-level diffs human-readable.

Automating Validation with CI/CD

The repository contains a toolbox that integrates with CI/CD pipelines. You can wire snapshot.py into GitHub Actions to validate metadata on every push to version branches.

The following workflow runs validation automatically:

name: Validate Metadata

on:
  push:
    branches: [ "metadata/*" ]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install -r samples/discovery/requirements.txt
      - name: Run validator
        run: |
          python -m samples.enrichment.src.enrichment.metadata.snapshot \
            --dir datasets/

This automation ensures that only valid Markdown reaches your production branches, catching syntax errors before they propagate to downstream consumers.

Practical Automation Scripts

Bash Script for Version Branch Creation

Automate the initialization of new metadata versions with this script:

#!/usr/bin/env bash
set -euo pipefail

VERSION=$1
BASE_BRANCH=${2:-main}

git checkout "$BASE_BRANCH"
git pull origin "$BASE_BRANCH"
git checkout -b "metadata/$VERSION"

echo "<!-- TODO: update description -->" >> datasets/sample/table.md

git add datasets/sample/table.md
git commit -m "Initialize metadata $VERSION"
git push -u origin "metadata/$VERSION"

Python Helper for Metadata Discovery

List all metadata files in the current branch for batch processing:

import pathlib

def list_metadata(dir_path: pathlib.Path) -> list[pathlib.Path]:
    """Return a list of all *.md metadata files under dir_path."""
    return sorted(dir_path.rglob("*.md"))

if __name__ == "__main__":
    from argparse import ArgumentParser

    p = ArgumentParser()
    p.add_argument("--dir", default="datasets")
    args = p.parse_args()

    for f in list_metadata(pathlib.Path(args.dir)):
        print(f.relative_to(pathlib.Path(args.dir)))

Key Files in the Repository

Understanding the repository structure helps implement these versioning strategies effectively:

• README.md – High-level project description and entry point for users
• SPEC.md (in okf/) – Defines the Knowledge Catalog data model and conventions
• samples/enrichment/src/enrichment/metadata/snapshot.py – Core utility that converts between Dataplex entries and Markdown files
• toolbox/mdcode/README.md – Documentation for Markdown-driven code generation
• samples/discovery/requirements.txt – Dependencies for the Cloud Dataplex client used in validation scripts

These files collectively illustrate how the repository structures metadata and provides utilities to manipulate it across different Git branches.

Summary

• Git-native versioning treats metadata as code, providing audit trails and rollback capabilities through standard Git operations.
• Branch isolation allows simultaneous development of multiple metadata versions without affecting production entries.
• Markdown-based storage enables line-level diffs and simple conflict resolution compared to binary or database formats.
• Automated validation using snapshot.py in CI/CD pipelines ensures metadata integrity before merging to main branches.
• Repository utilities in samples/enrichment/ provide ready-to-use tools for reading and writing versioned metadata files.

Frequently Asked Questions

How does Git branching prevent accidental overwrites of production metadata?

Git branching creates isolated workspaces where changes remain invisible to other branches until explicitly merged. When you work on a metadata/v1.2 branch, modifications to datasets/ Markdown files exist only in that branch, leaving the main or prod branch untouched. Only after review and explicit merge does the new version affect production, with Git’s merge algorithms preventing silent overwrites by flagging conflicts.

Can I use tags instead of branches for metadata versioning?

Yes, Git tags provide stable references to specific points in history, complementing branch-based workflows. While branches are ideal for active development and parallel versions, tags serve as immutable markers for released metadata states. According to the repository conventions, you can tag releases after merging: git tag -a v1.2 -m "Metadata version 1.2" provides a permanent reference that downstream tools can use to fetch exact metadata snapshots.

What happens when two users edit the same metadata file on different branches?

Git detects the conflict during merge and marks the file with conflict markers showing both versions. Because Knowledge Catalog uses plain Markdown, you can resolve conflicts using standard text editors or merge tools by choosing which line changes to keep. After editing, git add and git commit complete the merge, preserving both development histories in the commit graph.

How do I validate metadata before merging to the main branch?

Run the snapshot.py utility from samples/enrichment/src/enrichment/metadata/ against your working directory. This script parses the Markdown files and validates their structure against the Knowledge Catalog schema defined in SPEC.md. Integrating this command into CI/CD pipelines—triggered on pushes to metadata/* branches—ensures validation happens automatically before human review.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}