Authoring Semantic Graphs and ContextSet with Metadata as Code Format: A Complete Guide

The Google Cloud Knowledge Catalog repository enables developers to define data assets, semantic relationships, and execution contexts as version-controlled Python code, which the Reference Agent materializes into queryable knowledge graphs.

The Google Cloud Platform knowledge-catalog open-source repository provides a production-grade framework for authoring semantic graphs and ContextSet with Metadata as Code format. By treating metadata definitions as ordinary Python modules, data teams can version-control their knowledge graphs, automate validation pipelines, and programmatically register assets with Google Cloud's Data Catalog.

Understanding Metadata-as-Code Architecture

The repository implements a five-layer architecture that separates metadata authorship from graph materialization. Each layer corresponds to specific source files and responsibilities:

• Metadata Definition Layer: Developers author Python functions in modules like samples/enrichment/src/enrichment/metadata/catalog.py that return dictionaries describing asset schemas and semantic edges (related_to, derived_from).
• Ingestion & Enrichment Layer: The BundleBuilder class in okf/src/reference_agent/tools/bundle_tools.py validates these dictionaries and aggregates them into a graph bundle—a self-contained representation of the knowledge graph.
• Context Set Management Layer: Encapsulated in okf/src/reference_agent/tools/context.py, this layer captures execution context (project ID, region, credentials) that influences how queries are resolved.
• Reference Agent Layer: Orchestrated through okf/src/reference_agent/agent.py, this component registers bundles with Knowledge Catalog and maintains the semantic graph engine.
• Discovery Layer: Implemented in samples/discovery/agent.py, this provides natural-language query interfaces that decompose questions into semantic sub-queries against the graph.

Defining Semantic Assets in Python

Metadata-as-Code requires no external DSL. You define assets by returning plain Python dictionaries from functions that describe both the asset structure and its relationships to other resources.

In samples/enrichment/src/enrichment/metadata/catalog.py, asset definitions include semantic edges that establish lineage and dependency relationships:

# samples/enrichment/src/enrichment/metadata/catalog.py

def bigquery_table():
    return {
        "name": "my_dataset.sales",
        "description": "Sales transactions for the current fiscal year",
        "schema": [
            {"name": "order_id", "type": "STRING"},
            {"name": "order_date", "type": "DATE"},
            {"name": "amount", "type": "FLOAT"},
        ],
        # Semantic edges – link to the source system and downstream reports

        "semantic_edges": [
            {"type": "derived_from", "target": "pubsub.sales_events"},
            {"type": "feeds", "target": "dataview.monthly_sales_report"},
        ],
    }

Each dictionary becomes a node in the semantic graph, while the semantic_edges array creates directed edges linking related assets.

Building the Semantic Graph Bundle

The BundleBuilder transforms metadata modules into a deployable graph representation. Located in okf/src/reference_agent/tools/bundle_tools.py, this class walks the supplied Python files, extracts function returns, and constructs the bundle object.

from okf.src.reference_agent.tools.bundle_tools import BundleBuilder

builder = BundleBuilder(
    metadata_modules=["samples/enrichment/src/enrichment/metadata/catalog.py"],
    context={"project": "my-gcp-project", "region": "us-central1"},
)
graph_bundle = builder.build()

The build() method validates schema compliance and produces a graph bundle containing all entity nodes and relationship edges ready for registration.

Managing Execution Context with ContextSet

ContextSet management ensures that graph operations respect project boundaries and authentication scopes. The Context class in okf/src/reference_agent/tools/context.py encapsulates the execution environment, storing project identifiers, regions, and credential contexts.

This context object attaches to every bundle and query request, enabling context-aware semantic search. When the Reference Agent processes queries, it automatically filters results to assets within the specified project and region, preventing cross-contamination between environments.

Registering and Materializing the Graph

The Reference Agent orchestrates the transition from code to catalog. Using KnowledgeCatalogRunner from the agent module, you materialize the bundle as Knowledge Catalog entities:

from okf.src.reference_agent.runner import KnowledgeCatalogRunner

runner = KnowledgeCatalogRunner(context={"project": "my-gcp-project"})
runner.register_bundle(graph_bundle)

The register_bundle() method communicates with the Data Catalog API, persisting each dictionary as an entity node and each semantic edge as a relationship. This transforms your version-controlled Python definitions into searchable, queryable catalog entries.

Querying the Semantic Graph

Once materialized, the Discovery Agent provides AI-driven interfaces for complex traversals. Located in samples/discovery/agent.py, this component decomposes natural language questions into multiple semantic sub-queries, executes them against the graph, and re-ranks results.

from samples.discovery.agent import DiscoveryAgent

agent = DiscoveryAgent(project="my-gcp-project")
answer = agent.ask(
    "Which tables contain sales amounts for the last quarter and are used in the executive dashboard?"
)
print(answer)

Behind the scenes, the agent handles the graph traversal logic required to answer lineage questions like "who owns this column?" or "what downstream assets depend on X?".

Summary

• Metadata-as-Code eliminates manual UI steps by defining Google Cloud Knowledge Catalog assets as Python dictionaries in samples/enrichment/src/enrichment/metadata/catalog.py.
• The BundleBuilder in okf/src/reference_agent/tools/bundle_tools.py aggregates definitions into deployable graph bundles.
• ContextSet objects in okf/src/reference_agent/tools/context.py scope all operations to specific projects and regions.
• The Reference Agent (okf/src/reference_agent/agent.py) materializes code definitions into catalog entities and edges.
• Semantic edges (derived_from, feeds, etc.) create navigable relationships between data assets.
• The Discovery Agent (samples/discovery/agent.py) enables natural-language querying of the resulting knowledge graph.

Frequently Asked Questions

What is the Metadata-as-Code pattern in Knowledge Catalog?

Metadata-as-Code treats data asset definitions as version-controlled Python code rather than manual console entries. In the knowledge-catalog repository, you define assets by writing functions that return dictionaries containing schemas, descriptions, and semantic edges. This approach enables Git-based workflows, automated testing, and CI/CD pipelines for catalog management.

How do ContextSets scope semantic graph queries?

ContextSets encapsulate execution parameters (project ID, region, credentials) that constrain graph operations. When the Reference Agent processes a bundle or query, it attaches the Context object from okf/src/reference_agent/tools/context.py to ensure results only include assets from the specified environment, preventing unauthorized cross-project visibility.

What are semantic edges and how do they establish relationships?

Semantic edges are directional relationships defined in the semantic_edges array of asset dictionaries. Types like derived_from or feeds create linked connections between nodes (e.g., linking a BigQuery table to its Pub/Sub source). These edges materialize as graph relationships in Knowledge Catalog, enabling the Discovery Agent to traverse lineage and dependency paths.

How does the Reference Agent differ from direct Data Catalog API usage?

The Reference Agent (okf/src/reference_agent/agent.py) abstracts the Data Catalog API by providing bundle management—it validates local Python definitions, builds dependency graphs, and handles batch registration. Unlike direct API calls, the agent maintains the semantic graph engine that indexes entities for fast similarity lookups and supports the natural-language query capabilities of the Discovery Agent.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}