How the Enrichment Agent Integrates with Knowledge Catalog for Metadata Updates

The Enrichment Agent integrates with Google Knowledge Catalog by retrieving existing BigQuery table metadata via the Dataplex Catalog API, enriching it through an LLM-powered workflow, and persisting updates back to the catalog through a local snapshot mechanism.

The GoogleCloudPlatform/knowledge-catalog repository provides a Python-based Enrichment Agent that automates metadata management for BigQuery tables stored in Google Knowledge Catalog. This agent acts as an intelligent bridge that pulls structured metadata from the catalog, generates AI-enhanced descriptions and documentation, and pushes the enriched artifacts back into Knowledge Catalog as updated entries.

Retrieving Current Metadata from Knowledge Catalog

The integration begins with metadata discovery in samples/enrichment/src/enrichment/metadata/catalog.py. The agent creates a dataplex.CatalogServiceClient and executes a two-phase lookup:

1. Search Entries: Calls search_entries to locate the target table within the catalog namespace.
2. Get Entry Details: Retrieves the full entry using get_entry to extract the current schema, existing descriptions, and documentation.

The lookup_table_info() function formats this retrieved data into a context string suitable for LLM processing. This function returns both the formatted context and a boolean flag indicating whether documentation already exists.

Generating Enriched Content with LLM

Once the metadata is retrieved, the enrichment process moves to samples/enrichment/src/enrichment/enrich.py. This core CLI driver orchestrates the LLM generation phase:

• Prompt Construction: Builds a structured prompt containing the table name, source description, full schema, and existing documentation.
• LLM Execution: Invokes the kcagent enrich command with the constructed prompt to generate updated documentation or new descriptions.
• Content Processing: The runner processes the LLM output and prepares it for persistence.

This step transforms raw technical metadata into human-readable, enriched documentation using AI generation.

Persisting Updates via the Snapshot Mechanism

The final integration phase handles writing updates back to Knowledge Catalog through samples/enrichment/src/enrichment/metadata/snapshot.py. Rather than direct API writes, the agent uses a local snapshot workflow:

• Update Entry: The snapshot.update_entry() function writes generated content to a metadata snapshot on the local filesystem. This function takes the source metadata directory, output directory, table name, and new content as parameters.
• Snapshot Management: The snapshot maintains a local representation of the catalog state, allowing for batch updates and review before publication.
• Publish to Catalog: The enriched snapshot is later published to Knowledge Catalog using the publish CLI command (implemented in samples/enrichment/src/enrichment/publish.py), completing the round-trip update.

This decoupled approach enables validation and review of AI-generated content before it reaches the production catalog.

Practical Implementation Examples

The following examples demonstrate the key integration points:

Fetch existing table metadata from Knowledge Catalog:

from enrichment.metadata import catalog

# Returns formatted context and documentation status

context, has_doc = catalog.lookup_table_info(
    "my-project.my_dataset.my_table"
)
print(context)

Update a table entry in the local snapshot:

from enrichment.metadata import snapshot
import pathlib

metadata_dir = pathlib.Path("metadata").resolve()
output_dir = pathlib.Path("metadata_updated").resolve()

def update_table(table_name: str, content: str) -> str:
    try:
        snapshot.update_entry(metadata_dir, output_dir, table_name, content)
        return f"Updated {table_name}"
    except Exception as e:
        return f"Failed: {e}"

Execute the complete enrichment workflow:

# Initialize catalog snapshot

kcmd init --bigquery-dataset my-project.my_dataset

# Pull latest metadata from Knowledge Catalog

kcmd pull

# Enrich using LLM

kcagent enrich \
  --catalog-path . \
  --tools-path tools \
  --prompt-path prompt.md

# Publish updates back to Knowledge Catalog

kcmd publish

Summary

• The Enrichment Agent connects to Knowledge Catalog through the dataplex.CatalogServiceClient to retrieve current BigQuery table metadata via search_entries and get_entry.
• Retrieved metadata is processed through the kcagent enrich CLI, which constructs LLM prompts and generates enriched documentation.
• Updates are written to a local snapshot using snapshot.update_entry() in samples/enrichment/src/enrichment/metadata/snapshot.py before being published back to the catalog.
• This three-phase workflow (discovery, enrichment, persistence) enables automated, AI-generated metadata updates while maintaining data governance through snapshot review.

Frequently Asked Questions

How does the Enrichment Agent authenticate with Knowledge Catalog?

The agent uses the standard Google Cloud authentication chain via the dataplex.CatalogServiceClient implemented in samples/enrichment/src/enrichment/metadata/catalog.py. It relies on Application Default Credentials (ADC) or service account keys configured in the environment, leveraging the standard Google Cloud client libraries for secure API access to the Dataplex Catalog API.

What happens if the Knowledge Catalog entry does not exist?

The catalog.lookup_table_info() function handles discovery by first calling search_entries. If no entries are found or get_entry returns no results, the function typically returns an empty context or raises an exception depending on the implementation, preventing the enrichment process from proceeding with non-existent tables.

Can I review changes before they are published to Knowledge Catalog?

Yes. The snapshot mechanism in samples/enrichment/src/enrichment/metadata/snapshot.py decouples enrichment from publication. Updates are written to a local filesystem directory (metadata_updated by default), allowing teams to review AI-generated content using kcmd tools before executing the publish command to commit changes to Knowledge Catalog.

Which metadata fields can the Enrichment Agent update?

The agent primarily updates table descriptions and documentation fields stored in the catalog entry's metadata. The snapshot.update_entry() function handles arbitrary text content, allowing the LLM to generate comprehensive documentation, business descriptions, and contextual metadata that enhance the raw schema information stored in Knowledge Catalog.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}