How to Add Custom Third-Party Entry Types to Knowledge Catalog Metadata

You can add custom third-party entry types to Google Cloud Knowledge Catalog by creating an EntryGroup, defining a unique fully-qualified type identifier, and using the Dataplex client library to persist entries with custom aspects, as demonstrated in the GoogleCloudPlatform/knowledge-catalog repository.

The GoogleCloudPlatform/knowledge-catalog repository provides libraries and sample agents for managing metadata in Google Cloud Knowledge Catalog (formerly Dataplex). When cataloging assets beyond native support—such as proprietary data sources or custom SaaS services—you need to extend the catalog with custom third-party entry types that integrate seamlessly with existing enrichment pipelines.

Understanding Knowledge Catalog Entry Types

Knowledge Catalog organizes metadata as entries belonging to an EntryGroup, each with a specific entry type like BigQuery tables or Pub/Sub topics. For unsupported systems, you define custom entry types using fully-qualified names without modifying server-side configurations. The repository abstracts low-level REST calls behind Python client helpers in samples/enrichment/src/enrichment/metadata/catalog.py, allowing you to focus on metadata structure rather than API plumbing.

Step-by-Step: Adding Custom 3rd Party Entry Types

Create an Entry Group

EntryGroups act as logical containers for your custom entries. The sample code in samples/enrichment/src/enrichment/metadata/catalog.py demonstrates how to initialize a client and create a container for your assets.

from google.cloud import dataplex_v1

# Initialize the Catalog service client

catalog = dataplex_v1.CatalogServiceClient()

# Define the parent location for the EntryGroup

parent = "projects/{project}/locations/{location}"

# Create the EntryGroup

entry_group = dataplex_v1.EntryGroup()
entry_group.display_name = "My Custom Sources"

response = catalog.create_entry_group(
    parent=parent,
    entry_group_id="my_custom_group",
    entry_group=entry_group
)

Define the Custom Entry Type

Custom entry types are identified by fully-qualified names (e.g., projects/PROJECT/locations/LOCATION/entryTypes/my_custom_type). You do not need to register these types on the server; you simply reference a unique identifier when creating entries.

Choose a naming convention that reflects your system:

# Define your custom entry type identifier

custom_type = "projects/{project}/locations/{location}/entryTypes/my_custom_type"

Build and Persist the Entry

Use the dataplex.Entry() protobuf to construct your entry. Set entry_type to your custom identifier, populate the entrySource fields with system-specific details, and optionally attach custom aspects for richer metadata. The helper functions in samples/enrichment/src/enrichment/metadata/catalog.py handle the persistence layer.

# Build the entry with custom type

entry = dataplex_v1.Entry()
entry.name = f"projects/{project}/locations/{location}/entryGroups/my_group/entries/my_asset"
entry.display_name = "My Custom Asset"
entry.entry_type = custom_type
entry.entry_source.resource = "my-system://my-custom-asset/12345"
entry.entry_source.system = "my_custom_system"
entry.entry_source.description = "Metadata for a proprietary data source"

# Optional: attach a custom aspect with JSON schema

aspect = {"schema": {"fields": [{"name": "id", "type": "STRING"}]}}
entry.aspects["my_custom_aspect"] = aspect

# Persist the entry to Knowledge Catalog

catalog.create_entry(
    parent=f"projects/{project}/locations/{location}/entryGroups/my_group",
    entry_id="my_asset",
    entry=entry
)

To update existing entries, use catalog.update_entry(entry=entry) with the same pattern.

Consume the Entry

Downstream agents retrieve entries using catalog.get_entry() and process custom aspects. The enrichment agent in toolbox/enrichment/src/tools/md/main.ts and the reference agent in okf/src/reference_agent/cli.py demonstrate how to read entries—including custom types—for further processing.

# Retrieve the custom entry

entry_name = "projects/{project}/locations/{location}/entryGroups/my_group/entries/my_asset"
custom_entry = catalog.get_entry(name=entry_name)

# Access custom metadata

print(custom_entry.display_name)          # → My Custom Asset

print(custom_entry.entry_source.resource) # → my-system://my-custom-asset/12345

print(custom_entry.aspects["my_custom_aspect"])

Working with Custom Aspects and Markdown Conversion

For GitOps workflows, convert entries to markdown using utilities in samples/enrichment/src/enrichment/metadata/snapshot.py. The _entry_to_md function serializes entry objects to markdown format, while _md_to_entry performs the reverse conversion. This enables you to version control your custom entry type definitions alongside your code, as documented in toolbox/mdcode/README.md.

The built-in examples for BigQuery tables in snapshot.py illustrate the same conversion flow, which you can adapt for custom types by adjusting the entry_type and asset-specific fields.

Summary

• EntryGroups serve as logical containers for entries, created via catalog.create_entry_group() in samples/enrichment/src/enrichment/metadata/catalog.py.
• Custom entry types require only a unique fully-qualified name (e.g., projects/PROJECT/locations/LOCATION/entryTypes/my_custom_type), not server-side registration.
• Entry construction uses dataplex.Entry() with entry_type and entrySource fields to define proprietary assets.
• Persistence occurs through catalog.create_entry() or catalog.update_entry() methods.
• Markdown conversion helpers _entry_to_md and _md_to_entry in snapshot.py enable version control of custom metadata.

Frequently Asked Questions

What is an EntryGroup in Knowledge Catalog?

An EntryGroup is a logical container that organizes related entries within a specific location. According to the source code in samples/enrichment/src/enrichment/metadata/catalog.py, you create EntryGroups using the catalog.create_entry_group() method to house collections of custom third-party entries.

Do I need to register custom entry types before using them?

No. You define custom entry types by using a unique fully-qualified name when creating the entry. The Knowledge Catalog API accepts these identifiers (e.g., projects/PROJECT/locations/LOCATION/entryTypes/my_custom_type) without requiring pre-registration or server-side configuration changes.

How do I convert entries to markdown for version control?

Use the _entry_to_md helper function in samples/enrichment/src/enrichment/metadata/snapshot.py to serialize entries to markdown format, and _md_to_entry to parse them back into protobuf objects. This integrates with the mdcode workflow described in toolbox/mdcode/README.md for managing metadata as code.

Can enrichment agents process custom third-party entry types?

Yes. The enrichment agent in toolbox/enrichment/src/tools/md/main.ts and the reference agent in okf/src/reference_agent/cli.py read entries via catalog.get_entry() and can process custom aspects. This allows you to extend agent functionality to handle proprietary systems and custom metadata schemas.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}