How to Configure What Metadata to Publish vs What to Keep Local-Only in Knowledge Catalog

You control which metadata entries are published to Google Dataplex by selecting specific directories or implementing file-name filters in the publishing workflow, since the snapshot.publish_entries function iterates over all *.md files in the supplied directory by default.

The Knowledge Catalog repository stores data asset metadata as Markdown files with YAML front-matter that lives in a local metadata directory. When you run the publishing workflow, the system pushes these local entries to Google Dataplex, but you may need to keep certain draft or sensitive metadata local-only. Understanding how to configure what metadata to publish vs what to keep local-only ensures you maintain strict control over your data catalog visibility and compliance requirements.

How the Publishing Workflow Handles Metadata

The publishing logic is implemented in samples/enrichment/src/enrichment/metadata/snapshot.py. The publish_entries function follows a four-step process to convert local Markdown files into Dataplex entries:

1. Load and parse – The _md_to_entry function reads each Markdown file and extracts the YAML front-matter (containing name and resource) along with the body content.
2. Protobuf conversion – The code converts the Python dictionary into a Dataplex entry object using jsonpb.ParseDict(entry_data, updated_entry._pb, ignore_unknown_fields=True).
3. Selective aspect publishing – Only the aspects defined in OVERVIEW_ASPECT_KEY are sent to the catalog via catalog.update_entry(request=dataplex.UpdateEntryRequest(...)).
4. Directory iteration – The function loops through for table_path in dir.glob('*.md'): and processes every matching file in the supplied directory.

Because this workflow automatically publishes all Markdown files it finds, you must implement specific strategies to exclude certain entries from being sent to Dataplex.

Strategies for Keeping Metadata Local-Only

To prevent specific metadata entries from being published while keeping them available for local enrichment, you have two practical approaches that work within the existing codebase structure.

Directory-Based Isolation

Store publishable metadata in one folder and local-only metadata in another. The CLI entry point in samples/enrichment/src/enrichment/publish.py accepts a --dir argument that determines which directory gets processed.

Create a directory structure like:

metadata/
├── published/    # Entries to push to Dataplex

└── local/        # Entries that remain local-only

Then execute the publish command targeting only the publishable directory:

python -m enrichment.publish --dir=metadata/published

This method requires no code changes and works immediately with the existing snapshot.publish_entries implementation.

File-Name Pattern Filtering

Modify the publishing logic to skip files matching specific naming conventions. You can patch the loop in snapshot.publish_entries to exclude files with a .local.md suffix:

# Add this filter inside samples/enrichment/src/enrichment/metadata/snapshot.py

for table_path in dir.glob('*.md'):
    # Skip local-only files

    if table_path.name.endswith('.local.md'):
        continue
    entry_data = _md_to_entry(table_path.read_text())
    # ... existing publishing logic ...

This approach allows you to store both published and local-only entries in the same directory while maintaining granular control over visibility.

Key Configuration Points in the Codebase

Understanding where the configuration lives helps you implement the right level of control for your metadata workflow.

CLI Entry Point

The samples/enrichment/src/enrichment/publish.py file parses the --dir argument and forwards the path to snapshot.publish_entries. This is the primary interface for triggering the publishing workflow:

# From publish.py

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--dir', required=True, help='Metadata directory')
    args = parser.parse_args()
    # Calls snapshot.publish_entries(args.dir)

Core Metadata Handling

All parsing, conversion, and Dataplex interaction logic resides in samples/enrichment/src/enrichment/metadata/snapshot.py. This file contains the publish_entries function (lines 110-127) that handles the dataplex.UpdateEntryRequest and determines which aspects get published.

Non-Publishing Enrichment

The samples/enrichment/src/enrichment/enrich.py agent reads metadata files and updates them locally but never publishes to Dataplex. This allows you to enrich metadata (adding descriptions, tags, or documentation) while keeping entries local-only until you explicitly run the publish command.

Practical Implementation Examples

Here are complete implementations for common scenarios:

Publishing all entries in a directory:

python -m enrichment.publish --dir=metadata

Publishing only a curated subset using subdirectories:

# Place entries you want published in metadata/published/

python -m enrichment.publish --dir=metadata/published

Running enrichment without publishing (keeps everything local):

python -m enrichment.enrich \
  --dir=metadata \
  --config-dir=config

Filtering local-only files by suffix in snapshot.py:

def publish_entries(dir: Path):
    for table_path in dir.glob('*.md'):
        # Keep any file ending with .local.md local-only

        if table_path.name.endswith('.local.md'):
            continue
        entry_data = _md_to_entry(table_path.read_text())
        # ... rest of publishing logic ...

Summary

• The snapshot.publish_entries function in samples/enrichment/src/enrichment/metadata/snapshot.py publishes every *.md file in the supplied directory by default.
• Use directory isolation to control publishing by pointing the --dir argument at specific subdirectories.
• Implement file-name filtering (such as .local.md suffixes) to granularly exclude specific entries without restructuring your directories.
• The enrichment agent (enrich.py) processes metadata locally without publishing, allowing you to prepare entries before making them public.
• All publishing configuration flows through samples/enrichment/src/enrichment/publish.py, which serves as the CLI entry point.

Frequently Asked Questions

How does the Knowledge Catalog repository determine which metadata aspects to publish?

The publishing workflow specifically sends only the Overview aspect defined in OVERVIEW_ASPECT_KEY to Google Dataplex. When _md_to_entry parses the Markdown file, it extracts the YAML front-matter and body content, then jsonpb.ParseDict converts this into a Dataplex entry protobuf. The catalog.update_entry method receives only these defined aspects, ensuring that local-only extensions or custom front-matter fields not part of the standard aspect schema remain in your local files only.

Can I keep metadata in the same directory but prevent some files from publishing?

Yes, you can keep draft or sensitive metadata in the same directory by implementing a file-name filter in snapshot.publish_entries. Add a conditional check that skips files matching specific patterns, such as if table_path.name.endswith('.local.md'): continue, before the code calls _md_to_entry. This modification allows you to maintain a single metadata directory while controlling visibility at the individual file level.

What is the difference between the enrich and publish commands in Knowledge Catalog?

The enrich command (samples/enrichment/src/enrichment/enrich.py) reads local Markdown files, processes them through enrichment agents (such as metadata extraction or documentation generation), and writes updates back to the same local files without ever contacting Dataplex. The publish command (samples/enrichment/src/enrichment/publish.py) triggers snapshot.publish_entries, which converts the local Markdown into Dataplex entry objects and sends them to the Google Cloud catalog via the Dataplex API.

Where should I modify the code to change which directory gets published?

Modify the samples/enrichment/src/enrichment/publish.py file if you need to change how the --dir argument is processed, or update samples/enrichment/src/enrichment/metadata/snapshot.py if you need to change the file iteration logic within publish_entries. The publish.py file simply forwards the directory path to the snapshot module, so implementation of filtering logic belongs in snapshot.py where the dir.glob('*.md') iteration occurs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}