# How to Configure What Metadata to Publish vs What to Keep Local-Only in Knowledge Catalog

> Learn to configure what metadata to publish or keep local-only in Knowledge Catalog. Control publication by selecting directories or using file-name filters.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: how-to-guide
- Published: 2026-07-14

---

**You control which metadata entries are published to Google Dataplex by selecting specific directories or implementing file-name filters in the publishing workflow, since the `snapshot.publish_entries` function iterates over all `*.md` files in the supplied directory by default.**

The Knowledge Catalog repository stores data asset metadata as **Markdown files** with YAML front-matter that lives in a local metadata directory. When you run the publishing workflow, the system pushes these local entries to Google Dataplex, but you may need to keep certain draft or sensitive metadata local-only. Understanding how to configure what metadata to publish vs what to keep local-only ensures you maintain strict control over your data catalog visibility and compliance requirements.

## How the Publishing Workflow Handles Metadata

The publishing logic is implemented in **[`samples/enrichment/src/enrichment/metadata/snapshot.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/metadata/snapshot.py)**. The `publish_entries` function follows a four-step process to convert local Markdown files into Dataplex entries:

1. **Load and parse** – The `_md_to_entry` function reads each Markdown file and extracts the YAML front-matter (containing `name` and `resource`) along with the body content.
2. **Protobuf conversion** – The code converts the Python dictionary into a Dataplex entry object using `jsonpb.ParseDict(entry_data, updated_entry._pb, ignore_unknown_fields=True)`.
3. **Selective aspect publishing** – Only the aspects defined in `OVERVIEW_ASPECT_KEY` are sent to the catalog via `catalog.update_entry(request=dataplex.UpdateEntryRequest(...))`.
4. **Directory iteration** – The function loops through `for table_path in dir.glob('*.md'):` and processes every matching file in the supplied directory.

Because this workflow automatically publishes **all** Markdown files it finds, you must implement specific strategies to exclude certain entries from being sent to Dataplex.

## Strategies for Keeping Metadata Local-Only

To prevent specific metadata entries from being published while keeping them available for local enrichment, you have two practical approaches that work within the existing codebase structure.

### Directory-Based Isolation

Store publishable metadata in one folder and local-only metadata in another. The CLI entry point in **[`samples/enrichment/src/enrichment/publish.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/publish.py)** accepts a `--dir` argument that determines which directory gets processed.

Create a directory structure like:

```bash
metadata/
├── published/    # Entries to push to Dataplex

└── local/        # Entries that remain local-only

```

Then execute the publish command targeting only the publishable directory:

```bash
python -m enrichment.publish --dir=metadata/published

```

This method requires no code changes and works immediately with the existing `snapshot.publish_entries` implementation.

### File-Name Pattern Filtering

Modify the publishing logic to skip files matching specific naming conventions. You can patch the loop in `snapshot.publish_entries` to exclude files with a [`.local.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/.local.md) suffix:

```python

# Add this filter inside samples/enrichment/src/enrichment/metadata/snapshot.py

for table_path in dir.glob('*.md'):
    # Skip local-only files

    if table_path.name.endswith('.local.md'):
        continue
    entry_data = _md_to_entry(table_path.read_text())
    # ... existing publishing logic ...

```

This approach allows you to store both published and local-only entries in the same directory while maintaining granular control over visibility.

## Key Configuration Points in the Codebase

Understanding where the configuration lives helps you implement the right level of control for your metadata workflow.

### CLI Entry Point

The **[`samples/enrichment/src/enrichment/publish.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/publish.py)** file parses the `--dir` argument and forwards the path to `snapshot.publish_entries`. This is the primary interface for triggering the publishing workflow:

```python

# From publish.py

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--dir', required=True, help='Metadata directory')
    args = parser.parse_args()
    # Calls snapshot.publish_entries(args.dir)

```

### Core Metadata Handling

All parsing, conversion, and Dataplex interaction logic resides in **[`samples/enrichment/src/enrichment/metadata/snapshot.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/metadata/snapshot.py)**. This file contains the `publish_entries` function (lines 110-127) that handles the `dataplex.UpdateEntryRequest` and determines which aspects get published.

### Non-Publishing Enrichment

The **[`samples/enrichment/src/enrichment/enrich.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/enrich.py)** agent reads metadata files and updates them locally but **never** publishes to Dataplex. This allows you to enrich metadata (adding descriptions, tags, or documentation) while keeping entries local-only until you explicitly run the publish command.

## Practical Implementation Examples

Here are complete implementations for common scenarios:

**Publishing all entries in a directory:**

```bash
python -m enrichment.publish --dir=metadata

```

**Publishing only a curated subset using subdirectories:**

```bash

# Place entries you want published in metadata/published/

python -m enrichment.publish --dir=metadata/published

```

**Running enrichment without publishing (keeps everything local):**

```bash
python -m enrichment.enrich \
  --dir=metadata \
  --config-dir=config

```

**Filtering local-only files by suffix in snapshot.py:**

```python
def publish_entries(dir: Path):
    for table_path in dir.glob('*.md'):
        # Keep any file ending with .local.md local-only

        if table_path.name.endswith('.local.md'):
            continue
        entry_data = _md_to_entry(table_path.read_text())
        # ... rest of publishing logic ...

```

## Summary

- The **`snapshot.publish_entries`** function in [`samples/enrichment/src/enrichment/metadata/snapshot.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/metadata/snapshot.py) publishes every `*.md` file in the supplied directory by default.
- Use **directory isolation** to control publishing by pointing the `--dir` argument at specific subdirectories.
- Implement **file-name filtering** (such as [`.local.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/.local.md) suffixes) to granularly exclude specific entries without restructuring your directories.
- The **enrichment agent** ([`enrich.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/enrich.py)) processes metadata locally without publishing, allowing you to prepare entries before making them public.
- All publishing configuration flows through **[`samples/enrichment/src/enrichment/publish.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/publish.py)**, which serves as the CLI entry point.

## Frequently Asked Questions

### How does the Knowledge Catalog repository determine which metadata aspects to publish?

The publishing workflow specifically sends only the **Overview** aspect defined in `OVERVIEW_ASPECT_KEY` to Google Dataplex. When `_md_to_entry` parses the Markdown file, it extracts the YAML front-matter and body content, then `jsonpb.ParseDict` converts this into a Dataplex entry protobuf. The `catalog.update_entry` method receives only these defined aspects, ensuring that local-only extensions or custom front-matter fields not part of the standard aspect schema remain in your local files only.

### Can I keep metadata in the same directory but prevent some files from publishing?

Yes, you can keep draft or sensitive metadata in the same directory by implementing a file-name filter in `snapshot.publish_entries`. Add a conditional check that skips files matching specific patterns, such as `if table_path.name.endswith('.local.md'): continue`, before the code calls `_md_to_entry`. This modification allows you to maintain a single metadata directory while controlling visibility at the individual file level.

### What is the difference between the enrich and publish commands in Knowledge Catalog?

The **`enrich`** command ([`samples/enrichment/src/enrichment/enrich.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/enrich.py)) reads local Markdown files, processes them through enrichment agents (such as metadata extraction or documentation generation), and writes updates back to the same local files without ever contacting Dataplex. The **`publish`** command ([`samples/enrichment/src/enrichment/publish.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/publish.py)) triggers `snapshot.publish_entries`, which converts the local Markdown into Dataplex entry objects and sends them to the Google Cloud catalog via the Dataplex API.

### Where should I modify the code to change which directory gets published?

Modify the **[`samples/enrichment/src/enrichment/publish.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/publish.py)** file if you need to change how the `--dir` argument is processed, or update **[`samples/enrichment/src/enrichment/metadata/snapshot.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/samples/enrichment/src/enrichment/metadata/snapshot.py)** if you need to change the file iteration logic within `publish_entries`. The [`publish.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/publish.py) file simply forwards the directory path to the snapshot module, so implementation of filtering logic belongs in [`snapshot.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/snapshot.py) where the `dir.glob('*.md')` iteration occurs.