Entry File Naming Conventions and Hierarchical Directory Layout in Knowledge Catalog Metadata as Code

Knowledge Catalog implements a Metadata-as-Code model that maps entry identifiers directly to file system paths, using .yaml extensions for structured BigQuery resources and .md extensions for knowledge-base articles with YAML front-matter.

The Knowledge Catalog open-source repository stores catalog metadata as ordinary source-code artifacts, enabling version control and CI/CD pipelines. This system relies on deterministic entry file naming conventions and a hierarchical directory layout that mirrors Google Cloud resource hierarchies such as BigQuery datasets and tables or Dataplex entry groups.

Root of a Metadata Snapshot

Every metadata snapshot begins with a root directory containing a catalog descriptor (catalog.yaml) and entry files organized in subdirectories. The descriptor holds global processing directives and optional alias definitions that apply across the entire catalog.

According to the specification in toolbox/mdcode/docs/spec.md (lines 165-190), this root structure serves as the anchor point for all layout operations.

Entry File Naming Conventions

The naming convention depends on the scope of the metadata being stored. The library distinguishes between structured data resources and unstructured knowledge-base content through two distinct layout implementations.

Standard Layout for Structured Data

For BigQuery datasets, tables, views, and Dataplex entry groups, the system uses the Standard Layout with .yaml extensions. The filename derives directly from the entry identifier (entryId), where slashes in the fully-qualified resource name become underscores or sanitized characters to form a valid filename.

In toolbox/mdcode/src/libts/layouts/standard.ts (lines 31-42), the entryPath method implements this mapping:

// Example: entryId "projects/my-proj/datasets/sales/tables/orders"
// becomes "catalog/projects/my-proj/datasets/sales/tables/orders.yaml"
const entryPath = layout.entryPath(snapshotRoot, entry.id);

The entry ID serves as both the identifier and the path template, ensuring that projects/my-proj/datasets/sales/tables/orders resolves to catalog/projects/my-proj/datasets/sales/tables/orders.yaml.

Documents Layout for Knowledge Base Articles

For knowledge-base (kb) scopes, the Documents Layout stores entries as .md files. The entry ID maps to a markdown filename, with structured metadata stored in YAML front-matter and unstructured content in the markdown body.

As implemented in toolbox/mdcode/src/libts/layouts/documents.ts (lines 196-210), this layout handles the overview.content aspect by separating structured fields (id, title, tags) from the primary unstructured document content.

Hierarchical Directory Layout

The directory tree mirrors the resource hierarchy exactly. When pulling metadata via the MCP (Metadata Catalog Program), the library automatically creates nested subdirectories that reflect the BigQuery or Dataplex hierarchy.

A typical snapshot for a BigQuery dataset follows this structure:

catalog/
├── catalog.yaml                           # Global descriptor

├── projects/
│   └── my-proj/
│       └── datasets/
│           └── my_dataset/
│               ├── tables/
│               │   ├── my_table.yaml       # Entry metadata for a table

│               │   └── another_table.yaml
│               └── views/
│                   └── my_view.yaml

This layout is defined in toolbox/mdcode/docs/design.md (lines 75-82), which states that structured data is stored in a main YAML file per entry, while unstructured rich-text fields may use dedicated side-car Markdown files in the Documents layout.

Working with Entry Files Programmatically

The layout abstraction enforces naming conventions through the Layout interface. When writing entries, the library validates that the computed file path matches the entry ID and throws an error on mismatch, implementing fail-fast behavior.

The generic validation logic resides in toolbox/mdcode/src/libts/layout.ts (lines 1-10), while concrete implementations handle path resolution:

import { Layout } from '@googlecloud/mdcode';

const snapshotPath = '/path/to/catalog';
const layout = new Layout.Standard(); // or Layout.Documents for kb scopes

// Resolve file path from entry ID
const entryId = 'projects/my-proj/datasets/sales/tables/orders';
const entryPath = layout.entryPath(snapshotPath, entryId);
// Returns: '/path/to/catalog/projects/my-proj/datasets/sales/tables/orders.yaml'

// Read entry metadata
const entry = await layout.readEntry(snapshotPath, entryId);

// Update and write back
entry.resource.description = 'Updated description';
await layout.writeEntry(snapshotPath, entryId, entry);

Example: BigQuery Table Entry

A table entry file at catalog/projects/my-proj/datasets/sales/tables/orders.yaml contains:

id: projects/my-proj/datasets/sales/tables/orders
resource:
  displayName: orders
  description: Orders table for the sales dataset
  type: bq-table
schema:
  columns:
    - name: order_id
      type: STRING
    - name: order_date
      type: TIMESTAMP

The filename orders.yaml corresponds to the last component of the entry ID, with the directory structure encoding the full resource path.

Example: Knowledge Base Article

For knowledge-base content using the Documents layout, a file at catalog/articles/welcome.md contains:

---
id: articles/welcome
title: Welcome to the Data Platform
description: Overview of the data platform and best‑practice guides.
tags:
  - onboarding
  - overview
catalogEntry:
  type: kb-article
  resource:
    displayName: Welcome
---

# Welcome

This article introduces new users to the platform...

Here the markdown file itself serves as the entry artifact, with front-matter supplying structured metadata compatible with the catalog schema.

Summary

• Entry files are named after their entry ID, with slashes converted to directory separators or underscores depending on the layout type.
• The Standard Layout uses .yaml files for BigQuery tables, datasets, and Dataplex entries, storing purely structured metadata.
• The Documents Layout uses .md files for knowledge-base articles, combining YAML front-matter with markdown content.
• The directory hierarchy mirrors the Google Cloud resource hierarchy (projects → datasets → tables), created automatically during mcp pull operations.
• The Layout API in standard.ts and documents.ts enforces deterministic path resolution and validates naming conventions at write-time.

Frequently Asked Questions

How does the Standard layout differ from the Documents layout?

The Standard layout stores structured metadata in standalone .yaml files suitable for BigQuery and Dataplex resources, while the Documents layout stores knowledge-base articles as .md files with YAML front-matter. As implemented in documents.ts (lines 196-210), the Documents layout treats the markdown body itself as the unstructured aspect content, whereas Standard layout keeps all metadata in YAML structure.

What determines the file extension for an entry file?

The scope of the snapshot determines the extension. BigQuery dataset and table entries use the Standard layout and generate .yaml files, while knowledge-base (kb) scope entries use the Documents layout and generate .md files. The layout.ts interface abstracts these differences so that calling code does not need to handle extensions manually.

How does the MCP tool create directory hierarchies when pulling metadata?

When executing mcp pull, the tool invokes layout.writeEntry which computes the full path by decomposing the entry ID into directory components. For a BigQuery table with ID projects/my-proj/datasets/sales/tables/orders, the library automatically creates the nested directory structure projects/my-proj/datasets/sales/tables/ under the snapshot root before writing orders.yaml, as shown in standard.ts (lines 31-42).

Can entry files be manually edited outside the MCP tool?

Yes. Because Knowledge Catalog uses a Metadata-as-Code approach, developers can edit .yaml and .md files directly using any text editor or IDE. The layout.readEntry method in standard.ts parses these files back into the internal entry representation, provided the file paths adhere to the naming convention where the filename (minus extension) matches the final component of the entry ID.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}