How Entry Sidecar Markdown Files Work in kcmd: A Complete Guide to Unstructured Aspect Storage

kcmd uses entry sidecar markdown files to store unstructured text aspects (like overview and guidelines) separately from structured entry metadata, automatically mapping them to entry fields during pull and push operations within the GoogleCloudPlatform/knowledge-catalog repository.

The kcmd CLI tool manages Knowledge Catalog entries using a hybrid storage approach that separates structured machine-readable data from human-readable documentation. While entry schemas and profiles live in YAML files, entry sidecar markdown files handle rich-text content, enabling data stewards to edit documentation using standard Markdown editors while maintaining a coherent logical object model.

Understanding the Two Storage Layouts

kcmd supports two complementary disk layouts depending on the catalog scope, both of which leverage sidecar files for unstructured content.

Standard Layout (YAML + Markdown Sidecars)

The Standard layout applies to bq-dataset and entryGroup scopes. In this configuration, structured entry data resides in a primary YAML file named <entry-id>.yaml, while each unstructured aspect gets its own sidecar file following the pattern <entry-id>.<aspect>.md.

For example, an entry named orders would have:

• orders.yaml — Contains structured fields like schema definitions and profiling metadata
• orders.overview.md — Contains the overview aspect as plain Markdown
• orders.guidelines.md — Contains the guidelines aspect as plain Markdown

According to the specification in toolbox/mdcode/docs/spec.md, this layout "stores metadata structure in a YAML file per entry, and unstructured aspects are placed in sidecar Markdown files."

Documents Layout (Single Markdown File)

The Documents layout is used for kb (knowledge-base) scopes. Here, all metadata is encoded in the YAML front-matter of a single <entry-id>.md file, with the primary unstructured aspect (overview.content) stored as the Markdown body. Additional aspects can still reside in separate sidecar files if needed, maintaining the same naming convention as the Standard layout.

As illustrated in toolbox/mdcode/docs/concept.md, this layout allows an entire entry to be represented as a single document while optionally linking to supplementary sidecar files for specific aspects.

How kcmd Discovers and Processes Sidecar Files

The sidecar discovery mechanism is implemented in toolbox/mdcode/src/libts/layouts/documents.ts, which uses a glob pattern of **/*.md to locate all Markdown files within the catalog root. When kcmd pull or push executes, the CatalogSnapshot class performs the following operations:

1. Reads the main YAML file (or extracts front-matter from a single Markdown file in Documents layout) to build the base entry record
2. Detects any sibling *.<aspect>.md files that match known unstructured aspect names
3. Loads the file content and assigns it to the corresponding Entry field (e.g., entry.overview = fileContent)

During a push operation, the inverse process occurs: any modified aspect fields are written back to their respective sidecar files on disk, preserving the one-file-per-aspect convention. This bidirectional synchronization ensures that changes made programmatically via the API or manually via text editors remain consistent.

Practical Examples: Reading and Writing Sidecar Files

The following TypeScript examples demonstrate how to interact with entry sidecar markdown files using the kcmd library.

Reading a snapshot and accessing sidecar content:

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const snapshot = await kcmd.CatalogSnapshot.fromPath('/path/to/catalog', ctx);

// snapshot.entries now contains `overview` populated from
// `orders.overview.md` (if it exists)
const entry = snapshot.getEntry('orders');
console.log(entry.overview); // Markdown content from sidecar file

Updating an aspect and persisting changes:

import * as kcmd from 'kcmd';

const ctx = kcmd.gcp.ApiContext.default();
const snap = await kcmd.CatalogSnapshot.fromPath('.', ctx);
const entry = snap.getEntry('orders');

// Update the overview aspect
entry.overview = '# Updated overview\nNew description here.';

// Writes back to `orders.overview.md`
await snap.save();

// Syncs changes to GCP Knowledge Catalog
await new kcmd.CatalogSync(catalog, snap).push();

Key Implementation Files

File	Role
toolbox/mdcode/docs/spec.md	Defines the Standard and Documents layouts, specifying that unstructured aspects belong in sidecar Markdown files
toolbox/mdcode/docs/concept.md	Provides visual illustrations of directory structures showing sidecar files like orders.overview.md alongside orders.yaml
toolbox/mdcode/src/libts/layouts/documents.ts	Implements the glob-based discovery (**/*.md) and mapping of sidecar files to entry aspects
toolbox/mdcode/src/tool/commands.ts	Contains CLI commands (pull, push) that trigger CatalogSnapshot creation and sidecar handling
toolbox/mdcode/src/tool/mcp.ts	Uses CatalogSnapshot for MCP server operations, reading sidecar content into the entry model

Summary

• Entry sidecar markdown files follow the naming convention <entry-id>.<aspect>.md and store unstructured text aspects separately from structured YAML metadata
• The Standard layout (bq-dataset and entryGroup scopes) uses one YAML file per entry alongside multiple sidecar files for each unstructured aspect
• The Documents layout (kb scope) stores metadata in YAML front-matter with the primary content as the Markdown body, optionally supporting additional sidecar files
• CatalogSnapshot logic in src/libts/layouts/documents.ts automatically discovers sidecar files using glob patterns and maps them to entry fields during pull and push operations
• Changes to sidecar files are bidirectional: manual edits are read during pulls, and programmatic updates are written back to disk during saves

Frequently Asked Questions

What is the naming convention for entry sidecar markdown files in kcmd?

Sidecar files must follow the pattern <entry-id>.<aspect>.md, where <entry-id> matches the base name of the corresponding YAML file, and <aspect> is the specific unstructured aspect name (e.g., overview, guidelines). For an entry defined in orders.yaml, the overview sidecar would be named orders.overview.md.

How does kcmd handle sidecar files during a push operation?

During a push, kcmd calls CatalogSnapshot.save() to write any modified aspect fields back to their respective sidecar files on disk, maintaining the one-file-per-aspect structure. The CatalogSync class then compares these local files with the remote Knowledge Catalog state and uploads only the changed content.

Can an entry have multiple sidecar markdown files?

Yes. An entry can have one sidecar file for each unstructured aspect defined in its schema. For example, a dataset entry might have orders.overview.md for general description, orders.guidelines.md for usage instructions, and orders.access.md for access policies, all coexisting alongside the single orders.yaml structured metadata file.

What is the difference between the Standard layout and Documents layout for sidecar files?

The Standard layout separates structured data (YAML) from unstructured content (Markdown sidecars), while the Documents layout combines them into a single Markdown file with YAML front-matter. The Standard layout is used for bq-dataset and entryGroup scopes, whereas the Documents layout is specific to kb (knowledge-base) scopes where entries are treated as documents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}