What Is CatalogSnapshot and How Does It Manage Local Catalog Entries?

A CatalogSnapshot is the core abstraction that loads a Dataplex catalog manifest locally, builds in-memory maps of entry and aspect types, and delegates file-system operations to a layout implementation that manages how entries are stored on disk.

The CatalogSnapshot class serves as the bridge between remote Dataplex metadata and the local file-based catalog used by the GoogleCloudPlatform/knowledge-catalog toolbox. It ensures that only entry types and aspects declared in the catalog.yaml manifest are available, while respecting whether the catalog is ingested (read-only) or user-managed (editable).

Understanding the CatalogSnapshot Architecture

In toolbox/mdcode/src/libts/snapshot.ts, the CatalogSnapshot class encapsulates the entire lifecycle of a local catalog instance. Unlike direct API calls to Dataplex, the snapshot maintains a local copy of the catalog manifest and metadata definitions, providing offline-capable operations with synchronization capabilities.

The architecture centers on three pillars: manifest loading, type resolution, and layout abstraction. When you initialize a snapshot via CatalogSnapshot.fromPath(), it reads the catalog.yaml file, fetches type definitions from the Dataplex API, and selects a concrete CatalogLayout implementation based on the manifest's source configuration.

Core Responsibilities of CatalogSnapshot

Loading the Manifest

The entry point CatalogSnapshot.fromPath() (lines 32-38 in snapshot.ts) reads the catalog.yaml file from the specified directory and instantiates a CatalogManifest object. This manifest defines the catalog's structure, including whether it uses standard YAML files or document-centric storage.

Building Type Maps

After loading the manifest, the snapshot calls _buildTypes() (lines 35-76) to query the Dataplex API for each entry-type and aspect-type listed in manifest.snapshotConfig. The results are cached as Map<string, EntryType> and Map<string, AspectType> on the snapshot instance, ensuring type validation can occur locally without additional API calls.

Selecting the Layout

Based on manifest.source.layout, the snapshot uses createLayout() (lines 28-30) to obtain a CatalogLayout implementation. As defined in toolbox/mdcode/src/libts/layout.ts (lines 25-33), this layout determines whether entries are stored as separate YAML files (StandardLayout in layouts/standard.ts) or as document collections (DocumentsLayout in layouts/documents.ts).

CRUD Operations

The snapshot delegates all file-system operations to the layout while enforcing manifest rules:

• List entries: listEntries() (lines 54-62) returns all entry names from the local catalog/ directory
• Read entries: lookupEntry() retrieves specific entries via the layout
• Create/Update/Delete: createEntry(), updateEntry(), and deleteEntry() (lines 9-33, 64-84) manipulate on-disk representations while respecting "user-managed vs. ingested" rules
• Sync conversion: Private helpers _storeEntry() and _fetchEntry() (lines 78-108, 86-108) translate between the Dataplex service model and the local metadata model (md.Entry)

Working with CatalogSnapshot

The following examples demonstrate how to load a catalog snapshot and perform operations on local entries using the implementation from toolbox/mdcode/src/libts/snapshot.ts.

Loading a Snapshot from Local Directory

import { CatalogSnapshot } from './snapshot';
import * as gcp from './gcp/context';

async function openCatalog(dir: string) {
  const ctx = await gcp.createApiContext();   // creates an authenticated GCP client
  const snapshot = await CatalogSnapshot.fromPath(dir, ctx);
  return snapshot;
}

Listing and Reading Entries

// List all locally stored entries
async function listAll(dir: string) {
  const snap = await openCatalog(dir);
  const names = await snap.listEntries();   // e.g. ['mydataset.mytable', 'myview']
  console.log(names);
}

// Read a specific entry
async function readEntry(dir: string, name: string) {
  const snap = await openCatalog(dir);
  const entry = await snap.lookupEntry(name);
  console.log(entry);
}

Updating Local Entries

// Update the description of an entry's resource metadata
async function updateDescription(dir: string, name: string, newDesc: string) {
  const snap = await openCatalog(dir);
  const entry = await snap.lookupEntry(name);
  entry.resource = entry.resource || {};
  entry.resource.description = newDesc;
  await snap.updateEntry(entry, ['resource']);
}

Summary

• CatalogSnapshot acts as the local bridge between Dataplex cloud metadata and the file-based catalog in the GoogleCloudPlatform/knowledge-catalog repository.
• The class loads catalog.yaml via fromPath() and caches type definitions from the Dataplex API to enable offline validation.
• File-system operations are delegated to a CatalogLayout implementation (either StandardLayout or DocumentsLayout) selected based on the manifest configuration.
• CRUD operations respect the catalog's mode (ingested vs. user-managed) as defined in the manifest, preventing unauthorized modifications to read-only catalogs.
• Internal translation methods _storeEntry() and _fetchEntry() handle conversion between the Dataplex service model and the local md.Entry format.

Frequently Asked Questions

What is the difference between ingested and user-managed catalogs?

Ingested catalogs are read-only local copies of Dataplex metadata that cannot be modified through the snapshot, while user-managed catalogs allow local edits that can be pushed back to Dataplex. The CatalogSnapshot enforces these rules based on the manifest configuration, preventing accidental writes to ingested entries.

How does CatalogSnapshot handle entry type validation?

The snapshot validates entry types against cached Map<string, EntryType> and Map<string, AspectType> objects built during initialization. These maps are populated by querying the Dataplex API for each type listed in manifest.snapshotConfig, ensuring that only declared types can be created or modified locally.

What file formats does CatalogSnapshot support?

According to the source code in toolbox/mdcode/src/libts/layouts/, the snapshot supports two storage formats: StandardLayout stores each entry as a separate YAML file under the catalog/ directory, while DocumentsLayout stores entries as document collections for document-centric catalogs. The specific layout is determined by the manifest.source.layout configuration.

How does CatalogSnapshot sync with Dataplex?

The snapshot uses private helper methods _storeEntry() and _fetchEntry() to translate between the Dataplex service model and the local md.Entry metadata model. While the snapshot maintains local copies for offline work, these conversion methods ensure compatibility when synchronizing changes back to the Dataplex API or fetching updates from remote sources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}