# How to Configure Bi-Directional Metadata Sync Between a Local Workspace and the Knowledge Catalog Service

> Learn to configure bi-directional metadata sync between your local workspace and Google Cloud Knowledge Catalog. Use MaC and kcmd pull push for seamless data management.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: how-to-guide
- Published: 2026-07-14

---

**The Knowledge Catalog repository implements Metadata-as-Code (MaC) through the `CatalogSync` class, enabling bi-directional synchronization between your local filesystem and the remote service using `kcmd pull` and `kcmd push` commands.**

Bi-directional metadata sync allows teams to manage Google Cloud Data Catalog resources using version-controlled YAML files while keeping the local workspace and remote service in perfect alignment. The `GoogleCloudPlatform/knowledge-catalog` repository provides the `toolbox/mdcode` package, which orchestrates this two-way flow through a specialized sync engine and command-line interface.

## Understanding the Metadata-as-Code Architecture

The synchronization logic resides in [`toolbox/mdcode/src/libts/sync.ts`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/src/libts/sync.ts), where the **CatalogSync** class manages the bidirectional flow between the **CatalogClient** (GCP service client) and the **CatalogSnapshot** (local filesystem representation).

### Pull Operations

The `CatalogSync.pull` method iterates over entries defined in the snapshot's manifest, fetching each resource from the Knowledge Catalog service via `CatalogClient.lookupEntry`, and persists the results locally using the private `_storeEntry` method. According to the source code in [`toolbox/mdcode/src/libts/sync.ts`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/src/libts/sync.ts) (lines 31-55), this builds a local directory hierarchy that mirrors your remote BigQuery datasets, Pub/Sub topics, and other cataloged assets as YAML files.

### Push Operations

Conversely, `CatalogSync.push` walks the local snapshot using `listEntries` and `_fetchEntry`, compares the state against the remote service, and applies changes with granular update masks. As implemented in [`toolbox/mdcode/src/libts/sync.ts`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/src/libts/sync.ts) (lines 63-118), the system creates missing entries or updates existing ones using specific field masks (`aspects`, `entry_source`, `parent_entry`), reporting any conflicts back to the caller for resolution.

## Prerequisites and Authentication

Both the CLI and TypeScript library require **Application Default Credentials** to communicate with the Knowledge Catalog API.

Run the following command to authenticate:

```bash
gcloud auth application-default login

```

The CLI README at [`toolbox/mdcode/README.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/README.md) (lines 68-70) documents this requirement as mandatory for all sync operations. Ensure your account has appropriate Data Catalog permissions (e.g., `datacatalog.entries.get` and `datacatalog.entries.create`) for the target project.

## Implementing the Bi-Directional Sync Workflow

The `kcmd` CLI tool in [`toolbox/mdcode/src/tool/commands.ts`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/src/tool/commands.ts) (lines 61-69) exposes intuitive commands that wrap the underlying `CatalogSync` methods. Follow this workflow to establish continuous synchronization:

### 1. Initialize the Local Snapshot

Create a metadata manifest for your target resource. For example, to track a BigQuery dataset:

```bash
kcmd init --scope bq-dataset.my-project.sales --entries sales

```

This generates a [`catalog.yaml`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/catalog.yaml) manifest and an empty `catalog/` directory structure in your workspace.

### 2. Pull Remote State

Fetch the current metadata from the Knowledge Catalog service to establish your baseline:

```bash
kcmd pull

```

This command invokes `CatalogSync.pull`, writing each entry as a separate YAML file under `catalog/` with full aspect specifications.

### 3. Modify Local Metadata

Edit the generated YAML files or sidecar markdown documentation. Changes to aspects, descriptions, or business metadata remain purely local until you push.

### 4. Review Changes

Preview the differences between your local workspace and the remote service:

```bash
kcmd status

```

The status command computes a diff without transmitting modifications, allowing you to verify updates before application.

### 5. Push to Remote

Synchronize your local changes back to the Knowledge Catalog service:

```bash
kcmd push

```

The CLI calculates the minimal update mask, creates new entries if missing, and modifies existing resources to match your local definitions.

## Programmatic Synchronization with TypeScript

For CI/CD pipelines or custom tooling, import the library directly to execute sync operations programmatically.

**Pulling metadata via the library:**

```typescript
import * as kcmd from 'kcmd';

// Initialize manifest for BigQuery dataset
const manifest = new kcmd.CatalogManifest({
  scope: 'bq-dataset.my-project.sales',
  snapshot: { entries: ['sales'] },
});
await manifest.save('/tmp/kc-demo');

// Load local snapshot
const snapshot = kcmd.CatalogSnapshot.fromPath('/tmp/kc-demo');

// Execute bi-directional pull
const pullResult = await snapshot.pull();
if (!pullResult.success) {
  console.error('Pull failed:', pullResult.error);
}

```

**Pushing with conflict override:**

```typescript
const pushResult = await snapshot.push({ force: true });
if (!pushResult.success) {
  console.error('Push error:', pushResult.error);
}

```

The `force` option bypasses certain conflict checks, useful when your local workspace must become the authoritative source of truth.

## Summary

- **Metadata-as-Code (MaC)** enables treating catalog definitions as version-controlled artifacts in [`toolbox/mdcode/src/libts/sync.ts`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/src/libts/sync.ts).
- **Bi-directional sync** consists of `pull` (remote-to-local) and `push` (local-to-remote) operations orchestrated by the `CatalogSync` class.
- **CLI workflow** follows `init` → `pull` → `edit` → `status` → `push`, with `--dry-run` flags available for safe previews.
- **Authentication** requires `gcloud auth application-default login` for both CLI and library usage.
- **Conflict handling** occurs during `CatalogSync.push`, utilizing update masks to apply granular changes without overwriting entire entries.

## Frequently Asked Questions

### What is Metadata-as-Code (MaC) in the Knowledge Catalog context?

Metadata-as-Code is the practice of defining Data Catalog entries, aspects, and metadata as YAML files stored in version control. According to the `toolbox/mdcode` specification in [`docs/spec.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/docs/spec.md), this approach enables GitOps workflows, code reviews for metadata changes, and reproducible deployments across environments while maintaining synchronization with the live Knowledge Catalog service.

### How does the sync handle conflicts between local and remote changes?

The `CatalogSync.push` method implements optimistic concurrency control. When pushing modifications from [`toolbox/mdcode/src/libts/sync.ts`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/src/libts/sync.ts) (lines 63-118), the system compares local checksums against remote state. If changes are detected server-side that conflict with local edits, the operation reports specific conflicts to the caller without overwriting data. Use `kcmd status` to identify these discrepancies before pushing, or pass `{ force: true }` in the library to override remote changes with local definitions.

### Can I preview changes before pushing them to the Knowledge Catalog service?

Yes. The `kcmd` CLI supports dry-run capabilities via flags (`--dry-run`) that simulate the push operation without modifying the service. Additionally, the `kcmd status` command displays the computed diff between your local `catalog/` directory and the remote entries, showing exactly which aspects and fields will be created, updated, or deleted during the next push operation.

### Which authentication methods does the bi-directional sync support?

The system exclusively uses **Application Default Credentials** (ADC) as documented in [`toolbox/mdcode/README.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/toolbox/mdcode/README.md) (lines 68-70). Configure ADC by running `gcloud auth application-default login` or by setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to point to a service account key file. Both the `kcmd` CLI and the TypeScript `CatalogSync` class automatically retrieve credentials from the ADC chain when initializing the `CatalogClient`.