# ConceptRef Structure and Utilization Across Data Sources in Knowledge Catalog

> Explore ConceptRef structure and utilization in Knowledge Catalog. Learn how this immutable identifier ensures uniform reference across BigQuery tables, Markdown bundles, and enrichment pipelines.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-16

---

**`ConceptRef` is a frozen dataclass that serves as the immutable, hashable identifier for knowledge concepts, enabling uniform reference across BigQuery tables, Markdown bundles, and enrichment pipelines in the Knowledge Catalog.**

The `ConceptRef` class forms the backbone of the GoogleCloudPlatform/knowledge-catalog's abstraction layer, allowing diverse data sources to expose their metadata through a consistent interface. As the canonical bridge between platform-specific resources and the catalog's processing tools, understanding this structure is essential for extending the system with new sources or debugging enrichment workflows.

## ConceptRef Data Structure

The canonical definition lives in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py), where the class is declared as a frozen dataclass to guarantee immutability and hashability.

```python
@dataclass(frozen=True)
class ConceptRef:
    id: tuple[str, ...]          # hierarchical identifier, e.g. ("tables", "users")

    type: str                    # type of the concept, e.g. "BigQuery Table"

    resource: str | None = None # optional resource URI (e.g. BigQuery table ID)

    hint: dict[str, Any] = field(default_factory=dict)  # free‑form hints

```

This design produces hashable instances that can safely serve as dictionary keys or set elements, which is critical for graph construction and deduplication operations.

### Core Field Definitions

- **`id`**: A tuple composing a slash-separated path that uniquely identifies the concept within a bundle. For example, `("tables", "users")` represents the logical path `tables/users`.

- **`type`**: The logical category used by the catalog for classification, such as `"BigQuery Table"` or `"Cloud Storage Object"`.

- **`resource`**: An optional canonical reference to the underlying platform resource, typically the fully-qualified resource name like `"my-project.dataset.users"`.

- **`hint`**: A flexible dictionary storing additional metadata such as preview URLs or lineage hints that don't fit the structured schema.

## Production Across Data Sources

Every data source implementation emits `ConceptRef` objects through a standardized interface, ensuring downstream tools receive uniformly shaped identifiers regardless of the underlying platform.

### Source Implementations

Concrete `Source` subclasses enumerate native objects and return `ConceptRef` instances via the `list_concepts()` method. In [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), the BigQuery source constructs a `ConceptRef` for every table and sharded table family, populating the `resource` field with the fully-qualified table ID and setting the `type` to `"BigQuery Table"`.

### Markdown Bundle Generation

When scanning documentation bundles, the generator in [`okf/src/reference_agent/viewer/generator.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/viewer/generator.py) instantiates `ConceptRef` objects by parsing front-matter from Markdown files. The generator extracts the `id`, `type`, and optional `resource` fields from YAML front-matter, then constructs the frozen dataclass to represent the document as a catalog concept.

## Consumption Patterns

Downstream components consume `ConceptRef` instances to perform enrichment, serialization, and visualization without needing platform-specific logic.

### Tooling Layer Translation

Helper functions translate `ConceptRef` instances into plain dictionaries for CLI output or API payloads. The `source_tools.list_concepts()` function in [`okf/src/reference_agent/tools/source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/source_tools.py) calls the source's `list_concepts()` method and runs `_ref_to_dict()` to expose `id`, `type`, `resource`, and `hint` as serializable dictionaries.

For bundle operations, `bundle_tools.read_existing_doc()` and `bundle_tools.write_concept_doc()` in [`okf/src/reference_agent/tools/bundle_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/bundle_tools.py) accept slash-joined string identifiers, convert them using `parse_concept_id()`, and map them back to proper `ConceptRef` instances before file I/O operations.

### Enrichment Pipeline Integration

The enrichment runner processes concepts by receiving `ConceptRef` instances and retrieving underlying metadata via `src.read_concept(ref)`. The `enrich_concept` method in [`okf/src/reference_agent/runner.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/runner.py) uses the reference to fetch platform-specific details, apply transformations, and persist results while maintaining the immutable reference as the correlation key.

### Graph Construction for Visualization

The viewer builds a directed graph where nodes represent concepts. Each `ConceptRef` instance generates a graph node via the `to_node()` method. The `_build_graph` implementation in [`okf/src/reference_agent/viewer/generator.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/viewer/generator.py) uses the `id` tuple for node keys and the `type` field for styling, leveraging the hashability of frozen `ConceptRef` instances to ensure unique node identity.

## Cross-Source Consistency

All data source implementations—including [`bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/bigquery.py) and future sources such as Cloud Storage or Pub/Sub—share the same public interface defined in [`base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/base.py). By returning `ConceptRef` objects from `list_concepts()`, they guarantee that downstream enrichment tools, CLI utilities, and visualization engines can treat every concept uniformly, regardless of whether the underlying resource is a database table or a documentation file.

## Summary

- **`ConceptRef`** is a frozen dataclass defined in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py) with fields for `id`, `type`, `resource`, and `hint`.
- **Hashability** allows instances to serve as dictionary keys and graph nodes throughout the pipeline.
- **BigQuery source** produces references via `list_concepts()` in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), mapping tables to canonical identifiers.
- **Markdown bundles** generate references in [`okf/src/reference_agent/viewer/generator.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/viewer/generator.py) by parsing front-matter YAML.
- **Tooling layer** converts references to dictionaries in [`source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/source_tools.py) and handles string parsing in [`bundle_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/bundle_tools.py).
- **Enrichment runner** consumes references in [`okf/src/reference_agent/runner.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/runner.py) to drive the `enrich_concept` workflow.

## Frequently Asked Questions

### What makes ConceptRef immutable and why does it matter?

The `@dataclass(frozen=True)` decorator in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py) prevents field modification after instantiation. This immutability guarantees hashability, allowing `ConceptRef` instances to be used as keys in dictionaries and elements in sets, which is essential for graph deduplication and caching layers.

### How does the Knowledge Catalog handle different resource types with the same ConceptRef structure?

The `type` field provides logical classification while the `resource` field holds platform-specific identifiers. BigQuery tables, Cloud Storage objects, and Markdown documents all use the same `ConceptRef` structure but populate these fields differently, enabling the enrichment runner in [`okf/src/reference_agent/runner.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/runner.py) to process them through a unified interface.

### Can I extend ConceptRef with custom metadata for my data source?

Yes, the `hint` dictionary accepts free-form key-value pairs without breaking the frozen dataclass contract. Custom source implementations can store additional metadata—such as preview URLs or lineage hints—in this field, which tools like `_ref_to_dict()` in [`source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/source_tools.py) will serialize alongside standard fields.

### How is the hierarchical id converted between tuple and string formats?

The tooling layer handles bidirectional conversion: `parse_concept_id()` in [`okf/src/reference_agent/tools/bundle_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/bundle_tools.py) splits slash-separated strings into tuples for `ConceptRef` construction, while the `id` field remains a tuple internally to ensure immutability and proper hashing for graph operations in [`okf/src/reference_agent/viewer/generator.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/viewer/generator.py).