ConceptRef Structure and Utilization Across Data Sources in Knowledge Catalog

ConceptRef is a frozen dataclass that serves as the immutable, hashable identifier for knowledge concepts, enabling uniform reference across BigQuery tables, Markdown bundles, and enrichment pipelines in the Knowledge Catalog.

The ConceptRef class forms the backbone of the GoogleCloudPlatform/knowledge-catalog's abstraction layer, allowing diverse data sources to expose their metadata through a consistent interface. As the canonical bridge between platform-specific resources and the catalog's processing tools, understanding this structure is essential for extending the system with new sources or debugging enrichment workflows.

ConceptRef Data Structure

The canonical definition lives in okf/src/reference_agent/sources/base.py, where the class is declared as a frozen dataclass to guarantee immutability and hashability.

@dataclass(frozen=True)
class ConceptRef:
    id: tuple[str, ...]          # hierarchical identifier, e.g. ("tables", "users")

    type: str                    # type of the concept, e.g. "BigQuery Table"

    resource: str | None = None # optional resource URI (e.g. BigQuery table ID)

    hint: dict[str, Any] = field(default_factory=dict)  # free‑form hints

This design produces hashable instances that can safely serve as dictionary keys or set elements, which is critical for graph construction and deduplication operations.

Core Field Definitions

• id: A tuple composing a slash-separated path that uniquely identifies the concept within a bundle. For example, ("tables", "users") represents the logical path tables/users.

• type: The logical category used by the catalog for classification, such as "BigQuery Table" or "Cloud Storage Object".

• resource: An optional canonical reference to the underlying platform resource, typically the fully-qualified resource name like "my-project.dataset.users".

• hint: A flexible dictionary storing additional metadata such as preview URLs or lineage hints that don't fit the structured schema.

Production Across Data Sources

Every data source implementation emits ConceptRef objects through a standardized interface, ensuring downstream tools receive uniformly shaped identifiers regardless of the underlying platform.

Source Implementations

Concrete Source subclasses enumerate native objects and return ConceptRef instances via the list_concepts() method. In okf/src/reference_agent/sources/bigquery.py, the BigQuery source constructs a ConceptRef for every table and sharded table family, populating the resource field with the fully-qualified table ID and setting the type to "BigQuery Table".

Markdown Bundle Generation

When scanning documentation bundles, the generator in okf/src/reference_agent/viewer/generator.py instantiates ConceptRef objects by parsing front-matter from Markdown files. The generator extracts the id, type, and optional resource fields from YAML front-matter, then constructs the frozen dataclass to represent the document as a catalog concept.

Consumption Patterns

Downstream components consume ConceptRef instances to perform enrichment, serialization, and visualization without needing platform-specific logic.

Tooling Layer Translation

Helper functions translate ConceptRef instances into plain dictionaries for CLI output or API payloads. The source_tools.list_concepts() function in okf/src/reference_agent/tools/source_tools.py calls the source's list_concepts() method and runs _ref_to_dict() to expose id, type, resource, and hint as serializable dictionaries.

For bundle operations, bundle_tools.read_existing_doc() and bundle_tools.write_concept_doc() in okf/src/reference_agent/tools/bundle_tools.py accept slash-joined string identifiers, convert them using parse_concept_id(), and map them back to proper ConceptRef instances before file I/O operations.

Enrichment Pipeline Integration

The enrichment runner processes concepts by receiving ConceptRef instances and retrieving underlying metadata via src.read_concept(ref). The enrich_concept method in okf/src/reference_agent/runner.py uses the reference to fetch platform-specific details, apply transformations, and persist results while maintaining the immutable reference as the correlation key.

Graph Construction for Visualization

The viewer builds a directed graph where nodes represent concepts. Each ConceptRef instance generates a graph node via the to_node() method. The _build_graph implementation in okf/src/reference_agent/viewer/generator.py uses the id tuple for node keys and the type field for styling, leveraging the hashability of frozen ConceptRef instances to ensure unique node identity.

Cross-Source Consistency

All data source implementations—including bigquery.py and future sources such as Cloud Storage or Pub/Sub—share the same public interface defined in base.py. By returning ConceptRef objects from list_concepts(), they guarantee that downstream enrichment tools, CLI utilities, and visualization engines can treat every concept uniformly, regardless of whether the underlying resource is a database table or a documentation file.

Summary

• ConceptRef is a frozen dataclass defined in okf/src/reference_agent/sources/base.py with fields for id, type, resource, and hint.
• Hashability allows instances to serve as dictionary keys and graph nodes throughout the pipeline.
• BigQuery source produces references via list_concepts() in okf/src/reference_agent/sources/bigquery.py, mapping tables to canonical identifiers.
• Markdown bundles generate references in okf/src/reference_agent/viewer/generator.py by parsing front-matter YAML.
• Tooling layer converts references to dictionaries in source_tools.py and handles string parsing in bundle_tools.py.
• Enrichment runner consumes references in okf/src/reference_agent/runner.py to drive the enrich_concept workflow.

Frequently Asked Questions

What makes ConceptRef immutable and why does it matter?

The @dataclass(frozen=True) decorator in okf/src/reference_agent/sources/base.py prevents field modification after instantiation. This immutability guarantees hashability, allowing ConceptRef instances to be used as keys in dictionaries and elements in sets, which is essential for graph deduplication and caching layers.

How does the Knowledge Catalog handle different resource types with the same ConceptRef structure?

The type field provides logical classification while the resource field holds platform-specific identifiers. BigQuery tables, Cloud Storage objects, and Markdown documents all use the same ConceptRef structure but populate these fields differently, enabling the enrichment runner in okf/src/reference_agent/runner.py to process them through a unified interface.

Can I extend ConceptRef with custom metadata for my data source?

Yes, the hint dictionary accepts free-form key-value pairs without breaking the frozen dataclass contract. Custom source implementations can store additional metadata—such as preview URLs or lineage hints—in this field, which tools like _ref_to_dict() in source_tools.py will serialize alongside standard fields.

How is the hierarchical id converted between tuple and string formats?

The tooling layer handles bidirectional conversion: parse_concept_id() in okf/src/reference_agent/tools/bundle_tools.py splits slash-separated strings into tuples for ConceptRef construction, while the id field remains a tuple internally to ensure immutability and proper hashing for graph operations in okf/src/reference_agent/viewer/generator.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}