# What Is the Purpose of the Hint Dictionary in ConceptRef Objects?

> Discover the purpose of the hint dictionary in ConceptRef objects. This cache stores auxiliary metadata for efficient data disambiguation and contextual information transport.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-16

---

**The `hint` dictionary in a `ConceptRef` object serves as a lightweight, source-specific cache that stores auxiliary metadata for efficient disambiguation—such as distinguishing between wildcard sharded table families and concrete tables—while transporting contextual information that downstream tools can access without additional API calls.**

In the `GoogleCloudPlatform/knowledge-catalog` repository, `ConceptRef` objects act as lightweight identifiers returned by data sources like BigQuery to represent datasets, tables, and other entities. While these references contain mandatory fields such as `id` and `type`, they also include an optional **`hint` dictionary** that carries critical source-specific metadata. This compact data structure enables sources to optimize resource lookup and provides CLI and UI tools with immediate context that would otherwise require redundant API requests.

## Understanding ConceptRef and the Hint Dictionary

`ConceptRef` objects are defined in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py) as the standard interface for referencing concepts within the knowledge catalog. Each object contains:

- **`id`**: A unique identifier for the concept
- **`type`**: The classification of the concept (e.g., dataset, table)
- **`resource`**: (Optional) A URI or resource pointer
- **`hint`**: (Optional) A dictionary containing source-specific auxiliary metadata

The **hint dictionary** functions as a transport mechanism for data that the source deems necessary for efficient downstream operations. Unlike the core reference fields, hints are source-defined and can vary based on the specific implementation requirements of each connector.

## Two Primary Purposes of the ConceptRef Hint Dictionary

The hint dictionary serves two architectural functions within the knowledge catalog system: disambiguating complex resource patterns and caching contextual metadata for downstream consumers.

### Disambiguating Wildcard Families and Concrete Tables

For BigQuery implementations, the hint dictionary resolves ambiguity between **wildcard families** of sharded tables and single concrete tables. In [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), the source populates the hint with a `wildcard` boolean flag that determines how the system should interpret the reference.

When `wildcard` is `true`, the hint contains metadata about the sharded family:

```json
{
  "wildcard": true,
  "family_prefix": "events_",
  "shard_count": 3,
  "first_shard": "events_20210101",
  "last_shard": "events_20210103"
}

```

The `BigQuerySource` uses this information in the `_representative_table_id` method to select a representative table (typically the most recent shard) without enumerating all shards again. For concrete tables, the hint simply marks `wildcard` as `false` and includes the specific `table_id`:

```json
{ "wildcard": false, "table_id": "users" }

```

### Transporting Source-Specific Context

Beyond disambiguation, the hint dictionary acts as a **compact metadata cache** accessible via `source_tools._ref_to_dict` in [`okf/src/reference_agent/tools/source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/source_tools.py). This allows CLI and UI components to display rich information without invoking `read_concept`.

BigQuery stores additional context such as:

- `dataset_project` and `dataset_id`: Ownership information for the table
- `family_prefix`, `shard_count`, `first_shard`, `last_shard`: Detailed sharding metadata

Custom sources can extend this pattern by adding arbitrary key-value pairs—such as version numbers or ownership tags—that downstream agents require for decision-making.

## How to Access and Use the Hint Dictionary in Practice

The `source_tools` module exposes hint data through conversion functions that serialize `ConceptRef` objects into dictionaries. This enables inspection and utilization of hint metadata in agent workflows.

To list all concepts and inspect their hints:

```python
from reference_agent.tools.source_tools import list_concepts

for concept in list_concepts():
    print(f"{concept['id']} – type: {concept['type']}")
    print("  hint →", concept["hint"])

```

When reading a specific BigQuery wildcard family, the hint dictionary populates the response with sharding information:

```python
from reference_agent.tools.source_tools import read_concept_raw

# A wildcard family reference (note the trailing underscore)

family_id = "tables/events_"
metadata = read_concept_raw(family_id)

print(metadata["wildcard"])          # True

print(metadata["family_prefix"])     # "events_"

print(metadata["last_shard"])        # "events_20210103"

```

For concrete tables, sampling rows relies on the hint to resolve the correct table identifier:

```python
from reference_agent.tools.source_tools import sample_rows

rows_info = sample_rows("tables/users", n=3)
print(rows_info["rows"])   # List of up to 3 row dicts

```

## Implementation Details in the BigQuery Source

The BigQuery source implementation in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) demonstrates the practical application of hint dictionaries. When the source discovers tables, it evaluates whether they belong to a sharded family (identified by a common prefix and date suffixes).

For sharded families, the source calculates `shard_count`, identifies `first_shard` and `last_shard`, and sets `wildcard: true`. The `_representative_table_id` method then uses this hint to return the most recent shard when the system needs to sample data or retrieve metadata for the entire family.

Unit tests in [`okf/tests/test_bigquery_source.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/tests/test_bigquery_source.py) validate these behaviors, asserting that hint fields like `wildcard` and `shard_count` are correctly populated based on the underlying BigQuery metadata structure.

## Summary

- The **hint dictionary** in `ConceptRef` objects stores source-specific metadata that supplements the core reference fields (`id`, `type`, `resource`).
- In **BigQuery implementations**, the hint distinguishes between wildcard sharded table families and concrete tables, enabling efficient representative selection via `_representative_table_id`.
- The **transport mechanism** allows downstream tools to access contextual data—such as shard counts and dataset ownership—without additional API calls by exposing hints through `source_tools._ref_to_dict`.
- Located primarily in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py) and implemented in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), the hint system supports extensibility for custom sources.

## Frequently Asked Questions

### What fields are typically stored in a ConceptRef hint dictionary?

The specific fields depend on the source implementation. In the BigQuery source, hints commonly include `wildcard` (boolean), `table_id` (string), `family_prefix` (string), `shard_count` (integer), `first_shard` (string), `last_shard` (string), `dataset_project` (string), and `dataset_id` (string). Custom sources may add arbitrary key-value pairs relevant to their specific resource types.

### How does the hint dictionary improve performance?

The hint dictionary eliminates redundant API calls by caching source-specific metadata directly on the reference object. For example, when determining which table to query in a sharded family, the BigQuery source consults the hint rather than re-listing all tables, allowing `_representative_table_id` to instantly return the appropriate shard identifier.

### Can custom sources add their own hint fields?

Yes, the `ConceptRef` architecture in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py) supports arbitrary hint dictionaries. Custom sources can populate hints with any JSON-serializable data that downstream tools might need, such as versioning information, ownership tags, or access control metadata, which then becomes available through `source_tools._ref_to_dict`.

### Where is the hint dictionary defined in the codebase?

The hint dictionary is defined as part of the `ConceptRef` class in [`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py). The BigQuery source implements hint population logic in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), while [`okf/src/reference_agent/tools/source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/source_tools.py) handles the conversion of `ConceptRef` objects to dictionaries, exposing the hint field to CLI and UI components.