What Is the Purpose of the Hint Dictionary in ConceptRef Objects?

The hint dictionary in a ConceptRef object serves as a lightweight, source-specific cache that stores auxiliary metadata for efficient disambiguation—such as distinguishing between wildcard sharded table families and concrete tables—while transporting contextual information that downstream tools can access without additional API calls.

In the GoogleCloudPlatform/knowledge-catalog repository, ConceptRef objects act as lightweight identifiers returned by data sources like BigQuery to represent datasets, tables, and other entities. While these references contain mandatory fields such as id and type, they also include an optional hint dictionary that carries critical source-specific metadata. This compact data structure enables sources to optimize resource lookup and provides CLI and UI tools with immediate context that would otherwise require redundant API requests.

Understanding ConceptRef and the Hint Dictionary

ConceptRef objects are defined in okf/src/reference_agent/sources/base.py as the standard interface for referencing concepts within the knowledge catalog. Each object contains:

• id: A unique identifier for the concept
• type: The classification of the concept (e.g., dataset, table)
• resource: (Optional) A URI or resource pointer
• hint: (Optional) A dictionary containing source-specific auxiliary metadata

The hint dictionary functions as a transport mechanism for data that the source deems necessary for efficient downstream operations. Unlike the core reference fields, hints are source-defined and can vary based on the specific implementation requirements of each connector.

Two Primary Purposes of the ConceptRef Hint Dictionary

The hint dictionary serves two architectural functions within the knowledge catalog system: disambiguating complex resource patterns and caching contextual metadata for downstream consumers.

Disambiguating Wildcard Families and Concrete Tables

For BigQuery implementations, the hint dictionary resolves ambiguity between wildcard families of sharded tables and single concrete tables. In okf/src/reference_agent/sources/bigquery.py, the source populates the hint with a wildcard boolean flag that determines how the system should interpret the reference.

When wildcard is true, the hint contains metadata about the sharded family:

{
  "wildcard": true,
  "family_prefix": "events_",
  "shard_count": 3,
  "first_shard": "events_20210101",
  "last_shard": "events_20210103"
}

The BigQuerySource uses this information in the _representative_table_id method to select a representative table (typically the most recent shard) without enumerating all shards again. For concrete tables, the hint simply marks wildcard as false and includes the specific table_id:

{ "wildcard": false, "table_id": "users" }

Transporting Source-Specific Context

Beyond disambiguation, the hint dictionary acts as a compact metadata cache accessible via source_tools._ref_to_dict in okf/src/reference_agent/tools/source_tools.py. This allows CLI and UI components to display rich information without invoking read_concept.

BigQuery stores additional context such as:

• dataset_project and dataset_id: Ownership information for the table
• family_prefix, shard_count, first_shard, last_shard: Detailed sharding metadata

Custom sources can extend this pattern by adding arbitrary key-value pairs—such as version numbers or ownership tags—that downstream agents require for decision-making.

How to Access and Use the Hint Dictionary in Practice

The source_tools module exposes hint data through conversion functions that serialize ConceptRef objects into dictionaries. This enables inspection and utilization of hint metadata in agent workflows.

To list all concepts and inspect their hints:

from reference_agent.tools.source_tools import list_concepts

for concept in list_concepts():
    print(f"{concept['id']} – type: {concept['type']}")
    print("  hint →", concept["hint"])

When reading a specific BigQuery wildcard family, the hint dictionary populates the response with sharding information:

from reference_agent.tools.source_tools import read_concept_raw

# A wildcard family reference (note the trailing underscore)

family_id = "tables/events_"
metadata = read_concept_raw(family_id)

print(metadata["wildcard"])          # True

print(metadata["family_prefix"])     # "events_"

print(metadata["last_shard"])        # "events_20210103"

For concrete tables, sampling rows relies on the hint to resolve the correct table identifier:

from reference_agent.tools.source_tools import sample_rows

rows_info = sample_rows("tables/users", n=3)
print(rows_info["rows"])   # List of up to 3 row dicts

Implementation Details in the BigQuery Source

The BigQuery source implementation in okf/src/reference_agent/sources/bigquery.py demonstrates the practical application of hint dictionaries. When the source discovers tables, it evaluates whether they belong to a sharded family (identified by a common prefix and date suffixes).

For sharded families, the source calculates shard_count, identifies first_shard and last_shard, and sets wildcard: true. The _representative_table_id method then uses this hint to return the most recent shard when the system needs to sample data or retrieve metadata for the entire family.

Unit tests in okf/tests/test_bigquery_source.py validate these behaviors, asserting that hint fields like wildcard and shard_count are correctly populated based on the underlying BigQuery metadata structure.

Summary

• The hint dictionary in ConceptRef objects stores source-specific metadata that supplements the core reference fields (id, type, resource).
• In BigQuery implementations, the hint distinguishes between wildcard sharded table families and concrete tables, enabling efficient representative selection via _representative_table_id.
• The transport mechanism allows downstream tools to access contextual data—such as shard counts and dataset ownership—without additional API calls by exposing hints through source_tools._ref_to_dict.
• Located primarily in okf/src/reference_agent/sources/base.py and implemented in okf/src/reference_agent/sources/bigquery.py, the hint system supports extensibility for custom sources.

Frequently Asked Questions

What fields are typically stored in a ConceptRef hint dictionary?

The specific fields depend on the source implementation. In the BigQuery source, hints commonly include wildcard (boolean), table_id (string), family_prefix (string), shard_count (integer), first_shard (string), last_shard (string), dataset_project (string), and dataset_id (string). Custom sources may add arbitrary key-value pairs relevant to their specific resource types.

How does the hint dictionary improve performance?

The hint dictionary eliminates redundant API calls by caching source-specific metadata directly on the reference object. For example, when determining which table to query in a sharded family, the BigQuery source consults the hint rather than re-listing all tables, allowing _representative_table_id to instantly return the appropriate shard identifier.

Can custom sources add their own hint fields?

Yes, the ConceptRef architecture in okf/src/reference_agent/sources/base.py supports arbitrary hint dictionaries. Custom sources can populate hints with any JSON-serializable data that downstream tools might need, such as versioning information, ownership tags, or access control metadata, which then becomes available through source_tools._ref_to_dict.

Where is the hint dictionary defined in the codebase?

The hint dictionary is defined as part of the ConceptRef class in okf/src/reference_agent/sources/base.py. The BigQuery source implements hint population logic in okf/src/reference_agent/sources/bigquery.py, while okf/src/reference_agent/tools/source_tools.py handles the conversion of ConceptRef objects to dictionaries, exposing the hint field to CLI and UI components.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}