Understanding the _family_index in BigQuerySource: Purpose and Population Timing

The _family_index is a private dictionary that caches sorted shard lists for BigQuery table families, populated lazily during the first call to list_concepts() to enable efficient wildcard resolution and family-level metadata retrieval.

In the GoogleCloudPlatform/knowledge-catalog repository, the BigQuerySource class manages sharded tables as logical families. The private _family_index attribute serves as the core caching mechanism that maps family concept IDs to their constituent shard IDs, enabling the source to treat date-suffixed tables as unified entities while avoiding redundant API calls.

What Is the _family_index?

The _family_index is defined in okf/src/reference_agent/sources/bigquery.py as a private instance attribute with the type signature dict[tuple[str, ...], list[str]]. It stores a mapping where each key is a family concept ID (e.g., ("tables", "events_")) and each value is a chronologically sorted list of concrete table IDs belonging to that family (e.g., ["events_20210101", "events_20210102", ...]).

This index treats sharded tables sharing a common prefix as a single family, allowing the source to abstract away the complexity of individual daily or hourly shards behind a unified concept interface.

Purpose and Functionality

Resolving Wildcard Concepts

When BigQuerySource encounters a table family marked with wildcard=True, it uses the _family_index to identify a representative table via the internal _representative_table_id() method. Rather than querying metadata for every shard, the source selects the last shard in the sorted list (typically the most recent date) to provide schema and sampling information for the entire family.

Providing Family-Level Hints

The index enables rich metadata hints attached to family concepts. When you retrieve a concept's dictionary via list_concepts(), the source populates hints including:

• Shard count: Total number of tables in the family
• First and last shard: Chronological boundaries
• Family prefix: The common root string (e.g., "events_")

Caching for Performance

By building the index once during the initial enumeration, _family_index eliminates redundant API calls to client.list_tables(). Subsequent operations such as read_concept() or sample_rows() reuse the cached shard lists, significantly reducing BigQuery API quota consumption.

When Is _family_index Populated?

The index follows a lazy initialization pattern. It is populated exactly once during the first invocation of list_concepts():

1. The method calls client.list_tables(self._dataset_ref) to enumerate all tables
2. For each table matching the shard regex _SHARD_SUFFIX_RE = r"^(?P<prefix>.+?_)(?P<shard>\d{6,8})$", it extracts the prefix and shard identifier
3. Shards are collected into lists per prefix using families.setdefault(m.group("prefix"), []).append(tbl.table_id)
4. After processing, each prefix becomes a family concept, and the sorted shard list is stored: self._family_index[family_concept_id] = shards_sorted

Once initialized, the dictionary persists for the lifetime of the BigQuerySource instance, serving subsequent method calls without re-querying the dataset's table list.

Code Example: Working with the Family Index

from okf.src.reference_agent.sources.bigquery import BigQuerySource

# Initialize the source

src = BigQuerySource(dataset="my-project.my_dataset")

# First call triggers index population

concepts = src.list_concepts()  # _family_index is built here

# Locate a wildcard family concept

family_ref = next(
    c for c in concepts 
    if c.id[0] == "tables" and c.hint.get("wildcard")
)

# Access cached shards directly (internal reference)

shard_ids = src._family_index[family_ref.id]

# Returns: ['events_20210101', 'events_20210102', ...]

# Metadata reading uses the index automatically

metadata = src.read_concept(family_ref)  # Selects representative shard

Technical Implementation Details

The shard detection logic relies on the regex pattern _SHARD_SUFFIX_RE = r"^(?P<prefix>.+?_)(?P<shard>\d{6,8})$" defined in okf/src/reference_agent/sources/bigquery.py. This pattern identifies tables with numeric suffixes of 6-8 digits (typically YYYYMMDD or YYYYMMDDHH formats), grouping them by their underscore-terminated prefix.

The test suite in okf/tests/test_bigquery_source.py validates this behavior, ensuring that wildcard concepts return correct hints and that representative table selection consistently returns the lexicographically last shard in the family.

Summary

• The _family_index is a dict[tuple[str, ...], list[str]] caching shard lists per table family in BigQuerySource
• It enables wildcard resolution by mapping families to representative concrete tables via _representative_table_id()
• Population occurs lazily on first list_concepts() call, scanning tables matching _SHARD_SUFFIX_RE and grouping by prefix
• The index powers family-level hints including shard counts, first/last dates, and common prefixes
• Once built, it eliminates redundant BigQuery API calls for the instance lifetime

Frequently Asked Questions

What data structure does _family_index use?

The _family_index uses a dictionary mapping tuple-based concept IDs to lists of strings. Specifically, it is typed as dict[tuple[str, ...], list[str]], where keys represent family identifiers like ("tables", "events_") and values contain sorted lists of full table IDs such as ["events_20210101", "events_20210102"].

When exactly is the _family_index populated?

The index is populated lazily during the first execution of list_concepts(). The method scans the dataset using client.list_tables(), identifies sharded tables via regex matching against _SHARD_SUFFIX_RE, groups them by prefix, and caches the sorted results. No API enumeration occurs until you explicitly call list_concepts().

How does _family_index improve query performance?

By caching the complete enumeration of table shards, _family_index prevents redundant calls to BigQuery's table listing API. When methods like read_concept() or sample_rows() need to resolve a wildcard family or select a representative table, they access the pre-computed index rather than re-scanning the dataset, reducing latency and API quota usage.

Can I access _family_index directly in production code?

While the attribute exists at src._family_index, it is a private implementation detail (indicated by the leading underscore) of the BigQuerySource class in okf/src/reference_agent/sources/bigquery.py. Direct access is not recommended; instead, use the public list_concepts() and read_concept() methods which leverage the index internally while maintaining API compatibility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}