# Understanding the _family_index in BigQuerySource: Purpose and Population Timing

> Discover the purpose of the BigQuerySource _family_index and when it is populated. Learn how it optimizes wildcard resolution and metadata retrieval in BigQuery.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-16

---

**The `_family_index` is a private dictionary that caches sorted shard lists for BigQuery table families, populated lazily during the first call to `list_concepts()` to enable efficient wildcard resolution and family-level metadata retrieval.**

In the `GoogleCloudPlatform/knowledge-catalog` repository, the `BigQuerySource` class manages sharded tables as logical families. The private **`_family_index`** attribute serves as the core caching mechanism that maps family concept IDs to their constituent shard IDs, enabling the source to treat date-suffixed tables as unified entities while avoiding redundant API calls.

## What Is the _family_index?

The **`_family_index`** is defined in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) as a private instance attribute with the type signature `dict[tuple[str, ...], list[str]]`. It stores a mapping where each key is a **family concept ID** (e.g., `("tables", "events_")`) and each value is a chronologically sorted list of concrete table IDs belonging to that family (e.g., `["events_20210101", "events_20210102", ...]`).

This index treats sharded tables sharing a common prefix as a single *family*, allowing the source to abstract away the complexity of individual daily or hourly shards behind a unified concept interface.

## Purpose and Functionality

### Resolving Wildcard Concepts

When `BigQuerySource` encounters a table family marked with `wildcard=True`, it uses the **`_family_index`** to identify a **representative table** via the internal `_representative_table_id()` method. Rather than querying metadata for every shard, the source selects the last shard in the sorted list (typically the most recent date) to provide schema and sampling information for the entire family.

### Providing Family-Level Hints

The index enables rich metadata hints attached to family concepts. When you retrieve a concept's dictionary via `list_concepts()`, the source populates hints including:

- **Shard count**: Total number of tables in the family
- **First and last shard**: Chronological boundaries
- **Family prefix**: The common root string (e.g., `"events_"`)

### Caching for Performance

By building the index once during the initial enumeration, **`_family_index`** eliminates redundant API calls to `client.list_tables()`. Subsequent operations such as `read_concept()` or `sample_rows()` reuse the cached shard lists, significantly reducing BigQuery API quota consumption.

## When Is _family_index Populated?

The index follows a **lazy initialization pattern**. It is populated exactly once during the first invocation of `list_concepts()`:

1. The method calls `client.list_tables(self._dataset_ref)` to enumerate all tables
2. For each table matching the shard regex `_SHARD_SUFFIX_RE = r"^(?P<prefix>.+?_)(?P<shard>\d{6,8})$"`, it extracts the prefix and shard identifier
3. Shards are collected into lists per prefix using `families.setdefault(m.group("prefix"), []).append(tbl.table_id)`
4. After processing, each prefix becomes a family concept, and the sorted shard list is stored: `self._family_index[family_concept_id] = shards_sorted`

Once initialized, the dictionary persists for the lifetime of the `BigQuerySource` instance, serving subsequent method calls without re-querying the dataset's table list.

## Code Example: Working with the Family Index

```python
from okf.src.reference_agent.sources.bigquery import BigQuerySource

# Initialize the source

src = BigQuerySource(dataset="my-project.my_dataset")

# First call triggers index population

concepts = src.list_concepts()  # _family_index is built here

# Locate a wildcard family concept

family_ref = next(
    c for c in concepts 
    if c.id[0] == "tables" and c.hint.get("wildcard")
)

# Access cached shards directly (internal reference)

shard_ids = src._family_index[family_ref.id]

# Returns: ['events_20210101', 'events_20210102', ...]

# Metadata reading uses the index automatically

metadata = src.read_concept(family_ref)  # Selects representative shard

```

## Technical Implementation Details

The shard detection logic relies on the regex pattern `_SHARD_SUFFIX_RE = r"^(?P<prefix>.+?_)(?P<shard>\d{6,8})$"` defined in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py). This pattern identifies tables with numeric suffixes of 6-8 digits (typically YYYYMMDD or YYYYMMDDHH formats), grouping them by their underscore-terminated prefix.

The test suite in [`okf/tests/test_bigquery_source.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/tests/test_bigquery_source.py) validates this behavior, ensuring that wildcard concepts return correct hints and that representative table selection consistently returns the lexicographically last shard in the family.

## Summary

- The **`_family_index`** is a `dict[tuple[str, ...], list[str]]` caching shard lists per table family in `BigQuerySource`
- It enables **wildcard resolution** by mapping families to representative concrete tables via `_representative_table_id()`
- Population occurs **lazily on first `list_concepts()` call**, scanning tables matching `_SHARD_SUFFIX_RE` and grouping by prefix
- The index powers **family-level hints** including shard counts, first/last dates, and common prefixes
- Once built, it eliminates redundant BigQuery API calls for the instance lifetime

## Frequently Asked Questions

### What data structure does _family_index use?

The **`_family_index`** uses a dictionary mapping tuple-based concept IDs to lists of strings. Specifically, it is typed as `dict[tuple[str, ...], list[str]]`, where keys represent family identifiers like `("tables", "events_")` and values contain sorted lists of full table IDs such as `["events_20210101", "events_20210102"]`.

### When exactly is the _family_index populated?

The index is populated **lazily during the first execution of `list_concepts()`**. The method scans the dataset using `client.list_tables()`, identifies sharded tables via regex matching against `_SHARD_SUFFIX_RE`, groups them by prefix, and caches the sorted results. No API enumeration occurs until you explicitly call `list_concepts()`.

### How does _family_index improve query performance?

By caching the complete enumeration of table shards, **`_family_index`** prevents redundant calls to BigQuery's table listing API. When methods like `read_concept()` or `sample_rows()` need to resolve a wildcard family or select a representative table, they access the pre-computed index rather than re-scanning the dataset, reducing latency and API quota usage.

### Can I access _family_index directly in production code?

While the attribute exists at `src._family_index`, it is a **private implementation detail** (indicated by the leading underscore) of the `BigQuerySource` class in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py). Direct access is not recommended; instead, use the public `list_concepts()` and `read_concept()` methods which leverage the index internally while maintaining API compatibility.