Partitioning Strategies Extracted by the BigQuery Source in Knowledge Catalog

The BigQuery source extracts both time-based partitioning and range (integer) partitioning strategies from BigQuery tables, representing each as JSON-compatible dictionaries with fields like type, expiration_ms, and numeric range definitions.

The GoogleCloudPlatform/knowledge-catalog repository provides a BigQuerySource class that automatically discovers and encodes table metadata for knowledge-graph consumption. When crawling BigQuery datasets, this source specifically identifies two distinct partitioning strategies through the BigQuery Python client library and exposes them via the read_concept() method. Understanding these extracted partitioning strategies helps data engineers and catalog consumers optimize query performance and data lifecycle management.

Detected Partitioning Strategies

The BigQuery source distinguishes between two partitioning mechanisms by inspecting the BigQuery table object's properties. Each strategy is detected independently and stored in separate keys within the returned metadata dictionary.

Time-Based Partitioning

When a table uses time-based partitioning, the source checks if tbl.time_partitioning exists. If detected, it extracts:

• type: The granularity of partitioning (e.g., DAY, HOUR, MONTH, YEAR)
• field: The column used for partitioning (may be None for ingestion-time partitioning)
• expiration_ms: Partition expiration time in milliseconds if configured

This information is stored under the time_partitioning key in the result dictionary.

Range (Integer) Partitioning

For tables using integer-range partitioning, the source checks tbl.range_partitioning. When present, it extracts:

• field: The column used for range partitioning
• range: A nested dictionary containing:
  • start: The start of the range
  • end: The end of the range
  • interval: The width of each partition

This data is stored under the range_partitioning key.

Implementation in the BigQuery Source

The extraction logic resides in okf/src/reference_agent/sources/bigquery.py within the BigQuerySource class. The read_concept() method constructs the metadata dictionary by conditionally adding partitioning information based on the table properties:

if tbl.time_partitioning:
    data["time_partitioning"] = {
        "type": tbl.time_partitioning.type_,
        "field": tbl.time_partitioning.field,
        "expiration_ms": tbl.time_partitioning.expiration_ms,
    }

if tbl.range_partitioning:
    rp = tbl.range_partitioning
    data["range_partitioning"] = {
        "field": rp.field,
        "range": {
            "start": rp.range_.start,
            "end": rp.range_.end,
            "interval": rp.range_.interval,
        },
    }

This implementation ensures that only existing partitioning strategies appear in the output, with missing strategies represented by the absence of their respective keys (or None values depending on downstream processing).

Accessing Partitioning Metadata

To inspect partitioning strategies for a specific table, instantiate the source and call read_concept() on the table concept:

# Create a source for a public dataset

src = BigQuerySource(dataset="bigquery-public-data.crypto_bitcoin")

# Find the concept representing the "transactions" table (wildcard family)

family = next(c for c in src.list_concepts()
              if c.id == ("tables", "transactions_"))

# Pull the full concept metadata, including partitioning info

metadata = src.read_concept(family)

# Inspect the extracted partitioning strategies

print(metadata.get("time_partitioning"))   # → None (no time partitioning)

print(metadata.get("range_partitioning")) # e.g. {'field': 'block_timestamp_month', ...}

The returned metadata dictionary always includes basic table information (schema, clustering fields) and conditionally includes the partitioning dictionaries when present:

# Example output for a table that uses daily time partitioning

{
    "time_partitioning": {
        "type": "DAY",
        "field": None,
        "expiration_ms": None
    },
    "range_partitioning": None,
    "clustering_fields": ["user_pseudo_id"],
    # ... additional metadata

}

Summary

• The BigQuery source extracts two distinct partitioning strategies: time-based partitioning (daily, hourly, monthly, yearly) and range (integer) partitioning (fixed numeric ranges).
• Partitioning metadata is stored in okf/src/reference_agent/sources/bigquery.py within the BigQuerySource class, specifically in the read_concept() method.
• Time-based partitions expose type, field, and expiration_ms properties.
• Range partitions expose field and range definitions containing start, end, and interval values.
• Unit tests in okf/tests/test_bigquery_source.py verify extraction accuracy (see test_read_concept_returns_schema_and_partitioning).

Frequently Asked Questions

What two partitioning strategies does the BigQuery source extract?

The BigQuery source extracts time-based partitioning and range (integer) partitioning. Time-based partitioning organizes data by ingestion time or timestamp columns, while range partitioning divides data based on integer column values falling within specific numeric ranges.

How does the BigQuery source represent partitioning metadata?

The source represents each strategy as a JSON-compatible dictionary. Time-based partitioning uses keys for type, field, and expiration_ms. Range partitioning uses a field key and a nested range object containing start, end, and interval values. These dictionaries are attached to the time_partitioning and range_partitioning keys in the metadata returned by read_concept().

Where is the partitioning extraction logic implemented?

The extraction logic is implemented in the BigQuerySource class located at okf/src/reference_agent/sources/bigquery.py. The read_concept() method checks the BigQuery table object's time_partitioning and range_partitioning attributes and converts them into serializable dictionaries for the Knowledge Catalog consumers.

Can the BigQuery source detect both partitioning types on the same table?

While BigQuery tables typically use only one partitioning strategy, the source code is designed to check for both independently. If a table somehow contained both time_partitioning and range_partitioning configurations (which is not standard BigQuery behavior), the source would extract and represent both in the returned metadata dictionary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}