# Partitioning Strategies Extracted by the BigQuery Source in Knowledge Catalog

> Discover how the BigQuery source in Knowledge Catalog extracts and represents time-based and range partitioning strategies as JSON dictionaries. Learn about type, expiration, and numeric range definitions.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-16

---

**The BigQuery source extracts both time-based partitioning and range (integer) partitioning strategies from BigQuery tables, representing each as JSON-compatible dictionaries with fields like type, expiration_ms, and numeric range definitions.**

The GoogleCloudPlatform/knowledge-catalog repository provides a `BigQuerySource` class that automatically discovers and encodes table metadata for knowledge-graph consumption. When crawling BigQuery datasets, this source specifically identifies two distinct partitioning strategies through the BigQuery Python client library and exposes them via the `read_concept()` method. Understanding these extracted partitioning strategies helps data engineers and catalog consumers optimize query performance and data lifecycle management.

## Detected Partitioning Strategies

The BigQuery source distinguishes between two partitioning mechanisms by inspecting the BigQuery table object's properties. Each strategy is detected independently and stored in separate keys within the returned metadata dictionary.

### Time-Based Partitioning

When a table uses time-based partitioning, the source checks if `tbl.time_partitioning` exists. If detected, it extracts:

- **type**: The granularity of partitioning (e.g., `DAY`, `HOUR`, `MONTH`, `YEAR`)
- **field**: The column used for partitioning (may be `None` for ingestion-time partitioning)
- **expiration_ms**: Partition expiration time in milliseconds if configured

This information is stored under the `time_partitioning` key in the result dictionary.

### Range (Integer) Partitioning

For tables using integer-range partitioning, the source checks `tbl.range_partitioning`. When present, it extracts:

- **field**: The column used for range partitioning
- **range**: A nested dictionary containing:
  - **start**: The start of the range
  - **end**: The end of the range
  - **interval**: The width of each partition

This data is stored under the `range_partitioning` key.

## Implementation in the BigQuery Source

The extraction logic resides in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) within the `BigQuerySource` class. The `read_concept()` method constructs the metadata dictionary by conditionally adding partitioning information based on the table properties:

```python
if tbl.time_partitioning:
    data["time_partitioning"] = {
        "type": tbl.time_partitioning.type_,
        "field": tbl.time_partitioning.field,
        "expiration_ms": tbl.time_partitioning.expiration_ms,
    }

if tbl.range_partitioning:
    rp = tbl.range_partitioning
    data["range_partitioning"] = {
        "field": rp.field,
        "range": {
            "start": rp.range_.start,
            "end": rp.range_.end,
            "interval": rp.range_.interval,
        },
    }

```

This implementation ensures that only existing partitioning strategies appear in the output, with missing strategies represented by the absence of their respective keys (or `None` values depending on downstream processing).

## Accessing Partitioning Metadata

To inspect partitioning strategies for a specific table, instantiate the source and call `read_concept()` on the table concept:

```python

# Create a source for a public dataset

src = BigQuerySource(dataset="bigquery-public-data.crypto_bitcoin")

# Find the concept representing the "transactions" table (wildcard family)

family = next(c for c in src.list_concepts()
              if c.id == ("tables", "transactions_"))

# Pull the full concept metadata, including partitioning info

metadata = src.read_concept(family)

# Inspect the extracted partitioning strategies

print(metadata.get("time_partitioning"))   # → None (no time partitioning)

print(metadata.get("range_partitioning")) # e.g. {'field': 'block_timestamp_month', ...}

```

The returned metadata dictionary always includes basic table information (schema, clustering fields) and conditionally includes the partitioning dictionaries when present:

```python

# Example output for a table that uses daily time partitioning

{
    "time_partitioning": {
        "type": "DAY",
        "field": None,
        "expiration_ms": None
    },
    "range_partitioning": None,
    "clustering_fields": ["user_pseudo_id"],
    # ... additional metadata

}

```

## Summary

- The **BigQuery source** extracts two distinct partitioning strategies: **time-based partitioning** (daily, hourly, monthly, yearly) and **range (integer) partitioning** (fixed numeric ranges).
- Partitioning metadata is stored in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) within the `BigQuerySource` class, specifically in the `read_concept()` method.
- Time-based partitions expose `type`, `field`, and `expiration_ms` properties.
- Range partitions expose `field` and `range` definitions containing `start`, `end`, and `interval` values.
- Unit tests in [`okf/tests/test_bigquery_source.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/tests/test_bigquery_source.py) verify extraction accuracy (see `test_read_concept_returns_schema_and_partitioning`).

## Frequently Asked Questions

### What two partitioning strategies does the BigQuery source extract?

The BigQuery source extracts **time-based partitioning** and **range (integer) partitioning**. Time-based partitioning organizes data by ingestion time or timestamp columns, while range partitioning divides data based on integer column values falling within specific numeric ranges.

### How does the BigQuery source represent partitioning metadata?

The source represents each strategy as a JSON-compatible dictionary. Time-based partitioning uses keys for `type`, `field`, and `expiration_ms`. Range partitioning uses a `field` key and a nested `range` object containing `start`, `end`, and `interval` values. These dictionaries are attached to the `time_partitioning` and `range_partitioning` keys in the metadata returned by `read_concept()`.

### Where is the partitioning extraction logic implemented?

The extraction logic is implemented in the `BigQuerySource` class located at [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py). The `read_concept()` method checks the BigQuery table object's `time_partitioning` and `range_partitioning` attributes and converts them into serializable dictionaries for the Knowledge Catalog consumers.

### Can the BigQuery source detect both partitioning types on the same table?

While BigQuery tables typically use only one partitioning strategy, the source code is designed to check for both independently. If a table somehow contained both `time_partitioning` and `range_partitioning` configurations (which is not standard BigQuery behavior), the source would extract and represent both in the returned metadata dictionary.