# How the Reference Agent Handles Nested or Recursive Schema Fields in BigQuery Tables

> Discover how the Reference Agent transforms nested BigQuery schemas into hierarchical JSON. It preserves RECORD and STRUCT types with arbitrary depth for efficient data management.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-16

---

**The Reference Agent converts nested BigQuery schemas into hierarchical JSON by recursively walking `SchemaField` objects, preserving `RECORD` and `STRUCT` types up to arbitrary depth.**

The Knowledge Catalog Reference Agent enables seamless ingestion of BigQuery metadata by converting complex table schemas into a standardized JSON format. When working with nested or recursive schema fields in BigQuery tables, the agent employs a recursive traversal strategy implemented in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) that preserves the full hierarchical structure for downstream consumers like the knowledge-catalog UI or LLM enrichment pipelines.

## Recursive Schema Translation Algorithm

The conversion logic centers on the `_schema_to_dict` function, which translates BigQuery's `SchemaField` objects into plain Python dictionaries that can be serialized to JSON.

### Top-Level SchemaField Processing

According to the GoogleCloudPlatform/knowledge-catalog source code, `_schema_to_dict` receives a list of `bigquery.SchemaField` objects (lines 13-20). For each field, it constructs a dictionary containing the field's `name`, `type`, and `mode`. If a field includes a description, that metadata is also added to the dictionary, ensuring comprehensive schema documentation is preserved through the conversion process.

### Handling Nested Records and STRUCTs

When the function encounters a field with non-empty `field.fields`—indicating a `RECORD` or `STRUCT` type—it initiates recursive processing. As implemented in lines 23-25 of [`bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/bigquery.py), the function calls itself with `list(f.fields)` and stores the returned list of child-field dictionaries under the key `"fields"`. This self-referential pattern enables the Reference Agent to handle arbitrarily deep nesting structures without depth limitations.

### JSON Structure Aggregation

Each processed field dictionary is appended to an output list that ultimately mirrors the original BigQuery schema hierarchy. The resulting structure maintains parent-child relationships through nested `"fields"` arrays, preserving type information, mode constraints (`NULLABLE`, `REQUIRED`, `REPEATED`), and descriptions at every level of the hierarchy.

## Integration with Table Reading Operations

When the Reference Agent's `read_concept` method fetches a table, it invokes `_schema_to_dict` on the table's schema list (lines 52-53). If the table lacks a schema definition, the function receives an empty list and returns an empty JSON array. The resulting schema structure is included in the returned payload under the `"schema"` key, allowing downstream components to traverse the hierarchy identically to how they would navigate the original BigQuery schema.

## Practical Implementation Examples

The following examples demonstrate how to convert nested BigQuery schemas using the Reference Agent's utility functions:

```python

# Example: converting a table schema with nested fields

from google.cloud import bigquery
from reference_agent.sources.bigquery import _schema_to_dict

client = bigquery.Client()
table = client.get_table("myproject.mydataset.mytable")
json_schema = _schema_to_dict(list(table.schema or []))

# json_schema now looks like:

# [

#   {"name": "user_id", "type": "STRING", "mode": "REQUIRED"},

#   {

#     "name": "address",

#     "type": "RECORD",

#     "mode": "NULLABLE",

#     "fields": [

#       {"name": "street", "type": "STRING", "mode": "NULLABLE"},

#       {"name": "city",   "type": "STRING", "mode": "NULLABLE"},

#       {"name": "coords",

#        "type": "RECORD",

#        "mode": "NULLABLE",

#        "fields": [

#          {"name": "lat", "type": "FLOAT", "mode": "NULLABLE"},

#          {"name": "lon", "type": "FLOAT", "mode": "NULLABLE"}

#        ]}

#     ]

#   }

# ]

```

```python

# Example: using the Reference Agent to read a table, automatically getting the nested schema

from reference_agent.sources.bigquery import BigQuerySource

source = BigQuerySource(dataset="myproject.mydataset")
concepts = source.list_concepts()               # discovers tables

table_ref = next(c for c in concepts if c.id[1] == "mytable")
table_info = source.read_concept(table_ref)    # includes "schema" field with nested JSON

print(table_info["schema"])

```

## Summary

- The Reference Agent handles nested BigQuery schemas through the recursive `_schema_to_dict` function in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py).
- **Recursive traversal** occurs when encountering `RECORD` or `STRUCT` types, with child fields stored under the `"fields"` key in the resulting JSON.
- The conversion preserves all schema metadata including field names, types, modes, and descriptions at every nesting level.
- **Integration with `read_concept`** automatically includes the JSON schema in the response payload, enabling downstream consumers to process arbitrarily complex table structures.

## Frequently Asked Questions

### How does the Reference Agent handle arbitrarily deep nesting in BigQuery schemas?

The Reference Agent employs a recursive function call pattern in `_schema_to_dict` that invokes itself whenever it encounters a field with nested sub-fields. Because Python's recursion limit is the only constraint, this approach can theoretically handle schemas of any practical depth, with each level correctly parented under its respective `"fields"` array in the JSON output.

### What is the output format when converting BigQuery SchemaFields to JSON?

The output is a list of dictionaries where each dictionary represents a schema field with keys for `name`, `type`, and `mode`. Nested fields include an additional `fields` key containing a list of child field dictionaries. This structure preserves the hierarchical nature of BigQuery's `RECORD` and `STRUCT` types while remaining serializable to standard JSON for APIs and LLM prompts.

### Where is the schema conversion logic implemented in the Knowledge Catalog repository?

The core conversion logic resides in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), specifically lines 13-25 where `_schema_to_dict` processes individual SchemaField objects and handles recursion. The function is called by `read_concept` at lines 52-53 to include schema metadata when retrieving table information.

### Can the Reference Agent handle BigQuery tables without a defined schema?

Yes. When `read_concept` encounters a table with no schema, it passes an empty list to `_schema_to_dict`, which returns an empty JSON array. This ensures consistent response formatting without errors, allowing the downstream pipeline to handle schema-less tables gracefully.