How the Reference Agent Handles Nested or Recursive Schema Fields in BigQuery Tables

The Reference Agent converts nested BigQuery schemas into hierarchical JSON by recursively walking SchemaField objects, preserving RECORD and STRUCT types up to arbitrary depth.

The Knowledge Catalog Reference Agent enables seamless ingestion of BigQuery metadata by converting complex table schemas into a standardized JSON format. When working with nested or recursive schema fields in BigQuery tables, the agent employs a recursive traversal strategy implemented in okf/src/reference_agent/sources/bigquery.py that preserves the full hierarchical structure for downstream consumers like the knowledge-catalog UI or LLM enrichment pipelines.

Recursive Schema Translation Algorithm

The conversion logic centers on the _schema_to_dict function, which translates BigQuery's SchemaField objects into plain Python dictionaries that can be serialized to JSON.

Top-Level SchemaField Processing

According to the GoogleCloudPlatform/knowledge-catalog source code, _schema_to_dict receives a list of bigquery.SchemaField objects (lines 13-20). For each field, it constructs a dictionary containing the field's name, type, and mode. If a field includes a description, that metadata is also added to the dictionary, ensuring comprehensive schema documentation is preserved through the conversion process.

Handling Nested Records and STRUCTs

When the function encounters a field with non-empty field.fields—indicating a RECORD or STRUCT type—it initiates recursive processing. As implemented in lines 23-25 of bigquery.py, the function calls itself with list(f.fields) and stores the returned list of child-field dictionaries under the key "fields". This self-referential pattern enables the Reference Agent to handle arbitrarily deep nesting structures without depth limitations.

JSON Structure Aggregation

Each processed field dictionary is appended to an output list that ultimately mirrors the original BigQuery schema hierarchy. The resulting structure maintains parent-child relationships through nested "fields" arrays, preserving type information, mode constraints (NULLABLE, REQUIRED, REPEATED), and descriptions at every level of the hierarchy.

Integration with Table Reading Operations

When the Reference Agent's read_concept method fetches a table, it invokes _schema_to_dict on the table's schema list (lines 52-53). If the table lacks a schema definition, the function receives an empty list and returns an empty JSON array. The resulting schema structure is included in the returned payload under the "schema" key, allowing downstream components to traverse the hierarchy identically to how they would navigate the original BigQuery schema.

Practical Implementation Examples

The following examples demonstrate how to convert nested BigQuery schemas using the Reference Agent's utility functions:

# Example: converting a table schema with nested fields

from google.cloud import bigquery
from reference_agent.sources.bigquery import _schema_to_dict

client = bigquery.Client()
table = client.get_table("myproject.mydataset.mytable")
json_schema = _schema_to_dict(list(table.schema or []))

# json_schema now looks like:

# [

#   {"name": "user_id", "type": "STRING", "mode": "REQUIRED"},

#   {

#     "name": "address",

#     "type": "RECORD",

#     "mode": "NULLABLE",

#     "fields": [

#       {"name": "street", "type": "STRING", "mode": "NULLABLE"},

#       {"name": "city",   "type": "STRING", "mode": "NULLABLE"},

#       {"name": "coords",

#        "type": "RECORD",

#        "mode": "NULLABLE",

#        "fields": [

#          {"name": "lat", "type": "FLOAT", "mode": "NULLABLE"},

#          {"name": "lon", "type": "FLOAT", "mode": "NULLABLE"}

#        ]}

#     ]

#   }

# ]

# Example: using the Reference Agent to read a table, automatically getting the nested schema

from reference_agent.sources.bigquery import BigQuerySource

source = BigQuerySource(dataset="myproject.mydataset")
concepts = source.list_concepts()               # discovers tables

table_ref = next(c for c in concepts if c.id[1] == "mytable")
table_info = source.read_concept(table_ref)    # includes "schema" field with nested JSON

print(table_info["schema"])

Summary

• The Reference Agent handles nested BigQuery schemas through the recursive _schema_to_dict function in okf/src/reference_agent/sources/bigquery.py.
• Recursive traversal occurs when encountering RECORD or STRUCT types, with child fields stored under the "fields" key in the resulting JSON.
• The conversion preserves all schema metadata including field names, types, modes, and descriptions at every nesting level.
• Integration with read_concept automatically includes the JSON schema in the response payload, enabling downstream consumers to process arbitrarily complex table structures.

Frequently Asked Questions

How does the Reference Agent handle arbitrarily deep nesting in BigQuery schemas?

The Reference Agent employs a recursive function call pattern in _schema_to_dict that invokes itself whenever it encounters a field with nested sub-fields. Because Python's recursion limit is the only constraint, this approach can theoretically handle schemas of any practical depth, with each level correctly parented under its respective "fields" array in the JSON output.

What is the output format when converting BigQuery SchemaFields to JSON?

The output is a list of dictionaries where each dictionary represents a schema field with keys for name, type, and mode. Nested fields include an additional fields key containing a list of child field dictionaries. This structure preserves the hierarchical nature of BigQuery's RECORD and STRUCT types while remaining serializable to standard JSON for APIs and LLM prompts.

Where is the schema conversion logic implemented in the Knowledge Catalog repository?

The core conversion logic resides in okf/src/reference_agent/sources/bigquery.py, specifically lines 13-25 where _schema_to_dict processes individual SchemaField objects and handles recursion. The function is called by read_concept at lines 52-53 to include schema metadata when retrieving table information.

Can the Reference Agent handle BigQuery tables without a defined schema?

Yes. When read_concept encounters a table with no schema, it passes an empty list to _schema_to_dict, which returns an empty JSON array. This ensures consistent response formatting without errors, allowing the downstream pipeline to handle schema-less tables gracefully.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}