Authentication and Project Billing Requirements for the BigQuery Source

To connect the Knowledge Catalog to BigQuery, you must configure Application Default Credentials (ADC), ensure the caller has BigQuery Data Viewer permissions on the target dataset, and optionally designate a specific billing project to absorb API costs.

The BigQuerySource class in the GoogleCloudPlatform/knowledge-catalog repository provides the reference implementation for reading BigQuery metadata and table samples. Located in okf/src/reference_agent/sources/bigquery.py, this source handles authentication via the Google Cloud Python client library and supports cross-project billing configurations. Understanding these requirements ensures seamless integration without permission or billing errors.

Google Cloud Authentication via Application Default Credentials

The BigQuerySource class relies on Application Default Credentials (ADC) to authenticate with the BigQuery API. In the __init__ method of okf/src/reference_agent/sources/bigquery.py (lines 32-40), the constructor initializes a bigquery.Client instance using the optional billing_project parameter:

self.client = bigquery.Client(project=billing_project)

When billing_project is None, the client automatically infers the project from the ADC. The source accepts any valid ADC source, including:

• A service account JSON key file referenced by the GOOGLE_APPLICATION_CREDENTIALS environment variable
• User credentials obtained via gcloud auth application-default login
• Attached service accounts on Google Cloud compute environments

Required IAM Permissions for Dataset Access

The authenticated principal must possess specific permissions to inspect the target dataset. The source implicitly requires these permissions when calling list_tables, get_dataset, and get_table within methods such as list_concepts, read_concept, and sample_rows.

The caller needs the following permissions on the dataset resource:

• bigquery.tables.get
• bigquery.tables.list
• bigquery.datasets.get

Grant the BigQuery Data Viewer role (or a custom role containing these permissions) at the dataset level or higher. Without these permissions, the source raises a 403 Forbidden error when attempting to list concepts or sample rows.

Configuring the Billing Project

By default, BigQuery charges API usage to the project associated with the authenticated credentials. The BigQuerySource constructor accepts an optional billing_project parameter to override this behavior and charge to a different project:

def __init__(self, dataset: str, billing_project: str | None = None):

When provided, the bigquery.Client is instantiated with this specific project ID, directing all query and API costs to the designated billing project. This configuration is essential when the target dataset resides in a project separate from your budget allocation or when using a central billing account for data catalog operations.

Dataset Identifier Format Validation

The source enforces strict validation of the dataset parameter in __init__ (lines 32-38 of okf/src/reference_agent/sources/bigquery.py). The identifier must follow the project_id.dataset_id format. If the input string lacks the project prefix or contains invalid characters, the constructor raises a clear ValueError before attempting API calls.

Valid example: "my-data-project.my_dataset"

Invalid example: "my_dataset" (missing project component)

Handling Sharded Tables

For tables following the {prefix}{shard} naming convention (e.g., events_20230101), the source automatically groups them into "family" concepts. The list_concepts method uses the _SHARD_SUFFIX_RE regular expression (lines 75-101) to detect these patterns and constructs wildcard URIs. This aggregation requires no additional authentication or billing configuration—the existing client credentials apply uniformly to all shards in the family.

Complete Implementation Example

The following example demonstrates authenticating via ADC, configuring a separate billing project, and listing concepts from a BigQuery dataset:

from reference_agent.sources.bigquery import BigQuerySource

# Authenticate via ADC and charge to the credentials' project

src = BigQuerySource(dataset="my-data-project.my_dataset")

# Or, authenticate via ADC but charge to a specific billing project

src = BigQuerySource(
    dataset="my-data-project.my_dataset",
    billing_project="my-billing-project"
)

# List all concepts (datasets, tables, and sharded families)

concepts = src.list_concepts()
for concept in concepts:
    print(concept.id, concept.type, concept.resource)

Summary

• Authentication: The BigQuerySource uses Application Default Credentials (ADC) via bigquery.Client(project=billing_project) in okf/src/reference_agent/sources/bigquery.py.
• Permissions: Grant BigQuery Data Viewer (or bigquery.tables.get, bigquery.tables.list, bigquery.datasets.get) on the target dataset to avoid 403 errors.
• Billing: Pass the optional billing_project parameter to charge API usage to a project other than the one tied to your ADC.
• Format: Provide dataset identifiers as project_id.dataset_id; validation occurs in the constructor.
• Sharded Tables: The source automatically detects and groups sharded tables using _SHARD_SUFFIX_RE without extra auth steps.

Frequently Asked Questions

What happens if I do not specify a billing project?

If you omit the billing_project parameter, the BigQuery client defaults to the project associated with your Application Default Credentials. All API and query costs accrue to that project.

Which IAM role is required for the BigQuery source?

You must grant the BigQuery Data Viewer role (or a custom role with equivalent permissions) on the target dataset. The source requires bigquery.tables.get, bigquery.tables.list, and bigquery.datasets.get permissions to list metadata and sample rows.

How does the source handle sharded tables with daily or hourly suffixes?

The list_concepts method in okf/src/reference_agent/sources/bigquery.py detects sharded tables using the _SHARD_SUFFIX_RE regular expression (lines 75-101). It groups these into family concepts with wildcard URIs, allowing you to treat them as a single logical resource without additional configuration.

Can I use a service account JSON key instead of gcloud login?

Yes. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of your service account JSON file. The bigquery.Client automatically loads these credentials as part of the ADC chain.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}