# Authentication and Project Billing Requirements for the BigQuery Source

> Learn BigQuery source authentication and billing requirements. Configure ADC, set Data Viewer permissions, and optionally assign a billing project for API costs. Connect Knowledge Catalog to BigQuery seamlessly.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: how-to-guide
- Published: 2026-07-16

---

**To connect the Knowledge Catalog to BigQuery, you must configure Application Default Credentials (ADC), ensure the caller has BigQuery Data Viewer permissions on the target dataset, and optionally designate a specific billing project to absorb API costs.**

The `BigQuerySource` class in the GoogleCloudPlatform/knowledge-catalog repository provides the reference implementation for reading BigQuery metadata and table samples. Located in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), this source handles authentication via the Google Cloud Python client library and supports cross-project billing configurations. Understanding these requirements ensures seamless integration without permission or billing errors.

## Google Cloud Authentication via Application Default Credentials

The `BigQuerySource` class relies on **Application Default Credentials (ADC)** to authenticate with the BigQuery API. In the `__init__` method of [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) (lines 32-40), the constructor initializes a `bigquery.Client` instance using the optional `billing_project` parameter:

```python
self.client = bigquery.Client(project=billing_project)

```

When `billing_project` is `None`, the client automatically infers the project from the ADC. The source accepts any valid ADC source, including:

- A service account JSON key file referenced by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable
- User credentials obtained via `gcloud auth application-default login`
- Attached service accounts on Google Cloud compute environments

## Required IAM Permissions for Dataset Access

The authenticated principal must possess specific permissions to inspect the target dataset. The source implicitly requires these permissions when calling `list_tables`, `get_dataset`, and `get_table` within methods such as `list_concepts`, `read_concept`, and `sample_rows`.

The caller needs the following permissions on the dataset resource:

- `bigquery.tables.get`
- `bigquery.tables.list`
- `bigquery.datasets.get`

Grant the **BigQuery Data Viewer** role (or a custom role containing these permissions) at the dataset level or higher. Without these permissions, the source raises a `403 Forbidden` error when attempting to list concepts or sample rows.

## Configuring the Billing Project

By default, BigQuery charges API usage to the project associated with the authenticated credentials. The `BigQuerySource` constructor accepts an optional `billing_project` parameter to override this behavior and charge to a different project:

```python
def __init__(self, dataset: str, billing_project: str | None = None):

```

When provided, the `bigquery.Client` is instantiated with this specific project ID, directing all query and API costs to the designated billing project. This configuration is essential when the target dataset resides in a project separate from your budget allocation or when using a central billing account for data catalog operations.

## Dataset Identifier Format Validation

The source enforces strict validation of the `dataset` parameter in `__init__` (lines 32-38 of [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py)). The identifier must follow the `project_id.dataset_id` format. If the input string lacks the project prefix or contains invalid characters, the constructor raises a clear `ValueError` before attempting API calls.

Valid example: `"my-data-project.my_dataset"`

Invalid example: `"my_dataset"` (missing project component)

## Handling Sharded Tables

For tables following the `{prefix}{shard}` naming convention (e.g., `events_20230101`), the source automatically groups them into "family" concepts. The `list_concepts` method uses the `_SHARD_SUFFIX_RE` regular expression (lines 75-101) to detect these patterns and constructs wildcard URIs. This aggregation requires no additional authentication or billing configuration—the existing client credentials apply uniformly to all shards in the family.

## Complete Implementation Example

The following example demonstrates authenticating via ADC, configuring a separate billing project, and listing concepts from a BigQuery dataset:

```python
from reference_agent.sources.bigquery import BigQuerySource

# Authenticate via ADC and charge to the credentials' project

src = BigQuerySource(dataset="my-data-project.my_dataset")

# Or, authenticate via ADC but charge to a specific billing project

src = BigQuerySource(
    dataset="my-data-project.my_dataset",
    billing_project="my-billing-project"
)

# List all concepts (datasets, tables, and sharded families)

concepts = src.list_concepts()
for concept in concepts:
    print(concept.id, concept.type, concept.resource)

```

## Summary

- **Authentication**: The `BigQuerySource` uses Application Default Credentials (ADC) via `bigquery.Client(project=billing_project)` in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py).
- **Permissions**: Grant **BigQuery Data Viewer** (or `bigquery.tables.get`, `bigquery.tables.list`, `bigquery.datasets.get`) on the target dataset to avoid 403 errors.
- **Billing**: Pass the optional `billing_project` parameter to charge API usage to a project other than the one tied to your ADC.
- **Format**: Provide dataset identifiers as `project_id.dataset_id`; validation occurs in the constructor.
- **Sharded Tables**: The source automatically detects and groups sharded tables using `_SHARD_SUFFIX_RE` without extra auth steps.

## Frequently Asked Questions

### What happens if I do not specify a billing project?

If you omit the `billing_project` parameter, the BigQuery client defaults to the project associated with your Application Default Credentials. All API and query costs accrue to that project.

### Which IAM role is required for the BigQuery source?

You must grant the **BigQuery Data Viewer** role (or a custom role with equivalent permissions) on the target dataset. The source requires `bigquery.tables.get`, `bigquery.tables.list`, and `bigquery.datasets.get` permissions to list metadata and sample rows.

### How does the source handle sharded tables with daily or hourly suffixes?

The `list_concepts` method in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py) detects sharded tables using the `_SHARD_SUFFIX_RE` regular expression (lines 75-101). It groups these into family concepts with wildcard URIs, allowing you to treat them as a single logical resource without additional configuration.

### Can I use a service account JSON key instead of gcloud login?

Yes. Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of your service account JSON file. The `bigquery.Client` automatically loads these credentials as part of the ADC chain.