# How to Configure dbt with BigQuery for Production Deployments

> Learn to configure dbt with BigQuery for production. Set up service-account authentication and reference your profile for smooth deployments.

- Repository: [DataTalksClub/data-engineering-zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
- Tags: how-to-guide
- Published: 2026-05-31

---

**To configure dbt with BigQuery for production deployments, create a [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml) with service-account authentication, reference it from your [`dbt_project.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/dbt_project.yml), and execute runs using the `--target prod` flag while injecting credentials via environment variables.**

Production deployments of dbt (data build tool) on Google BigQuery require strict separation of credentials, environments, and execution contexts. This guide walks through the exact configuration patterns used in the DataTalksClub/data-engineering-zoomcamp repository to run dbt models securely against BigQuery in production workloads. You will learn how to structure connection profiles, isolate datasets, and automate deployments without exposing sensitive keys.

## Establishing the BigQuery Connection Profile

The foundation of any production dbt setup is the [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml) file, which stores connection details outside your project repository. dbt reads this file to authenticate with BigQuery and determine where to materialize models.

### Creating the Production Profile

Place your configuration in `~/.dbt/profiles.yml` or a custom directory specified with `--profiles-dir`. For BigQuery production workloads, you must specify `type: bigquery` and `method: service-account` to authenticate via JSON service-account keys.

```yaml
taxi_rides_ny:
  target: prod
  outputs:
    prod:
      type: bigquery
      method: service-account
      project: your-gcp-project-id
      dataset: taxi_rides_prod
      keyfile: "{{ env_var('BIGQUERY_KEYFILE') }}"
      threads: 8
      timeout_seconds: 300
      location: US

```

This profile configuration references the `BIGQUERY_KEYFILE` environment variable to locate the service-account JSON, ensuring credentials never reside in version control. The `taxi_rides_ny` profile name must match the `profile` field defined in your project's configuration.

## Linking the dbt Project Configuration

Your dbt project must explicitly reference the profile defined in [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml). In the DataTalksClub/data-engineering-zoomcamp repository, the [`04-analytics-engineering/taxi_rides_ny/dbt_project.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/04-analytics-engineering/taxi_rides_ny/dbt_project.yml) file sets `profile: taxi_rides_ny`, establishing the connection between project logic and BigQuery infrastructure.

The project file also enforces production safeguards through `require-dbt-version`, which locks the dbt version to prevent breaking changes, and the `require_generic_test_arguments_property` flag, which mandates explicit test arguments. These settings ensure that production runs maintain consistency and quality standards.

## Environment Separation Strategy

Production deployments demand strict isolation between development experimentation and production data assets.

### Using Targets for Environment Isolation

Define separate targets within your profile—typically `dev` and `prod`—each pointing to different BigQuery datasets or projects. The `target: prod` declaration in your profile specifies which configuration block to use by default, while the `--target prod` command-line flag explicitly directs dbt to use production-specific credentials and destinations during execution.

### Configuring Runtime Variables

Override development variables at runtime to process full historical datasets in production. The repository pattern uses `dev_start_date` and `dev_end_date` variables to limit data volumes during development while allowing complete loads in production environments.

Execute a full production run with variable overrides:

```bash
dbt run --vars '{"dev_start_date": "2020-01-01", "dev_end_date": "2022-12-31"}' --target prod

```

## Executing Production dbt Runs

Run dbt commands against BigQuery production environments by specifying the target and profiles directory:

```bash
export BIGQUERY_KEYFILE=/secrets/bq/key.json
export DBT_PROFILES_DIR=$HOME/.dbt
dbt deps
dbt seed --profiles-dir $DBT_PROFILES_DIR --target prod
dbt run --profiles-dir $DBT_PROFILES_DIR --target prod
dbt test --profiles-dir $DBT_PROFILES_DIR --target prod

```

This sequence installs dependencies, loads seed data, materializes models as views or tables in the production dataset, and runs tests against your BigQuery warehouse.

## CI/CD Pipeline Integration

Automate production deployments using continuous integration pipelines such as GitHub Actions or Cloud Build. Store the service-account key in Google Secret Manager or a comparable vault, inject it during the workflow, and execute dbt with explicit target flags.

A typical production deployment script mounts the secret and runs dbt:

```bash
export BIGQUERY_KEYFILE=/path/to/key.json
export DBT_PROFILES_DIR=/path/to/profiles
dbt deps
dbt run --profile taxi_rides_ny --target prod

```

## Production Security Best Practices

Secure production deployments require credential isolation and strict version control.

### Credential Management

Never commit service-account keys to version control. Store secrets in Google Secret Manager, HashiCorp Vault, or your CI platform's native secret store. Reference the key file path through the `BIGQUERY_KEYFILE` environment variable in your [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml), and inject the actual key content only at runtime during CI/CD execution.

### Version Locking and Configuration Flags

Lock your dbt version using `require-dbt-version` in [`dbt_project.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/dbt_project.yml) to prevent breaking changes from automatic updates. Enable `require_generic_test_arguments_property` under the `flags` configuration to enforce explicit test arguments, reducing ambiguity and ensuring consistent test behavior across development and production environments.

## Summary

- Configure **BigQuery authentication** in [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml) using the `service-account` method and environment variables for the `keyfile` path to keep credentials out of repositories.
- Match the **profile name** in [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml) with the `profile` field in [`dbt_project.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/dbt_project.yml), as implemented in the `taxi_rides_ny` example.
- Use **`--target prod`** to isolate production runs from development environments and direct models to production datasets.
- **Override variables** at runtime using `--vars` to switch between sample data for development and full historical datasets for production.
- **Store credentials** in secret managers and inject them via the `BIGQUERY_KEYFILE` environment variable during CI/CD execution to maintain security compliance.

## Frequently Asked Questions

### Where should I store the [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml) file in production?

Store [`profiles.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/profiles.yml) outside your project repository, typically in `~/.dbt/` or a dedicated configuration directory. In containerized or CI environments, mount it as a secret volume or generate it dynamically during the workflow, specifying its location using the `--profiles-dir` flag when running dbt commands.

### How do I switch between development and production datasets in BigQuery?

Define separate targets within your profile configuration—one for development pointing to a sample dataset and one for production pointing to your full dataset. Use the `--target` flag (e.g., `--target prod`) to specify which environment dbt should execute against during each run, ensuring development queries never write to production tables.

### What authentication method should I use for production dbt deployments on BigQuery?

Use the `service-account` authentication method with a JSON key file. Reference the key file path through an environment variable like `BIGQUERY_KEYFILE` rather than hardcoding it, and store the actual key file in a secure secret manager rather than in your code repository or Docker images.

### How do I handle large data volumes in production versus small samples in development?

Use dbt's `vars` configuration to define date ranges or sampling limits. Set conservative defaults in [`dbt_project.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/dbt_project.yml) for development (e.g., recent 30 days), then override these variables at runtime in production using the `--vars` flag to process full historical datasets without modifying model code.