How to Configure dbt with BigQuery for Production Deployments

To configure dbt with BigQuery for production deployments, create a profiles.yml with service-account authentication, reference it from your dbt_project.yml, and execute runs using the --target prod flag while injecting credentials via environment variables.

Production deployments of dbt (data build tool) on Google BigQuery require strict separation of credentials, environments, and execution contexts. This guide walks through the exact configuration patterns used in the DataTalksClub/data-engineering-zoomcamp repository to run dbt models securely against BigQuery in production workloads. You will learn how to structure connection profiles, isolate datasets, and automate deployments without exposing sensitive keys.

Establishing the BigQuery Connection Profile

The foundation of any production dbt setup is the profiles.yml file, which stores connection details outside your project repository. dbt reads this file to authenticate with BigQuery and determine where to materialize models.

Creating the Production Profile

Place your configuration in ~/.dbt/profiles.yml or a custom directory specified with --profiles-dir. For BigQuery production workloads, you must specify type: bigquery and method: service-account to authenticate via JSON service-account keys.

taxi_rides_ny:
  target: prod
  outputs:
    prod:
      type: bigquery
      method: service-account
      project: your-gcp-project-id
      dataset: taxi_rides_prod
      keyfile: "{{ env_var('BIGQUERY_KEYFILE') }}"
      threads: 8
      timeout_seconds: 300
      location: US

This profile configuration references the BIGQUERY_KEYFILE environment variable to locate the service-account JSON, ensuring credentials never reside in version control. The taxi_rides_ny profile name must match the profile field defined in your project's configuration.

Linking the dbt Project Configuration

Your dbt project must explicitly reference the profile defined in profiles.yml. In the DataTalksClub/data-engineering-zoomcamp repository, the 04-analytics-engineering/taxi_rides_ny/dbt_project.yml file sets profile: taxi_rides_ny, establishing the connection between project logic and BigQuery infrastructure.

The project file also enforces production safeguards through require-dbt-version, which locks the dbt version to prevent breaking changes, and the require_generic_test_arguments_property flag, which mandates explicit test arguments. These settings ensure that production runs maintain consistency and quality standards.

Environment Separation Strategy

Production deployments demand strict isolation between development experimentation and production data assets.

Using Targets for Environment Isolation

Define separate targets within your profile—typically dev and prod—each pointing to different BigQuery datasets or projects. The target: prod declaration in your profile specifies which configuration block to use by default, while the --target prod command-line flag explicitly directs dbt to use production-specific credentials and destinations during execution.

Configuring Runtime Variables

Override development variables at runtime to process full historical datasets in production. The repository pattern uses dev_start_date and dev_end_date variables to limit data volumes during development while allowing complete loads in production environments.

Execute a full production run with variable overrides:

dbt run --vars '{"dev_start_date": "2020-01-01", "dev_end_date": "2022-12-31"}' --target prod

Executing Production dbt Runs

Run dbt commands against BigQuery production environments by specifying the target and profiles directory:

export BIGQUERY_KEYFILE=/secrets/bq/key.json
export DBT_PROFILES_DIR=$HOME/.dbt
dbt deps
dbt seed --profiles-dir $DBT_PROFILES_DIR --target prod
dbt run --profiles-dir $DBT_PROFILES_DIR --target prod
dbt test --profiles-dir $DBT_PROFILES_DIR --target prod

This sequence installs dependencies, loads seed data, materializes models as views or tables in the production dataset, and runs tests against your BigQuery warehouse.

CI/CD Pipeline Integration

Automate production deployments using continuous integration pipelines such as GitHub Actions or Cloud Build. Store the service-account key in Google Secret Manager or a comparable vault, inject it during the workflow, and execute dbt with explicit target flags.

A typical production deployment script mounts the secret and runs dbt:

export BIGQUERY_KEYFILE=/path/to/key.json
export DBT_PROFILES_DIR=/path/to/profiles
dbt deps
dbt run --profile taxi_rides_ny --target prod

Production Security Best Practices

Secure production deployments require credential isolation and strict version control.

Credential Management

Never commit service-account keys to version control. Store secrets in Google Secret Manager, HashiCorp Vault, or your CI platform's native secret store. Reference the key file path through the BIGQUERY_KEYFILE environment variable in your profiles.yml, and inject the actual key content only at runtime during CI/CD execution.

Version Locking and Configuration Flags

Lock your dbt version using require-dbt-version in dbt_project.yml to prevent breaking changes from automatic updates. Enable require_generic_test_arguments_property under the flags configuration to enforce explicit test arguments, reducing ambiguity and ensuring consistent test behavior across development and production environments.

Summary

• Configure BigQuery authentication in profiles.yml using the service-account method and environment variables for the keyfile path to keep credentials out of repositories.
• Match the profile name in profiles.yml with the profile field in dbt_project.yml, as implemented in the taxi_rides_ny example.
• Use --target prod to isolate production runs from development environments and direct models to production datasets.
• Override variables at runtime using --vars to switch between sample data for development and full historical datasets for production.
• Store credentials in secret managers and inject them via the BIGQUERY_KEYFILE environment variable during CI/CD execution to maintain security compliance.

Frequently Asked Questions

Where should I store the profiles.yml file in production?

Store profiles.yml outside your project repository, typically in ~/.dbt/ or a dedicated configuration directory. In containerized or CI environments, mount it as a secret volume or generate it dynamically during the workflow, specifying its location using the --profiles-dir flag when running dbt commands.

How do I switch between development and production datasets in BigQuery?

Define separate targets within your profile configuration—one for development pointing to a sample dataset and one for production pointing to your full dataset. Use the --target flag (e.g., --target prod) to specify which environment dbt should execute against during each run, ensuring development queries never write to production tables.

What authentication method should I use for production dbt deployments on BigQuery?

Use the service-account authentication method with a JSON key file. Reference the key file path through an environment variable like BIGQUERY_KEYFILE rather than hardcoding it, and store the actual key file in a secure secret manager rather than in your code repository or Docker images.

How do I handle large data volumes in production versus small samples in development?

Use dbt's vars configuration to define date ranges or sampling limits. Set conservative defaults in dbt_project.yml for development (e.g., recent 30 days), then override these variables at runtime in production using the --vars flag to process full historical datasets without modifying model code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}