How to Set Up CI/CD Pipelines for dbt Projects: A Complete Guide

Automate your dbt workflows by implementing a GitHub Actions pipeline that installs dependencies, validates connections, builds models, runs tests, and generates documentation on every push or pull request.

Setting up robust CI/CD pipelines for dbt projects ensures that every change to your analytics code is automatically validated before reaching production. The DataTalksClub/data-engineering-zoomcamp repository provides a complete reference implementation in the 04-analytics-engineering/taxi_rides_ny/ directory, demonstrating how to transform a local dbt development workflow into an automated, production-grade deployment process.

Understanding the dbt Project Structure

Before implementing CI/CD, you must understand the key components of the Zoomcamp dbt project. The repository organizes its analytics engineering code under 04-analytics-engineering/taxi_rides_ny/, containing several critical configuration files that your pipeline must respect.

The dbt_project.yml file defines core project settings including model paths and target configurations. External dependencies are declared in packages.yml, which specifies packages like dbt_utils that must be installed via dbt deps before building. Model configurations and tests reside in models/**/schema.yml files, particularly in models/marts/schema.yml, while seed data sits in the seeds/ directory.

Building the GitHub Actions Workflow

Create a new file at .github/workflows/dbt.yml in your repository root. This workflow automates the entire dbt lifecycle from dependency installation through documentation generation, executing commands within the 04-analytics-engineering/taxi_rides_ny working directory.

# .github/workflows/dbt.yml

name: dbt CI/CD

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  dbt:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: 04-analytics-engineering/taxi_rides_ny

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dbt‑duckdb
        run: |
          python -m pip install --upgrade pip
          pip install dbt-duckdb

      - name: Install project packages
        run: dbt deps

      - name: Verify dbt profile
        run: dbt debug

      - name: Build & test dbt models
        run: dbt build --target prod

      - name: Generate dbt docs
        run: dbt docs generate

      - name: Upload docs as artifact
        uses: actions/upload-artifact@v4
        with:
          name: dbt-docs
          path: target/

Configuring Triggers and Environment

Set the workflow to trigger on pushes and pull requests to the main branch. The defaults.run.working-directory must point to 04-analytics-engineering/taxi_rides_ny so that all subsequent dbt commands execute from the correct project root containing dbt_project.yml.

Installing Python and dbt Dependencies

Use actions/setup-python@v5 with Python 3.11 to ensure reproducible builds. Install the dbt-duckdb adapter to match the Zoomcamp project's dependencies, then run dbt deps to resolve external packages defined in packages.yml.

Validation and Build Execution

Execute dbt debug to verify your profile configuration at ~/.dbt/profiles.yml can connect to the target database. Then run dbt build --target prod, which orchestrates run, test, and snapshot commands in the correct sequence, ensuring all models compile and pass schema tests defined in your schema.yml files.

Documentation and Artifact Management

Generate project documentation with dbt docs generate, which creates HTML files in the target/ directory. Upload these as artifacts using actions/upload-artifact@v4 to make lineage graphs and model documentation accessible directly from the workflow run.

Alternative Orchestration with Kestra

If you prefer a cloud-native workflow engine over GitHub Actions, the repository includes reference implementations in the flows/ directory. The files flows/03_postgres_dbt.yaml and flows/07_gcp_dbt.yaml demonstrate how to execute identical dbt commands within Kestra orchestration flows, supporting both PostgreSQL and BigQuery/GCP targets while maintaining the same validation and build logic.

Summary

• Automate validation: Use dbt debug in CI to catch profile misconfigurations early before attempting builds.
• Use dbt build: This single command runs models, tests, and snapshots in the correct dependency order, simplifying your pipeline steps.
• Pin your environment: Install specific adapter versions like dbt-duckdb and use dbt deps to ensure external packages from packages.yml are consistently resolved.
• Generate documentation: Automatically create and archive dbt docs as workflow artifacts to maintain up-to-date project documentation without manual intervention.
• Consider alternatives: The Zoomcamp repository also provides Kestra flow examples for teams using dedicated orchestration platforms rather than GitHub Actions.

Frequently Asked Questions

What is the difference between dbt run and dbt build in a CI/CD pipeline?

dbt build executes run, test, and snapshot commands in the correct dependency order, ensuring that models are built and immediately validated against tests defined in your schema.yml files. Using dbt build --target prod in CI guarantees that any failing tests will halt the pipeline, preventing broken models from reaching production, whereas dbt run alone skips test validation.

How do I handle database credentials securely in GitHub Actions?

Store sensitive connection details as encrypted repository secrets and reference them in your workflow using ${{ secrets.SECRET_NAME }}. The dbt profile configuration at ~/.dbt/profiles.yml can be dynamically generated from these secrets during the workflow execution, or you can use environment variables that dbt automatically picks up for authentication parameters.

Can I use this pipeline with BigQuery instead of DuckDB?

Yes, simply replace dbt-duckdb with dbt-bigquery in the pip install step and ensure your profiles.yml contains a BigQuery connection configuration. The dbt build and dbt test commands remain identical regardless of the underlying warehouse, making the pipeline adapter-agnostic as long as the correct dbt package is installed.

How do I set up scheduled production deployments instead of push-based triggers?

Modify the on: section of your workflow file to include a schedule trigger using cron syntax: schedule: - cron: '0 2 * * *' would run daily at 2 AM UTC. Combine this with environment protection rules requiring manual approval for production targets to ensure scheduled runs don't automatically deploy without review gates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}