How to Integrate dbt with Airbyte for ELT Pipelines: A Complete 9-Step Guide

Integrating dbt with Airbyte creates a modern ELT pipeline where Airbyte handles extraction and loading into your warehouse's raw schema, while dbt transforms that data into analytics-ready models using SQL-based logic and dependency management.

The DataExpert-io/data-engineer-handbook identifies both Airbyte and dbt as foundational tools in the modern data stack. When you integrate dbt with Airbyte for ELT pipelines, you separate data ingestion from transformation, allowing Airbyte to focus on reliable data movement while dbt manages business logic, testing, and documentation.

Understanding the ELT Architecture

Modern ELT pipelines follow a strict two-step pattern that leverages the data warehouse as the transformation engine.

Step 1: Extract and Load (EL)

Airbyte pulls data from source systems and lands it directly into a destination data warehouse such as Snowflake, BigQuery, or Postgres. According to the handbook's Data Integration section in README.md (line 106), Airbyte serves as the core data-integration tool, offering a broad catalog of connectors that eliminate the need for custom extraction code. Airbyte handles scheduling, incremental loading via Change Data Capture (CDC), and schema evolution.

Step 2: Transform (T)

Once raw tables populate the warehouse, dbt takes over. As noted in the handbook's Data Quality section in README.md (line 76), dbt enables analysts to write transformation logic in plain SQL, manage dependencies through the ref() function, and version-control models. dbt provides data quality testing, automatic documentation generation, and lineage visualization.

The integration succeeds because both tools operate on the same destination schema. Airbyte’s sync jobs populate tables with a raw_ prefix (e.g., raw_customers), while dbt models read these tables, apply business logic, and materialize results as clean, aggregated tables (e.g., dim_customers) in a separate analytics schema.

Typical data flow:

Source → Airbyte connectors → Destination (raw schema) → dbt models → Destination (analytics schema)

Step-by-Step Integration Guide

Phase 1: Configure Airbyte Extraction

1. Install and Configure Airbyte Deploy Airbyte via Docker or use the hosted Cloud version. Create a connection by selecting your source (e.g., MySQL, Salesforce) and destination (e.g., Snowflake). Set the sync mode to Full Refresh for initial loads or Incremental for ongoing replication.

2. Define the Raw Schema Isolate inbound data by configuring a dedicated "raw" schema. In the Airbyte UI, set the Destination Namespace to raw (or raw_<source_name>). This prevents extraction jobs from overwriting production analytics tables.

3. Execute the Initial Sync Run the connection to verify that Airbyte writes tables with the expected raw_ prefix (e.g., raw_orders, raw_users). Confirm these tables exist in your warehouse before proceeding to transformation logic.

Phase 2: Configure dbt Transformation

4. Scaffold the dbt Project Initialize a new dbt project that targets the same warehouse where Airbyte lands data:

dbt init my_project && cd my_project

5. Configure Database Profiles Edit profiles.yml to point to your warehouse. Store credentials in environment variables—never hard-code secrets. Specify separate schemas for development and production:

my_project:
  target: dev
  outputs:
    dev:
      type: snowflake
      account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
      user: "{{ env_var('SNOWFLAKE_USER') }}"
      password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
      role: "{{ env_var('SNOWFLAKE_ROLE') }}"
      database: "{{ env_var('SNOWFLAKE_DATABASE') }}"
      warehouse: "{{ env_var('SNOWFLAKE_WAREHOUSE') }}"
      schema: analytics
      threads: 4

6. Build Transformation Models Create SQL models that reference the raw tables using the {{ ref() }} function. This establishes the dependency graph and ensures models run in the correct order.

-- models/dim_customers.sql
with source as (
    select *
    from {{ ref('raw_customers') }}
),
cleaned as (
    select
        id as customer_id,
        lower(trim(first_name)) as first_name,
        lower(trim(last_name)) as last_name,
        email,
        case when is_active = 'true' then true else false end as is_active,
        cast(created_at as timestamp) as created_at
    from source
)

select *
from cleaned

Phase 3: Orchestrate and Automate

7. Execute dbt Runs Run dbt build (which executes dbt run followed by dbt test) to create transformed tables and verify data quality assertions. This command materializes models in the analytics schema configured in profiles.yml.

8. Add Testing and Documentation Implement schema tests (uniqueness, not-null constraints) and custom data tests in .yml files. Generate documentation and lineage diagrams:

dbt docs generate && dbt docs serve

9. Automate the Workflow Connect Airbyte and dbt through an orchestrator to ensure dbt runs only after successful Airbyte syncs. Use Airbyte’s Jobs API to trigger syncs programmatically, then invoke dbt:

import requests
import os

AIRBYTE_URL = "https://your-airbyte-instance/api/v1"
HEADERS = {"Content-Type": "application/json"}

# Trigger Airbyte sync

resp = requests.post(
    f"{AIRBYTE_URL}/connections/sync",
    headers=HEADERS,
    json={"connectionId": os.getenv("AIRBYTE_CONNECTION_ID")}
)

# After successful sync, trigger dbt (via CLI or API)

# dbt run --target prod

Sample Airbyte Connection Configuration

When defining connections via Airbyte’s API or UI, specify incremental sync modes to minimize warehouse load:

{
  "sourceId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "destinationId": "d9e8f7g6-5432-10ba-fedc-0987654321cd",
  "connectionId": "c0ffee00-1234-5678-9abc-def012345678",
  "name": "MySQL → Snowflake",
  "syncCatalog": {
    "streams": [
      {
        "stream": {
          "name": "customers",
          "namespace": "public"
        },
        "config": {
          "syncMode": "incremental",
          "destinationSyncMode": "append"
        }
      }
    ]
  },
  "schedule": {
    "units": 1,
    "timeUnit": "hours"
  }
}

Key Resources from the Data Engineer Handbook

The DataExpert-io/data-engineer-handbook provides authoritative references for both tools:

• README.md (line 106) – Lists Airbyte as a core data-integration tool for extraction and loading workflows.
• README.md (line 76) – Identifies dbt as the standard for data quality and transformation logic.
• communities.md – Contains links to the dbt Community Slack and forums for troubleshooting transformation logic.
• books.md – References dbt-focused literature covering advanced testing strategies and warehouse optimization.

Summary

• Separation of concerns allows Airbyte to specialize in reliable data ingestion while dbt handles deterministic transformations.
• Schema isolation via raw_ prefixes prevents extraction jobs from interfering with production analytics tables.
• Dependency management through dbt’s ref() function ensures transformations execute only after source data arrives.
• Observability combines Airbyte’s sync logs with dbt’s built-in testing and lineage documentation for end-to-end visibility.
• Scalability permits independent scaling of Airbyte workers for high-volume sources and dbt Cloud for distributed model execution.

Frequently Asked Questions

What is the difference between ETL and ELT when using Airbyte and dbt?

In traditional ETL, transformation happens in a dedicated processing engine before loading to the warehouse. With Airbyte and dbt, you use ELT: Airbyte performs the Extract and Load phases directly into the warehouse, then dbt handles Transform using the warehouse's compute power. This approach reduces data movement and allows analysts to transform data using SQL rather than proprietary scripting languages.

How does dbt know when Airbyte has finished loading data?

dbt does not inherently monitor Airbyte sync status. You must orchestrate the dependency using a workflow tool (such as Prefect, Dagster, or Airflow) or Airbyte’s API. The orchestrator triggers a dbt run only after receiving confirmation that the Airbyte sync completed successfully, preventing dbt from processing stale or incomplete data.

Can I use Airbyte's built-in orchestrator to trigger dbt jobs?

Airbyte Cloud offers basic scheduling for sync jobs, but it does not natively execute dbt commands. For tight integration, use Airbyte’s API to trigger syncs from your orchestration layer, then invoke dbt CLI commands upon completion. Alternatively, some data platforms offer native integrations that combine both steps in a single pipeline configuration.

What naming conventions should I use for raw tables ingested by Airbyte?

Adopt a consistent prefix such as raw_ or source_ followed by the table name (e.g., raw_customers, raw_orders). Store these in a dedicated schema (e.g., raw or staging) separate from your analytics schema. This convention makes dependencies explicit in dbt models when referencing {{ ref('raw_customers') }} and prevents naming collisions with transformed tables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}