dbt Sources and Ref Functions: Understanding the Difference in Data Engineering

dbt source() functions reference raw external tables outside dbt's control while ref() functions reference models built by dbt itself, creating distinct dependency patterns and DAG behaviors in your data warehouse.

The DataTalksClub/data-engineering-zoomcamp repository demonstrates how dbt sources and ref functions serve complementary yet distinct roles in building maintainable data pipelines. While both functions resolve to fully qualified table names at compile time, they differ fundamentally in what they reference, how they impact the directed acyclic graph (DAG), and what dependencies they create.

Core Concepts: External Data vs. dbt Models

What source() References

The source() function points to raw tables or views that exist outside dbt's managed layer, typically landing tables loaded by external ETL pipelines or ingestion processes. These sources are declared in schema.yml files and serve as immutable entry points to your pipeline.

Syntax:

{{ source('source_name', 'table_name') }}

In the Zoomcamp project, staging models use source() to ingest NYC taxi data from the raw schema without assuming ownership of those tables.

What ref() References

The ref() function points to models that are defined and materialized by dbt itself, creating explicit dependencies between SQL files in your project. This enables dbt to track lineage and ensure proper execution order.

Syntax:

{{ ref('model_name') }}

When you use ref('stg_yellow_tripdata'), dbt establishes a dependency edge, guaranteeing that model builds before any downstream model referencing it.

Critical Differences Between source() and ref()

Dependency Management and DAG Impact

ref() creates DAG nodes and edges, establishing dependencies that dictate build order. If fct_trips references int_trips via ref(), dbt knows to materialize the intermediate model first.

source() creates leaf nodes that dbt assumes already exist. These do not trigger builds but serve as starting points for your pipeline, enabling dbt to distinguish external data from internally managed transformations.

Configuration and Monitoring Capabilities

Sources support freshness checks through schema.yml configurations, allowing you to monitor when external data was last loaded and alert on stale data. References inherit their configuration from the referenced model's properties and support change tracking through model hashing.

Compilation and Caching Behavior

dbt tracks model hashes for ref() functions, triggering recompilation of downstream models when upstream logic changes. source() functions perform no caching—dbt simply injects the quoted table name at compile time without tracking changes to the underlying external table.

Practical Implementation in DataTalksClub/data-engineering-zoomcamp

Ingesting Raw Data with source()

Staging models in 04-analytics-engineering/taxi_rides_ny/models/staging/ demonstrate proper source() usage for external taxi trip data:

-- 04-analytics-engineering/taxi_rides_ny/models/staging/stg_yellow_tripdata.sql
select 
    *,
    {{ safe_cast('passenger_count', 'INTEGER') }} as passenger_cnt
from {{ source('raw', 'yellow_tripdata') }}

Similarly, the green taxi staging model uses:

-- 04-analytics-engineering/taxi_rides_ny/models/staging/stg_green_tripdata.sql
select *
from {{ source('raw', 'green_tripdata') }}

These statements tell dbt to use existing tables raw.yellow_tripdata and raw.green_tripdata as immutable inputs without attempting to recreate them.

Building Model Dependencies with ref()

Intermediate models transition from source() to ref() as data enters dbt's control. The int_trips_unioned.sql model combines staged tables using references:

-- models/intermediate/int_trips_unioned.sql
select * from {{ ref('stg_green_tripdata') }}
union all
select * from {{ ref('stg_yellow_tripdata') }}

Downstream mart models like fct_trips.sql establish complex dependency chains:

-- 04-analytics-engineering/taxi_rides_ny/models/marts/fct_trips.sql
from {{ ref('int_trips') }} as trips
left join {{ ref('dim_zones') }} as pz
left join {{ ref('dim_zones') }} as dz

Here, ref('int_trips') resolves to the compiled name of the intermediate model, ensuring dbt builds int_trips before fct_trips.

Best Practices for dbt Sources and Ref Functions

When to Use source()

Use source() exclusively in staging models that represent the first dbt touchpoint for external data. Define these sources in schema.yml files with freshness blocks to enable monitoring and documentation of upstream systems.

When to Use ref()

Use ref() for all inter-model dependencies after the staging layer. Never hardcode table names in production models—always use ref() to maintain lineage, enable automatic dependency resolution, and support environment-specific naming.

Summary

• source() references external raw tables outside dbt's control, creating leaf nodes in the DAG that support freshness checks and external documentation.
• ref() references dbt-managed models, creating explicit dependencies that ensure correct build order and enable automatic recompilation when upstream logic changes.
• The DataTalksClub/data-engineering-zoomcamp project uses source() in stg_yellow_tripdata.sql and stg_green_tripdata.sql to ingest raw NYC taxi data from external schemas.
• Downstream models like fct_trips.sql and int_trips_unioned.sql use ref() to build sophisticated transformation pipelines with clear lineage and dependency management.

Frequently Asked Questions

Can I use source() to reference another dbt model?

No. source() specifically references tables defined in your sources configuration within schema.yml files, pointing to external data outside dbt's scope. To reference a dbt model, you must use ref() to ensure proper dependency tracking and DAG visualization.

How does ref() handle model name resolution?

The ref() function accepts the model filename without the .sql extension as its argument. At compile time, dbt resolves this to the fully qualified table name based on your target schema and database configurations, ensuring portability across development, staging, and production environments.

Do sources appear in the dbt DAG visualization?

Yes, sources appear as green nodes in the DAG interface, but they serve as leaf nodes without upstream dependencies within dbt. They represent the entry points of your pipeline, visually distinguishing external data sources from blue dbt-transformed models.

What happens if I hardcode a table name instead of using source()?

Hardcoding table names bypasses dbt's source monitoring, documentation, and freshness checking features. You lose the ability to track when external data was last loaded and the visual distinction between external data and dbt models in your lineage graphs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}