# Pathway Rust Engine Underlying Architecture: How Differential Dataflow Powers Incremental Computation

> Explore Pathway's Rust engine architecture. Learn how Differential Dataflow powers incremental computation for low-latency stream processing. Discover the underlying DD collections.

- Repository: [Pathway/pathway](https://github.com/pathwaycom/pathway)
- Tags: architecture
- Published: 2026-03-06

---

**Pathway's Rust engine is a thin abstraction layer built directly on Differential Dataflow, which itself runs on Timely Dataflow, translating high-level Pathway concepts like universes and tables into incremental DD collections for low-latency stream processing.**

Pathway is a high-performance data processing framework that combines Python ergonomics with Rust-level performance. The underlying architecture of Pathway's Rust engine relies on Differential Dataflow (DD) to provide incremental, arranged computation capabilities that enable efficient real-time stream processing and batch analytics.

## The Core Stack: Timely and Differential Dataflow

The engine sits atop two foundational Rust libraries. **Timely Dataflow** provides the low-level data-parallel execution engine, handling workers, threads, and message passing. **Differential Dataflow** adds incremental and arranged collections, allowing Pathway to update results efficiently when new data arrives or old data is retracted.

According to the `pathwaycom/pathway` source code, the bundled DD crate in [`external/differential-dataflow/Cargo.toml`](https://github.com/pathwaycom/pathway/blob/main/external/differential-dataflow/Cargo.toml) declares its dependency on Timely directly:

```toml
[dependencies]
timely = { path = "../timely-dataflow/timely", default-features = false }

```

This dependency chain means every Pathway computation ultimately executes within Timely's worker framework while leveraging DD's difference-tracking capabilities.

## Architectural Layers of the Pathway Rust Engine

The engine implements a thin "graph" abstraction that translates Pathway-level concepts into DD primitives across five distinct layers:

### Execution Runtime Layer

The foundation consists of Timely Dataflow workers that handle scheduling and progress tracking. In [`src/engine/graph.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/graph.rs), the engine defines the `Graph` trait and handle types (`UniverseHandle`, `ColumnHandle`, `TableHandle`) that wrap the underlying Timely runtime. The concrete implementation `DataflowGraphInner<S>` wraps a Timely `Scope` that implements the `Timestamp` trait, creating the execution context for all subsequent operations.

### Incremental Data Model Layer

This layer exposes DD's collections and arrangements to the rest of the system. The file [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs) contains heavy DD imports and implements the core data model:

- `Collection<S, ...>` types represent streams of updates (insertions and deletions)
- `Arrange` and `Trace` types provide indexed views for fast lookups
- Automatic consolidation happens via DD's `concatenate` and `join_core` primitives

### Pathway Abstraction Layer

Pathway-specific concepts map directly onto DD collections. In [`src/engine/graph.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/graph.rs), the structs `Universe`, `Column`, and `Table` translate into DD collections of keys and values. Operators like `join`, `reduce`, and `filter` become thin wrappers around corresponding DD operators, allowing Python-level API calls to execute as optimized Rust dataflows.

### Persistence and Fault Tolerance

The engine persists intermediate DD state using wrapper traits defined in [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs). The `MaybePersist` and `PersistenceWrapper` traits enable saving arranged collections to external storage, then replaying those collections through DD when a job restarts. This leverages DD's native ability to reconstitute state from compacted traces.

### External Index Integration

For fast point-lookups against external systems, the engine uses DD's ability to treat external streams as arranged collections. The function `use_external_index_as_of_now` in [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs) creates arranged views of remote data sources, enabling efficient joins without loading entire datasets into memory.

## The Computational Loop: From Timely Workers to DD Collections

The core architectural workflow follows a strict initialization pattern:

1. **Create a Timely worker** via `timely::execute`, supplying a `Scope` that implements the `Timestamp` trait
2. **Wrap the scope** in a `DataflowGraphInner<S>` (the concrete graph implementation from [`src/engine/graph.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/graph.rs))
3. **Allocate universes and columns** — each becomes a DD collection (`Collection<S, Key>` or `Collection<S, (Key, Value)>`)
4. **Compose operators** using DD primitives (`join_core`, `reduce`, `map_wrapped_named`) to implement Pathway operations
5. **Run the graph** — DD automatically maintains arrangements (indexed versions of collections) and propagates only minimal deltas when input data changes

This loop executes continuously, with DD's difference tracking ensuring only affected computations re-run when new data arrives.

## Why Differential Dataflow?

Pathway chose DD for three specific architectural advantages:

- **Incrementality** — DD tracks differences via the `Diff` type, recomputing only affected sub-graphs rather than full re-computation
- **Arrangements** — Indexed views enable fast joins and lookups without full shuffles across workers, critical for low-latency results
- **Temporal semantics** — Pathway's `Timestamp` type in [`src/engine/timestamp.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/timestamp.rs) implements DD's `Lattice` trait, supporting event-time processing, watermarks, and time-column operators with proper partial ordering

## Code Examples: Working with the Engine

### Creating Universes and Columns

The following pattern allocates DD collections through Pathway's graph abstraction:

```rust
use pathway::engine::{Graph, UniverseHandle, ColumnHandle, Timestamp};

fn example_create_universe_and_column<G: Graph>(graph: &mut G) -> Result<(UniverseHandle, ColumnHandle)> {
    // Create an empty universe (set of keys)
    let universe = graph.empty_universe()?;

    // Populate a column with (key, value) pairs
    let values = vec![
        (Key::from_u64(1), Value::from(42_i64)),
        (Key::from_u64(2), Value::from(7_i64)),
    ];
    let column_props = Arc::new(ColumnProperties {
        dtype: Type::Int64,
        append_only: false,
        trace: Arc::new(Trace::Empty),
    });
    let column = graph.empty_column(universe, column_props)?;

    Ok((universe, column))
}

```

Key point: `empty_universe` and `empty_column` allocate DD collections and register them with internal `Arena` handles, hiding the low-level DD code while maintaining zero-cost abstractions.

### Adding Expression Operators

Custom expressions exposed to Python flow through the graph as mapped collections:

```rust
use pathway::engine::{Graph, Expression, ColumnHandle, ColumnProperties};

fn add_expression_column<G: Graph>(graph: &mut G,
                                   universe: UniverseHandle,
                                   input_cols: &[ColumnHandle],
                                   expr: Expression) -> Result<ColumnHandle> {
    let props = Arc::new(ColumnProperties {
        dtype: Type::Float64,
        append_only: false,
        trace: Arc::new(Trace::Empty),
    });
    graph.expression_column(
        BatchWrapper::None,
        Arc::new(expr),
        universe,
        input_cols,
        props,
    )
}

```

Internally, `engine::dataflow::expression_column` builds a tuple collection from inputs, maps each tuple through `expression.eval(...)`, and creates a new column collection from the results using DD's `map` primitives.

### Performing Joins with Arrangements

Joins leverage DD's arranged collection API for efficient execution:

```rust
fn join_two_tables<G: Graph>(graph: &mut G,
                             left: TableHandle,
                             right: TableHandle,
                             left_path: ColumnPath,
                             right_path: ColumnPath) -> Result<TableHandle> {
    let left_data = JoinData::new(left, vec![left_path]);
    let right_data = JoinData::new(right, vec![right_path]);
    let shard_policy = ShardPolicy::RoundRobin;
    let join_type = JoinType::Inner;

    graph.join_tables(
        left_data,
        right_data,
        shard_policy,
        join_type,
        JoinExactlyOnce::new(false, false),
        Arc::new(TableProperties::Empty),
    )
}

```

This call resolves to `DataflowGraphInner::join_tables` in [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs), which uses DD's `join_core` on `ArrangedByKey` collections to avoid expensive re-shuffling of data.

## Summary

- **Pathway's Rust engine** is a thin layer on top of Differential Dataflow, which runs on Timely Dataflow
- **Five architectural layers** handle execution runtime, incremental data models, Pathway abstractions, persistence, and external indexes
- **Core files** include [`src/engine/graph.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/graph.rs) (abstractions), [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs) (DD integration), and [`src/engine/timestamp.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/timestamp.rs) (temporal semantics)
- **Computation follows a strict loop**: Timely worker creation → graph wrapping → collection allocation → operator composition → incremental execution
- **Key DD features** utilized include difference tracking (`Diff`), arrangements for fast joins, and lattice-based timestamps for event-time processing

## Frequently Asked Questions

### How does Pathway's Rust engine handle incremental updates?

The engine relies on Differential Dataflow's difference tracking capabilities. When new data arrives or old data is retracted, DD's `Diff` type tracks the magnitude of change, and the system recomputes only the affected subgraphs rather than re-running the entire computation. This happens automatically through arranged collections maintained in [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs).

### What is the relationship between Timely Dataflow and Differential Dataflow in Pathway?

Timely Dataflow provides the low-level execution engine handling workers and scheduling, while Differential Dataflow adds the incremental computation model on top. Pathway's engine creates Timely workers first, then wraps them in a `DataflowGraphInner` that provides Differential Dataflow's collection types and operators, as seen in [`src/engine/graph.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/graph.rs) and [`external/differential-dataflow/Cargo.toml`](https://github.com/pathwaycom/pathway/blob/main/external/differential-dataflow/Cargo.toml).

### How are Pathway tables and columns represented in the underlying engine?

Pathway concepts map directly to Differential Dataflow primitives. A `Universe` becomes a `Collection<S, Key>`, while a `Column` becomes a `Collection<S, (Key, Value)>`. These mappings occur in [`src/engine/graph.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/graph.rs), where the `Graph` trait defines methods like `empty_universe()` and `empty_column()` that allocate the corresponding DD collections.

### Where does the engine handle persistence and fault tolerance?

Persistence logic resides in [`src/engine/dataflow.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/dataflow.rs) through traits like `MaybePersist` and `PersistenceWrapper`. These wrappers save intermediate DD state (particularly arranged collections) to external storage. Upon restart, the engine replays these persisted traces through Differential Dataflow, allowing the computation to resume from its exact previous state without reprocessing historical data from source.