How Kestra Handles Task Dependencies and DAG Execution: A Complete Guide

Kestra models workflows as Directed Acyclic Graphs (DAGs) where tasks declare dependencies implicitly through YAML ordering or explicitly via the dependsOn field, enabling sequential, parallel, and conditional execution patterns.

Kestra's workflow engine powers the orchestration lessons in the DataTalksClub/data-engineering-zoomcamp repository, demonstrating production-grade patterns for data pipelines. Understanding how Kestra manages task dependencies and DAG execution is essential for building reliable, scalable data workflows that handle complex branching logic and parallel processing.

Understanding Kestra's DAG Architecture

Kestra treats every workflow as a Directed Acyclic Graph (DAG) where individual tasks represent nodes. The engine computes execution order by resolving dependency edges between nodes, ensuring that downstream tasks only start after their prerequisites complete successfully.

Implicit Sequential Execution

When tasks appear sequentially in the YAML file without explicit dependency declarations, Kestra automatically treats each preceding task as a prerequisite for the next. This implicit ordering creates a simple linear chain ideal for basic ETL pipelines.

The file 02-workflow-orchestration/flows/04_postgres_taxi.yaml demonstrates this pattern with a clear chain:

tasks:
  - id: extract
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - wget https://example.com/taxi_data.csv

  - id: if_yellow_taxi
    type: io.kestra.plugin.core.flow.If
    condition: "{{ inputs.taxi_type == 'yellow' }}"
    then:
      - id: process_yellow
        type: io.kestra.plugin.jdbc.postgresql.CopyIn

  - id: purge_files
    type: io.kestra.plugin.core.storage.Delete
    files: ["{{ outputs.extract.outputFiles['taxi_data.csv'] }}"]

In this flow from the Data Engineering Zoomcamp, the purge_files task implicitly waits for if_yellow_taxi to finish, which itself waits for extract, creating a sequential dependency chain without explicit dependsOn declarations.

Explicit Dependencies with dependsOn

For complex workflows requiring parallel branches or non-linear execution paths, Kestra provides the dependsOn field. This parameter accepts a list of task IDs that must complete successfully before the current task starts.

When a task lists multiple upstream dependencies in dependsOn, the engine applies an AND condition—the task only runs when all specified dependencies succeed. This enables sophisticated DAG patterns where a single task aggregates outputs from multiple parallel branches.

id: parallel_etl
namespace: zoomcamp
tasks:
  - id: extract_customers
    type: io.kestra.plugin.scripts.shell.Commands
    commands: 
      - echo "Extracting customer data" > customers.csv

  - id: extract_orders
    type: io.kestra.plugin.scripts.shell.Commands
    commands: 
      - echo "Extracting order data" > orders.csv

  - id: join_datasets
    type: io.kestra.plugin.scripts.shell.Commands
    dependsOn: [extract_customers, extract_orders]
    commands:
      - echo "Joining data" && cat customers.csv orders.csv > joined.csv

In this example, join_datasets waits for both extraction tasks to finish. Because extract_customers and extract_orders have no mutual dependencies, Kestra schedules them concurrently, reducing total execution time.

Execution Patterns in Kestra

Parallel Task Execution

Kestra's worker pool automatically identifies tasks with satisfied dependencies and no mutual conflicts, scheduling them for concurrent execution. The engine maintains a ready set—a collection of tasks whose dependencies are all complete—and dispatches them to available workers immediately.

This parallelism is particularly evident in the 02-workflow-orchestration/flows/06_gcp_taxi.yaml file, where independent cloud operations can run simultaneously. The engine computes the ready set dynamically during runtime, adjusting to task completion times without manual intervention.

Conditional Branching

The io.kestra.plugin.core.flow.If plugin enables dynamic DAG modification based on runtime conditions. When a condition evaluates to true, Kestra injects the then branch into the execution graph; otherwise, it executes the else branch (if specified).

Tasks within conditional blocks inherit the parent flow's dependency context. In 04_postgres_taxi.yaml, the if_yellow_taxi and if_green_taxi blocks demonstrate how conditional logic integrates with the DAG structure—only the matching branch executes, but both branches maintain proper dependency links to upstream tasks.

id: conditional_pipeline
namespace: zoomcamp
inputs:
  - id: pipeline_mode
    type: SELECT
    values: [full, incremental]

tasks:
  - id: check_input
    type: io.kestra.plugin.core.flow.If
    condition: "{{ inputs.pipeline_mode == 'full' }}"
    then:
      - id: full_extract
        type: io.kestra.plugin.scripts.shell.Commands
        commands: ["python full_load.py"]
    else:
      - id: incremental_extract
        type: io.kestra.plugin.scripts.shell.Commands
        commands: ["python incremental_load.py"]

Failure Handling and Retries

Kestra propagates failure states through the dependency graph according to strict rules. If any upstream task fails, downstream tasks that depend on it are skipped or marked as failed, preventing execution of logic that relies on incomplete data.

The engine provides granular control through configuration parameters:

• retry: Maximum number of retry attempts for transient failures
• retryDelay: Duration between retry attempts
• continueOnError: Boolean flag allowing downstream tasks to proceed despite upstream failures

These settings are defined at the task level in the YAML specification, allowing different reliability policies for different pipeline stages.

Execution Lifecycle and Validation

Kestra's execution engine follows a strict six-phase lifecycle to ensure DAG integrity:

1. Parse YAML: The engine ingests the workflow definition and constructs an internal graph representation where each node maps to a task ID.

2. Resolve dependencies: The system adds directed edges based on dependsOn declarations and implicit sequential ordering.

3. Topological sort: Kestra performs a topological sort to determine valid execution sequences. During this phase, the engine validates that the graph is acyclic. If a cycle is detected (e.g., Task A depends on B, B depends on C, and C depends on A), the system raises a compile-time error before execution begins.

4. Run ready tasks: The scheduler identifies all nodes with no incoming edges (no dependencies) and dispatches them to the worker pool.

5. Graph reduction: When a task completes successfully, the engine removes its outgoing edges and re-evaluates the ready set, potentially unlocking new tasks for execution.

6. Completion detection: The process repeats until the graph is empty (indicating workflow success) or a task fails irrecoverably (triggering workflow failure).

This validation occurs automatically when uploading flows via the Kestra UI or API, preventing runtime deadlocks through static analysis.

Practical Examples from the Data Engineering Zoomcamp

The DataTalksClub/data-engineering-zoomcamp repository provides concrete implementations of these concepts across several key files:

• 02-workflow-orchestration/flows/04_postgres_taxi.yaml: Demonstrates implicit ordering, conditional If blocks for yellow vs. green taxi data processing, and final cleanup tasks that depend on conditional branches.

• 02-workflow-orchestration/flows/02_python.yaml: Shows how Python tasks emit outputs that downstream tasks consume via dependency relationships, illustrating data flow between nodes.

• 02-workflow-orchestration/flows/06_gcp_taxi.yaml: Implements cross-plugin dependencies where cloud storage operations, BigQuery loads, and data transformations form a complex multi-cloud DAG.

• 02-workflow-orchestration/README.md: Contains architectural overview documentation explaining DAG fundamentals and best practices for dependency management.

These examples demonstrate how Kestra translates declarative YAML configurations into optimized execution plans that handle both simple linear pipelines and complex multi-branch data workflows.

Summary

• Kestra models workflows as Directed Acyclic Graphs (DAGs) where tasks are nodes and dependencies are edges.
• Implicit ordering occurs when tasks appear sequentially in YAML files, while the dependsOn field creates explicit dependencies for complex branching.
• The engine computes a ready set of tasks with satisfied dependencies and executes them concurrently when possible, optimizing for parallel processing.
• Conditional branching via io.kestra.plugin.core.flow.If modifies the execution graph dynamically based on runtime conditions without breaking dependency chains.
• Static validation ensures the DAG remains acyclic before execution, preventing runtime deadlocks through topological sorting.
• Failure propagation follows dependency paths, with configurable retry, retryDelay, and continueOnError options for fine-grained control.

Frequently Asked Questions

How does Kestra determine the execution order of tasks?

Kestra determines execution order by first building a dependency graph from the YAML definition. It adds edges based on the dependsOn field and implicit sequential ordering. The engine then performs a topological sort to identify which tasks have no unmet dependencies (the ready set). These tasks execute first. As tasks complete, the engine removes their outgoing edges and recalculates the ready set, continuing until all tasks finish or a failure occurs.

Can tasks run in parallel in Kestra, or are they always sequential?

Tasks can run in parallel when they have no mutual dependencies. If two tasks do not depend on each other (neither appears in the other's dependsOn list and they are not implicitly ordered), Kestra schedules them concurrently using its worker pool. This parallelism is automatic and requires no additional configuration beyond proper dependency declaration.

What happens if a task fails in a Kestra workflow?

If a task fails, Kestra propagates the failure state to all downstream tasks that depend on it. These dependent tasks are skipped or marked as failed, preventing execution of logic that requires the failed task's outputs. You can configure retry policies using the retry and retryDelay parameters for transient failures, or set continueOnError: true to allow downstream tasks to proceed despite the failure.

How does Kestra prevent circular dependencies in workflows?

Kestra performs static DAG validation before execution begins. During the parsing phase, the engine constructs the dependency graph and checks for cycles using topological sorting algorithms. If the system detects a circular dependency (e.g., Task A depends on B, B depends on C, and C depends on A), it raises a compile-time error when you attempt to save or execute the flow. This validation occurs in the Kestra UI or API, ensuring only valid acyclic graphs enter the execution queue.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}