Kestra Task Dependencies and Execution Ordering in Complex ETL Pipelines

Kestra orchestrates complex ETL workflows as a directed-acyclic graph (DAG), using implicit sequential ordering by default, explicit dependsOn declarations for parallelism, and output-to-input references to enforce runtime data dependencies.

The DataTalksClub/data-engineering-zoomcamp repository demonstrates how Kestra handles task dependencies and execution ordering through production-ready YAML definitions. By modeling workflows as DAGs where tasks connect via explicit edges or implicit sequencing, Kestra enables both simple linear pipelines and sophisticated parallel ETL architectures.

Implicit vs. Explicit Task Dependencies

Kestra provides two mechanisms for controlling execution order: implicit sequencing based on task list position, and explicit dependency declarations using the dependsOn field.

Sequential Execution by Default

By default, Kestra executes tasks sequentially in the order they appear in the tasks: list. Each task waits for the previous one to complete before starting.

In 02-workflow-orchestration/flows/03_getting_started_data_pipeline.yaml, the pipeline follows a linear progression:

tasks:
  - id: extract
    type: io.kestra.plugin.jdbc.postgresql.CopyOut
    sql: "SELECT * FROM public.my_table"
    format: CSV

  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    containerImage: python:3.11-slim
    script: |
      import pandas as pd
      # Transformation logic here

  - id: query
    type: io.kestra.plugin.jdbc.postgresql.Query
    sql: "SELECT COUNT(*) FROM public.transformed_data"

Here, transform automatically waits for extract to finish, and query waits for transform, creating an implicit chain without explicit dependency declarations.

Parallel Execution with dependsOn

To enable concurrency, add the dependsOn field to create explicit DAG edges. This allows independent tasks to run simultaneously while ensuring dependent tasks wait for all prerequisites.

The following pattern demonstrates parallel extraction tasks converging on a single transformation step:

# demo_parallel.yaml

id: demo_parallel
namespace: examples

tasks:
  - id: extract_raw
    type: io.kestra.plugin.core.http.Download
    uri: https://example.com/raw.csv

  - id: extract_metadata
    type: io.kestra.plugin.core.http.Download
    uri: https://example.com/meta.json

  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    dependsOn: [extract_raw, extract_metadata]
    containerImage: python:3.11-alpine
    inputFiles:
      raw.csv: "{{outputs.extract_raw.uri}}"
      meta.json: "{{outputs.extract_metadata.uri}}"
    script: |
      import json
      import pandas as pd
      # Python code that merges raw data with metadata

  - id: load
    type: io.kestra.plugin.gcp.bigquery.Query
    dependsOn: [transform]
    sql: |
      INSERT INTO my_dataset.my_table 
      SELECT * FROM read_csv_auto('{{workingDir}}/merged.parquet')

In this configuration, extract_raw and extract_metadata execute in parallel. The transform task waits for both to complete, while load depends solely on transform.

Conditional Branching for Dynamic Pipelines

Complex ETL pipelines often require conditional logic to handle different data sources or scenarios. Kestra implements this through the io.kestra.plugin.core.flow.If task, which evaluates Jinja expressions at runtime to determine execution paths.

The 02-workflow-orchestration/flows/09_gcp_taxi_scheduled.yaml file illustrates this pattern with taxi type branching:

tasks:
  - id: if_yellow_taxi
    type: io.kestra.plugin.core.flow.If
    condition: "{{inputs.taxi == 'yellow'}}"
    then:
      - id: extract_yellow
        type: io.kestra.plugin.gcp.gcs.Download
        uri: "{{inputs.uri}}"

  - id: if_green_taxi
    type: io.kestra.plugin.core.flow.If
    condition: "{{inputs.taxi == 'green'}}"
    then:
      - id: extract_green
        type: io.kestra.plugin.gcp.gcs.Download
        uri: "{{inputs.uri}}"

Each If task creates a subgraph that executes only when its condition evaluates to true, enabling the same workflow definition to handle multiple data sources without duplication.

Data-Driven Dependencies Through Outputs

Kestra enforces data dependencies automatically when tasks reference outputs from previous steps. Using the {{outputs.<taskId>.<field>}} syntax creates an implicit execution constraint, ensuring the producer task completes before the consumer begins.

In 03_getting_started_data_pipeline.yaml, the transform task consumes the URI produced by the extract task:

tasks:
  - id: extract
    type: io.kestra.plugin.jdbc.postgresql.CopyOut
    format: CSV
    
  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      data.csv: "{{outputs.extract.uri}}"
    script: |
      import pandas as pd
      df = pd.read_csv('{{outputs.extract.uri}}')
      # Processing logic here

Even without an explicit dependsOn declaration, Kestra recognizes that transform requires extract's output and schedules them accordingly. This output-to-input wiring ensures data availability while maintaining clean dependency graphs.

Global Variables and Trigger-Based Orchestration

Complex workflows require flexible configuration and multiple entry points. The repository demonstrates variable scoping and trigger patterns in 02-workflow-orchestration/flows/09_gcp_taxi_scheduled.yaml:

variables:
  file: "{{inputs.taxi}}_tripdata_{{inputs.year}}-{{inputs.month}}.csv"
  gcs_file: "gs://{{kv('GCP_BUCKET_NAME')}}/{{vars.file}}"
  table: "{{kv('GCP_DATASET')}}.{{inputs.taxi}}_tripdata"

tasks:
  # ... task definitions using {{vars.file}} and {{vars.table}}

triggers:
  - id: green_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 9 1 * *"
    inputs:
      taxi: green

  - id: yellow_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 10 1 * *"
    inputs:
      taxi: yellow

Global variables and KV store references ({{kv('KEY')}}) provide DRY configuration across tasks, while multiple schedule triggers create independent DAG instances with different input parameters. The docker-compose.yml in the same directory provides the underlying infrastructure to test these orchestration patterns locally.

Summary

• DAG-based architecture: Kestra models workflows as directed-acyclic graphs, enabling both sequential and parallel execution patterns.
• Implicit ordering: Tasks execute in list order by default, suitable for linear ETL pipelines like those in 03_getting_started_data_pipeline.yaml.
• Explicit parallelism: The dependsOn field creates explicit edges, allowing independent tasks to run concurrently while ensuring dependents wait for prerequisites.
• Conditional execution: io.kestra.plugin.core.flow.If tasks enable runtime branching for handling multiple data sources or scenarios.
• Output references: Using {{outputs.taskId.field}} automatically enforces data dependencies without explicit dependsOn declarations.
• Trigger flexibility: Multiple schedule triggers can invoke the same workflow with different inputs, creating isolated execution contexts.

Frequently Asked Questions

How does Kestra determine task execution order if I don't specify dependsOn?

Kestra executes tasks sequentially based on their position in the tasks: list. Each task waits for the previous one to complete, creating an implicit chain. However, if a task references another task's outputs using {{outputs.taskId.field}}, Kestra treats that as a dependency regardless of list position.

Can tasks run in parallel in Kestra?

Yes. Tasks run in parallel when they have no dependency relationship. Use the dependsOn field to explicitly declare which tasks must complete before others start. Tasks not listed in each other's dependency chains execute concurrently, maximizing resource utilization for independent operations like extracting from multiple sources simultaneously.

How do I handle conditional logic in Kestra ETL pipelines?

Use the io.kestra.plugin.core.flow.If task to create conditional branches. This task evaluates a Jinja expression in its condition field; the tasks inside the then: block only execute when the condition returns true. The repository's 09_gcp_taxi_scheduled.yaml demonstrates this pattern for processing different taxi types based on runtime inputs.

What is the difference between dependsOn and output references for dependencies?

dependsOn explicitly declares execution prerequisites but does not handle data passing, while output references ({{outputs.taskId.field}}) both enforce execution order and provide data access. Output references automatically create implicit dependencies, meaning you often don't need dependsOn when passing data between tasks, but you need dependsOn for parallel execution coordination or when tasks don't exchange data directly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}