# Kestra Task Dependencies and Execution Ordering in Complex ETL Pipelines > Learn how Kestra expertly manages task dependencies and execution ordering in complex ETL pipelines. Discover DAGs, explicit dependencies, and runtime data flow control for efficient workflows. - Repository: [DataTalksClub/data-engineering-zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp) - Tags: how-to-guide - Published: 2026-05-30 --- **Kestra orchestrates complex ETL workflows as a directed-acyclic graph (DAG), using implicit sequential ordering by default, explicit `dependsOn` declarations for parallelism, and output-to-input references to enforce runtime data dependencies.** The DataTalksClub/data-engineering-zoomcamp repository demonstrates how Kestra handles task dependencies and execution ordering through production-ready YAML definitions. By modeling workflows as DAGs where tasks connect via explicit edges or implicit sequencing, Kestra enables both simple linear pipelines and sophisticated parallel ETL architectures. ## Implicit vs. Explicit Task Dependencies Kestra provides two mechanisms for controlling execution order: implicit sequencing based on task list position, and explicit dependency declarations using the `dependsOn` field. ### Sequential Execution by Default By default, Kestra executes tasks sequentially in the order they appear in the `tasks:` list. Each task waits for the previous one to complete before starting. In [`02-workflow-orchestration/flows/03_getting_started_data_pipeline.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/flows/03_getting_started_data_pipeline.yaml), the pipeline follows a linear progression: ```yaml tasks: - id: extract type: io.kestra.plugin.jdbc.postgresql.CopyOut sql: "SELECT * FROM public.my_table" format: CSV - id: transform type: io.kestra.plugin.scripts.python.Script containerImage: python:3.11-slim script: | import pandas as pd # Transformation logic here - id: query type: io.kestra.plugin.jdbc.postgresql.Query sql: "SELECT COUNT(*) FROM public.transformed_data" ``` Here, `transform` automatically waits for `extract` to finish, and `query` waits for `transform`, creating an implicit chain without explicit dependency declarations. ### Parallel Execution with dependsOn To enable concurrency, add the `dependsOn` field to create explicit DAG edges. This allows independent tasks to run simultaneously while ensuring dependent tasks wait for all prerequisites. The following pattern demonstrates parallel extraction tasks converging on a single transformation step: ```yaml # demo_parallel.yaml id: demo_parallel namespace: examples tasks: - id: extract_raw type: io.kestra.plugin.core.http.Download uri: https://example.com/raw.csv - id: extract_metadata type: io.kestra.plugin.core.http.Download uri: https://example.com/meta.json - id: transform type: io.kestra.plugin.scripts.python.Script dependsOn: [extract_raw, extract_metadata] containerImage: python:3.11-alpine inputFiles: raw.csv: "{{outputs.extract_raw.uri}}" meta.json: "{{outputs.extract_metadata.uri}}" script: | import json import pandas as pd # Python code that merges raw data with metadata - id: load type: io.kestra.plugin.gcp.bigquery.Query dependsOn: [transform] sql: | INSERT INTO my_dataset.my_table SELECT * FROM read_csv_auto('{{workingDir}}/merged.parquet') ``` In this configuration, `extract_raw` and `extract_metadata` execute in parallel. The `transform` task waits for both to complete, while `load` depends solely on `transform`. ## Conditional Branching for Dynamic Pipelines Complex ETL pipelines often require conditional logic to handle different data sources or scenarios. Kestra implements this through the `io.kestra.plugin.core.flow.If` task, which evaluates Jinja expressions at runtime to determine execution paths. The [`02-workflow-orchestration/flows/09_gcp_taxi_scheduled.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/flows/09_gcp_taxi_scheduled.yaml) file illustrates this pattern with taxi type branching: ```yaml tasks: - id: if_yellow_taxi type: io.kestra.plugin.core.flow.If condition: "{{inputs.taxi == 'yellow'}}" then: - id: extract_yellow type: io.kestra.plugin.gcp.gcs.Download uri: "{{inputs.uri}}" - id: if_green_taxi type: io.kestra.plugin.core.flow.If condition: "{{inputs.taxi == 'green'}}" then: - id: extract_green type: io.kestra.plugin.gcp.gcs.Download uri: "{{inputs.uri}}" ``` Each `If` task creates a subgraph that executes only when its condition evaluates to true, enabling the same workflow definition to handle multiple data sources without duplication. ## Data-Driven Dependencies Through Outputs Kestra enforces **data dependencies** automatically when tasks reference outputs from previous steps. Using the `{{outputs..}}` syntax creates an implicit execution constraint, ensuring the producer task completes before the consumer begins. In [`03_getting_started_data_pipeline.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/03_getting_started_data_pipeline.yaml), the transform task consumes the URI produced by the extract task: ```yaml tasks: - id: extract type: io.kestra.plugin.jdbc.postgresql.CopyOut format: CSV - id: transform type: io.kestra.plugin.scripts.python.Script inputFiles: data.csv: "{{outputs.extract.uri}}" script: | import pandas as pd df = pd.read_csv('{{outputs.extract.uri}}') # Processing logic here ``` Even without an explicit `dependsOn` declaration, Kestra recognizes that `transform` requires `extract`'s output and schedules them accordingly. This **output-to-input wiring** ensures data availability while maintaining clean dependency graphs. ## Global Variables and Trigger-Based Orchestration Complex workflows require flexible configuration and multiple entry points. The repository demonstrates variable scoping and trigger patterns in [`02-workflow-orchestration/flows/09_gcp_taxi_scheduled.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/flows/09_gcp_taxi_scheduled.yaml): ```yaml variables: file: "{{inputs.taxi}}_tripdata_{{inputs.year}}-{{inputs.month}}.csv" gcs_file: "gs://{{kv('GCP_BUCKET_NAME')}}/{{vars.file}}" table: "{{kv('GCP_DATASET')}}.{{inputs.taxi}}_tripdata" tasks: # ... task definitions using {{vars.file}} and {{vars.table}} triggers: - id: green_schedule type: io.kestra.plugin.core.trigger.Schedule cron: "0 9 1 * *" inputs: taxi: green - id: yellow_schedule type: io.kestra.plugin.core.trigger.Schedule cron: "0 10 1 * *" inputs: taxi: yellow ``` Global variables and KV store references (`{{kv('KEY')}}`) provide DRY configuration across tasks, while multiple schedule triggers create independent DAG instances with different input parameters. The [`docker-compose.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/docker-compose.yml) in the same directory provides the underlying infrastructure to test these orchestration patterns locally. ## Summary - **DAG-based architecture**: Kestra models workflows as directed-acyclic graphs, enabling both sequential and parallel execution patterns. - **Implicit ordering**: Tasks execute in list order by default, suitable for linear ETL pipelines like those in [`03_getting_started_data_pipeline.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/03_getting_started_data_pipeline.yaml). - **Explicit parallelism**: The `dependsOn` field creates explicit edges, allowing independent tasks to run concurrently while ensuring dependents wait for prerequisites. - **Conditional execution**: `io.kestra.plugin.core.flow.If` tasks enable runtime branching for handling multiple data sources or scenarios. - **Output references**: Using `{{outputs.taskId.field}}` automatically enforces data dependencies without explicit `dependsOn` declarations. - **Trigger flexibility**: Multiple schedule triggers can invoke the same workflow with different inputs, creating isolated execution contexts. ## Frequently Asked Questions ### How does Kestra determine task execution order if I don't specify dependsOn? Kestra executes tasks sequentially based on their position in the `tasks:` list. Each task waits for the previous one to complete, creating an implicit chain. However, if a task references another task's outputs using `{{outputs.taskId.field}}`, Kestra treats that as a dependency regardless of list position. ### Can tasks run in parallel in Kestra? Yes. Tasks run in parallel when they have no dependency relationship. Use the `dependsOn` field to explicitly declare which tasks must complete before others start. Tasks not listed in each other's dependency chains execute concurrently, maximizing resource utilization for independent operations like extracting from multiple sources simultaneously. ### How do I handle conditional logic in Kestra ETL pipelines? Use the `io.kestra.plugin.core.flow.If` task to create conditional branches. This task evaluates a Jinja expression in its `condition` field; the tasks inside the `then:` block only execute when the condition returns true. The repository's [`09_gcp_taxi_scheduled.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/09_gcp_taxi_scheduled.yaml) demonstrates this pattern for processing different taxi types based on runtime inputs. ### What is the difference between dependsOn and output references for dependencies? `dependsOn` explicitly declares execution prerequisites but does not handle data passing, while output references (`{{outputs.taskId.field}}`) both enforce execution order and provide data access. Output references automatically create implicit dependencies, meaning you often don't need `dependsOn` when passing data between tasks, but you need `dependsOn` for parallel execution coordination or when tasks don't exchange data directly.