How Kestra's Concurrency Control Prevents Duplicate Data Processing

Kestra prevents duplicate data processing by enforcing per-flow concurrency limits that reject or queue new executions when the configured limit is reached, ensuring only one pipeline run processes a specific time window at any given moment.

The DataTalksClub/data-engineering-zoomcamp repository demonstrates how Kestra's concurrency control mechanisms safeguard data pipelines against duplicate ingestion. By configuring concurrency limits directly in flow definitions, data engineers can enforce strict execution boundaries without writing custom locking logic. This built-in capability ensures that scheduled triggers or backfill operations never overlap, eliminating the risk of processing the same source files twice.

Understanding Kestra's Concurrency Control Mechanism

Kestra implements concurrency control through a declarative concurrency block within flow definitions. When a trigger fires—whether from a schedule, backfill request, or external event—Kestra evaluates the number of currently active executions for that specific flow before allowing a new run to start.

The mechanism operates through four sequential stages:

1. Flow Definition – The concurrency block declares a numeric limit and a behavior (FAIL, WAIT, or SKIP).
2. Trigger Evaluation – When a trigger fires, Kestra checks the count of live executions against the configured limit.
3. Execution Decision – If active executions are below the limit, the new run starts immediately. If the count meets or exceeds the limit, Kestra applies the specified behavior.
4. Idempotent Processing – Because overlapping runs never start simultaneously, downstream tasks execute exactly once per trigger interval, preventing duplicate records in target databases or storage systems.

Configuring Concurrency Limits in Flow Definitions

Basic Syntax and Parameters

In the DataTalksClub/data-engineering-zoomcamp repository, concurrency configuration appears in 02-workflow-orchestration/flows/01_hello_world.yaml. The concurrency block accepts two required parameters: limit (integer) and behavior (string).

# 02-workflow-orchestration/flows/01_hello_world.yaml

id: 01_hello_world
namespace: zoomcamp
inputs:
  - id: name
    type: STRING
    defaults: Will

concurrency:
  behavior: FAIL          # reject a new run when the limit is reached

  limit: 2                # at most two executions may run concurrently

The limit parameter defines the maximum number of concurrent executions allowed for this flow. Once this threshold is reached, Kestra evaluates the behavior parameter to determine how to handle additional execution requests.

Failure Behavior Options

Kestra provides three distinct behaviors for handling limit violations:

• FAIL (default) – Immediately rejects the new execution attempt, preventing any overlapping runs that could process identical data windows.
• WAIT – Queues the new execution until an active run completes, maintaining sequential processing without data duplication.
• SKIP – Silently drops the new execution request, useful for time-sensitive flows where stale runs hold no value.

Real-World Implementation: Preventing Duplicate Taxi Data Loads

The repository's 02-workflow-orchestration/flows/05_postgres_taxi_scheduled.yaml demonstrates strict concurrency control for data ingestion pipelines. This flow processes monthly taxi CSV files and uses limit: 1 to guarantee that only one execution handles a specific month's data at any time.

# 02-workflow-orchestration/flows/05_postgres_taxi_scheduled.yaml

id: 05_postgres_taxi_scheduled
namespace: zoomcamp
description: |
  Best to add a label `backfill:true` from the UI to track executions
  created via a backfill.

concurrency:
  limit: 1                # enforce a single active run → no duplicate CSV loads

With this configuration, if a scheduled trigger fires while the previous monthly ingestion is still downloading or processing files, Kestra fails the second run according to the default FAIL behavior. This prevents the same source CSV files from being downloaded and loaded into PostgreSQL twice, eliminating duplicate records without requiring manual checks or database constraints.

Why Concurrency Control Eliminates Duplicates Without Custom Logic

Traditional data pipelines often require external locking mechanisms—such as database advisory locks or distributed semaphore systems—to prevent concurrent processes from ingesting identical datasets. Kestra's built-in concurrency control removes this operational burden.

Because each execution typically corresponds to a specific temporal partition (for example, a month of taxi trips), the concurrency limit guarantees temporal isolation. When limit: 1 is set, the platform essentially serializes access to each data partition, ensuring that backfill operations and scheduled runs never compete for the same source files. This is particularly critical in the DataTalksClub/data-engineering-zoomcamp workflows, where monthly CSV downloads from external sources must occur exactly once to maintain data integrity.

Summary

• Kestra's concurrency control operates through a declarative concurrency block in flow definitions, specifying a numeric limit and rejection behavior.
• The default FAIL behavior prevents new executions from starting when the limit is reached, immediately rejecting overlapping runs that could cause duplicate processing.
• limit: 1 ensures strict single-execution semantics for time-windowed data ingestion, as demonstrated in the taxi data pipeline examples.
• This mechanism eliminates the need for custom locking logic, as the orchestration layer guarantees that only one run accesses a specific data partition at any given moment.
• Configuration examples in 01_hello_world.yaml and 05_postgres_taxi_scheduled.yaml illustrate both general patterns and production-ready implementations for preventing duplicate data loads.

Frequently Asked Questions

What happens when a Kestra flow reaches its concurrency limit?

When the number of active executions reaches the configured limit, Kestra applies the specified behavior to any new execution requests. With the default FAIL behavior, Kestra immediately rejects the new run and marks it as failed, preventing overlapping execution windows. Alternatively, setting behavior: WAIT queues the request until capacity becomes available, while behavior: SKIP discards the request entirely.

Can I use Kestra concurrency control for backfill operations?

Yes. The 05_postgres_taxi_scheduled.yaml flow in the DataTalksClub/data-engineering-zoomcamp repository specifically addresses backfill scenarios. By maintaining limit: 1, you ensure that backfill requests for historical data partitions do not overlap with scheduled runs or other backfill attempts targeting the same time window, preventing duplicate ingestion of historical records.

What is the difference between FAIL, WAIT, and SKIP behaviors?

FAIL rejects the execution immediately and records it as a failure, which is ideal for data pipelines where overlapping runs would create duplicates. WAIT holds the execution in a queue until concurrency slots become available, preserving execution order but potentially delaying processing. SKIP silently aborts the execution without recording a failure, suitable for workflows where only the latest run matters and missed intervals require no retry.

Do I need to implement additional locking mechanisms when using Kestra concurrency limits?

No. Kestra's concurrency control acts as an orchestration-level semaphore, eliminating the need for custom database locks or external coordination services. For most data ingestion patterns—particularly those processing distinct time partitions like the monthly taxi datasets in the zoomcamp repository—setting an appropriate limit (typically 1 for sequential processing) provides sufficient protection against duplicate data processing without additional application code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}