How to Debug Failed Kestra Flow Executions in Data Engineering Zoomcamp

Debug failed Kestra flow executions by inspecting the UI task logs, querying the /api/v1/executions/{id} REST API, verifying Docker network connectivity, and implementing onError handlers in your YAML flow definitions.

The DataTalksClub/data-engineering-zoomcamp repository utilizes Kestra for workflow orchestration, where failed executions require systematic debugging to identify root causes. Understanding how to access logs, validate configurations, and leverage Kestra's built-in error handling mechanisms is essential for maintaining reliable data pipelines in the Zoomcamp environment.

Inspect Execution Details in the Kestra UI

Start debugging by navigating to the Executions tab in the Kestra UI (default http://localhost:8080). According to 02-workflow-orchestration/README.md, click any failed run to view high-level status indicators, start/end timestamps, and the outcome summary for each task.

Expand individual task blocks to reveal stdout/stderr logs. These logs are persisted in the Kestra container under /app/logs and streamed to the UI in real-time. Look for stack traces, non-zero exit codes, or explicit throw statements that indicate the specific failure point.

Query Execution Metadata via REST API

When UI inspection requires programmatic access, use Kestra's REST API to fetch JSON-encoded execution details. The endpoint /api/v1/executions/{executionId} returns complete metadata including log file paths and the exact error payload.

# Replace <EXEC_ID> with the UUID shown in the UI

curl -s http://localhost:8080/api/v1/executions/<EXEC_ID> \
     -H "Accept: application/json" \
| jq .

To isolate specific failed tasks, query the tasks endpoint and filter for failed states:

# Retrieve failed task IDs

curl -s http://localhost:8080/api/v1/executions/<EXEC_ID>/tasks \
     -H "Accept: application/json" \
| jq '.[] | select(.state=="FAILED") | .id'

# Fetch logs for a specific failed task

curl -s http://localhost:8080/api/v1/executions/<EXEC_ID>/tasks/<TASK_ID>/logs \
     -H "Accept: text/plain"

Verify Docker Compose Network Configuration

Many failures in the Zoomcamp environment stem from infrastructure misconfiguration rather than code errors. The 02-workflow-orchestration/docker-compose.yml defines the service topology, including the postgres_zoomcamp container that Kestra must reach.

Check for port conflicts—such as pgAdmin also listening on 8080—and verify that the Kestra container can resolve external services. The cohorts/2025/02-workflow-orchestration/README.md contains Linux-specific networking quirks regarding Docker network aliases (postgres_zoomcamp vs host.docker.internal).

Enable Verbose Logging and Dry-Run Validation

For deeper insight into internal Kestra operations, enable DEBUG logging by setting logging.level.root=DEBUG in the Kestra server's application.yml or adding -Dlog.level=DEBUG to JVM options.

Validate flow definitions without invoking external services using dry-run mode:

docker exec -it kestra-server \
  kestra flow execute data-engineering/postgres_taxi --dry-run

This catches YAML syntax errors, missing plugin defaults, or mis-typed parameters before actual execution.

Implement Error Handling in Flow Definitions

Add an onError clause to tasks in your flow YAML to capture error payloads programmatically. As demonstrated in 02-workflow-orchestration/flows/04_postgres_taxi.yaml:

id: postgres_taxi
namespace: data-engineering
tasks:
  - id: ingest
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "{{ secret('POSTGRES_URL') }}"
    sql: "COPY taxi FROM '{{ inputs.file }}' CSV HEADER;"
    onError:
      - id: save_error
        type: io.kestra.plugin.fs.LocalCreate
        filename: "/tmp/failed_ingest_{{ execution.id }}.log"
        content: "{{ taskrun.errorMessage }}"

This pattern writes error details to persistent storage when tasks fail, preventing data loss and facilitating post-mortem analysis.

Check Plugin Defaults and Secrets

Configuration errors often involve undefined secrets or KV entries. Verify that required values like KESTRA_POSTGRES_URL exist in Kestra UI → Secrets or as environment variables in docker-compose.yml.

The repository README emphasizes keeping sensitive data out of flow sources using the {{ secret('KEY') }} syntax. Ensure pluginDefaults are properly configured in your flow YAML to avoid PluginException: Missing required property errors.

Retry or Backfill Failed Executions

After fixing the root cause, recover failed executions using the Retry API:

curl -X POST http://localhost:8080/api/v1/executions/<EXEC_ID>/retry \
     -H "Content-Type: application/json"

Alternatively, use the UI's scheduling tools to Back-fill date ranges, as documented in the ELT pipelines section of 02-workflow-orchestration/README.md.

Common Error Categories and Solutions

When debugging failed Kestra flow executions, identify the error pattern using these categories:

Plugin Mis-configuration Symptoms include PluginException: Missing required property. Fix by ensuring required fields are set in pluginDefaults or directly in the task properties.

External Service Unreachable Look for Connection refused or timeout errors. Verify Docker network aliases and port mappings in docker-compose.yml, ensuring the Kestra container can reach services like postgres_zoomcamp.

Permission or Secret Errors Unauthorized or SecretNotFoundException indicates missing credentials. Add them via the Kestra UI Secrets page or define them as environment variables in the Docker Compose configuration.

Task Exit Code Failures When logs show Process finished with exit code 1, examine the script's stdout for Python or Shell errors. Fix the underlying code and add onError handling to capture future failures.

YAML Syntax Errors Flows failing to load with "Invalid flow definition" messages require validation. Run kestra flow validate <file.yaml> or use yamllint before deployment to catch indentation or syntax issues.

Summary

• Start with the UI: Check the Executions tab at localhost:8080 for high-level status and expandable task logs stored in /app/logs.
• Use the REST API: Query /api/v1/executions/{id} and /api/v1/executions/{id}/tasks for detailed JSON metadata and specific log paths.
• Verify Infrastructure: Check Docker Compose network configuration in 02-workflow-orchestration/docker-compose.yml for port conflicts and service connectivity.
• Implement Error Handling: Use onError blocks in flow YAML to capture error messages, as demonstrated in 04_postgres_taxi.yaml.
• Validate Before Running: Use --dry-run mode to catch configuration errors without executing external tasks.
• Manage Secrets: Ensure all required secrets are defined in the Kestra UI or environment variables, never hardcoded in flow sources.

Frequently Asked Questions

How do I access Kestra logs when the UI is unavailable?

Query the REST API directly using curl http://localhost:8080/api/v1/executions/<EXEC_ID>/tasks/<TASK_ID>/logs or exec into the Kestra container to inspect files in /app/logs. The logs are stored persistently in the container filesystem even when the UI connection fails.

Why does my Kestra flow fail with "Connection refused" to PostgreSQL?

This typically indicates Docker networking issues between the Kestra server and the postgres_zoomcamp container. Verify in 02-workflow-orchestration/docker-compose.yml that both services share the same network and that the hostname matches the service name. Linux users should check the specific networking notes in cohorts/2025/02-workflow-orchestration/README.md.

Can I automatically capture errors when a Kestra task fails?

Yes. Implement an onError task block in your flow YAML to handle failures gracefully. As shown in the 04_postgres_taxi.yaml example, you can write error messages to local files or KV stores using {{ taskrun.errorMessage }} templating, preventing data loss and facilitating post-mortem analysis.

How do I re-run a failed execution without changing the flow code?

Use the retry endpoint: POST /api/v1/executions/{executionId}/retry. Alternatively, navigate to the failed execution in the Kestra UI and click the Retry button. For batch recovery of multiple runs, use the Back-fill feature in the UI's scheduling tools to reprocess specific date ranges.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}