# How to Debug Failed Kestra Flow Executions in Data Engineering Zoomcamp

> Debug failed Kestra flow executions in your Data Engineering Zoomcamp project Monitor task logs use the REST API check Docker connectivity and implement onError handlers for robust data pipelines.

- Repository: [DataTalksClub/data-engineering-zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
- Tags: how-to-guide
- Published: 2026-05-30

---

**Debug failed Kestra flow executions by inspecting the UI task logs, querying the `/api/v1/executions/{id}` REST API, verifying Docker network connectivity, and implementing `onError` handlers in your YAML flow definitions.**

The DataTalksClub/data-engineering-zoomcamp repository utilizes Kestra for workflow orchestration, where failed executions require systematic debugging to identify root causes. Understanding how to access logs, validate configurations, and leverage Kestra's built-in error handling mechanisms is essential for maintaining reliable data pipelines in the Zoomcamp environment.

## Inspect Execution Details in the Kestra UI

Start debugging by navigating to the **Executions** tab in the Kestra UI (default `http://localhost:8080`). According to [`02-workflow-orchestration/README.md`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/README.md), click any failed run to view high-level status indicators, start/end timestamps, and the outcome summary for each task.

Expand individual task blocks to reveal **stdout/stderr logs**. These logs are persisted in the Kestra container under `/app/logs` and streamed to the UI in real-time. Look for stack traces, non-zero exit codes, or explicit `throw` statements that indicate the specific failure point.

## Query Execution Metadata via REST API

When UI inspection requires programmatic access, use Kestra's REST API to fetch JSON-encoded execution details. The endpoint `/api/v1/executions/{executionId}` returns complete metadata including log file paths and the exact error payload.

```bash

# Replace <EXEC_ID> with the UUID shown in the UI

curl -s http://localhost:8080/api/v1/executions/<EXEC_ID> \
     -H "Accept: application/json" \
| jq .

```

To isolate specific failed tasks, query the tasks endpoint and filter for failed states:

```bash

# Retrieve failed task IDs

curl -s http://localhost:8080/api/v1/executions/<EXEC_ID>/tasks \
     -H "Accept: application/json" \
| jq '.[] | select(.state=="FAILED") | .id'

# Fetch logs for a specific failed task

curl -s http://localhost:8080/api/v1/executions/<EXEC_ID>/tasks/<TASK_ID>/logs \
     -H "Accept: text/plain"

```

## Verify Docker Compose Network Configuration

Many failures in the Zoomcamp environment stem from infrastructure misconfiguration rather than code errors. The [`02-workflow-orchestration/docker-compose.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/docker-compose.yml) defines the service topology, including the `postgres_zoomcamp` container that Kestra must reach.

Check for port conflicts—such as `pgAdmin` also listening on `8080`—and verify that the Kestra container can resolve external services. The [`cohorts/2025/02-workflow-orchestration/README.md`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2025/02-workflow-orchestration/README.md) contains Linux-specific networking quirks regarding Docker network aliases (`postgres_zoomcamp` vs `host.docker.internal`).

## Enable Verbose Logging and Dry-Run Validation

For deeper insight into internal Kestra operations, enable **DEBUG** logging by setting `logging.level.root=DEBUG` in the Kestra server's [`application.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/application.yml) or adding `-Dlog.level=DEBUG` to JVM options.

Validate flow definitions without invoking external services using **dry-run** mode:

```bash
docker exec -it kestra-server \
  kestra flow execute data-engineering/postgres_taxi --dry-run

```

This catches YAML syntax errors, missing plugin defaults, or mis-typed parameters before actual execution.

## Implement Error Handling in Flow Definitions

Add an `onError` clause to tasks in your flow YAML to capture error payloads programmatically. As demonstrated in [`02-workflow-orchestration/flows/04_postgres_taxi.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/flows/04_postgres_taxi.yaml):

```yaml
id: postgres_taxi
namespace: data-engineering
tasks:
  - id: ingest
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "{{ secret('POSTGRES_URL') }}"
    sql: "COPY taxi FROM '{{ inputs.file }}' CSV HEADER;"
    onError:
      - id: save_error
        type: io.kestra.plugin.fs.LocalCreate
        filename: "/tmp/failed_ingest_{{ execution.id }}.log"
        content: "{{ taskrun.errorMessage }}"

```

This pattern writes error details to persistent storage when tasks fail, preventing data loss and facilitating post-mortem analysis.

## Check Plugin Defaults and Secrets

Configuration errors often involve undefined secrets or KV entries. Verify that required values like `KESTRA_POSTGRES_URL` exist in **Kestra UI → Secrets** or as environment variables in [`docker-compose.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/docker-compose.yml).

The repository README emphasizes keeping sensitive data out of flow sources using the `{{ secret('KEY') }}` syntax. Ensure `pluginDefaults` are properly configured in your flow YAML to avoid `PluginException: Missing required property` errors.

## Retry or Backfill Failed Executions

After fixing the root cause, recover failed executions using the **Retry** API:

```bash
curl -X POST http://localhost:8080/api/v1/executions/<EXEC_ID>/retry \
     -H "Content-Type: application/json"

```

Alternatively, use the UI's scheduling tools to **Back-fill** date ranges, as documented in the ELT pipelines section of [`02-workflow-orchestration/README.md`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/README.md).

## Common Error Categories and Solutions

When debugging failed Kestra flow executions, identify the error pattern using these categories:

**Plugin Mis-configuration**
Symptoms include `PluginException: Missing required property`. Fix by ensuring required fields are set in `pluginDefaults` or directly in the task properties.

**External Service Unreachable**
Look for `Connection refused` or timeout errors. Verify Docker network aliases and port mappings in [`docker-compose.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/docker-compose.yml), ensuring the Kestra container can reach services like `postgres_zoomcamp`.

**Permission or Secret Errors**
`Unauthorized` or `SecretNotFoundException` indicates missing credentials. Add them via the Kestra UI Secrets page or define them as environment variables in the Docker Compose configuration.

**Task Exit Code Failures**
When logs show `Process finished with exit code 1`, examine the script's stdout for Python or Shell errors. Fix the underlying code and add `onError` handling to capture future failures.

**YAML Syntax Errors**
Flows failing to load with "Invalid flow definition" messages require validation. Run `kestra flow validate <file.yaml>` or use `yamllint` before deployment to catch indentation or syntax issues.

## Summary

- **Start with the UI**: Check the Executions tab at `localhost:8080` for high-level status and expandable task logs stored in `/app/logs`.
- **Use the REST API**: Query `/api/v1/executions/{id}` and `/api/v1/executions/{id}/tasks` for detailed JSON metadata and specific log paths.
- **Verify Infrastructure**: Check Docker Compose network configuration in [`02-workflow-orchestration/docker-compose.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/docker-compose.yml) for port conflicts and service connectivity.
- **Implement Error Handling**: Use `onError` blocks in flow YAML to capture error messages, as demonstrated in [`04_postgres_taxi.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/04_postgres_taxi.yaml).
- **Validate Before Running**: Use `--dry-run` mode to catch configuration errors without executing external tasks.
- **Manage Secrets**: Ensure all required secrets are defined in the Kestra UI or environment variables, never hardcoded in flow sources.

## Frequently Asked Questions

### How do I access Kestra logs when the UI is unavailable?

Query the REST API directly using `curl http://localhost:8080/api/v1/executions/<EXEC_ID>/tasks/<TASK_ID>/logs` or exec into the Kestra container to inspect files in `/app/logs`. The logs are stored persistently in the container filesystem even when the UI connection fails.

### Why does my Kestra flow fail with "Connection refused" to PostgreSQL?

This typically indicates Docker networking issues between the Kestra server and the `postgres_zoomcamp` container. Verify in [`02-workflow-orchestration/docker-compose.yml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/docker-compose.yml) that both services share the same network and that the hostname matches the service name. Linux users should check the specific networking notes in [`cohorts/2025/02-workflow-orchestration/README.md`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2025/02-workflow-orchestration/README.md).

### Can I automatically capture errors when a Kestra task fails?

Yes. Implement an `onError` task block in your flow YAML to handle failures gracefully. As shown in the [`04_postgres_taxi.yaml`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/04_postgres_taxi.yaml) example, you can write error messages to local files or KV stores using `{{ taskrun.errorMessage }}` templating, preventing data loss and facilitating post-mortem analysis.

### How do I re-run a failed execution without changing the flow code?

Use the retry endpoint: `POST /api/v1/executions/{executionId}/retry`. Alternatively, navigate to the failed execution in the Kestra UI and click the **Retry** button. For batch recovery of multiple runs, use the **Back-fill** feature in the UI's scheduling tools to reprocess specific date ranges.