# How Probe Handles Job Dependencies with the `needs` Keyword

> Discover how Probe manages job dependencies using the `needs` keyword. Learn about circular dependency checks and successful upstream job completion for reliable execution.

- Repository: [Tomohisa Oda/probe](https://github.com/linyows/probe)
- Tags: how-to-guide
- Published: 2026-03-06

---

**Probe implements job dependencies through a `Needs []string` field on the `Job` struct, validated by a `JobScheduler` that checks for circular dependencies and ensures a job only runs after all declared upstream jobs complete successfully.**

Probe is a lightweight workflow runner inspired by GitHub Actions. When you define jobs in a workflow file, the `needs` keyword lets you specify which jobs must finish before another can start. This article explains how Probe parses these dependencies, validates them at startup, and enforces them during execution according to the source code in `linyows/probe`.

## How the `needs` Field Declares Dependencies

### The Job Struct Definition

In [`main/job.go`](https://github.com/linyows/probe/blob/main/main/job.go), the `Job` struct contains a `Needs` field that captures the dependency list from your YAML workflow:

```go
type Job struct {
    ID      string            `yaml:"id,omitempty"`
    Name    string            `yaml:"name,omitempty"`
    Needs   []string          `yaml:"needs,omitempty"`   // ← dependency names
    Steps   []Step            `yaml:"steps,omitempty"`
    // … other fields …
}

```

When Probe loads a workflow, it unmarshals the YAML into these structs. The `needs` array holds the `id` or `name` values of jobs that must complete before this job can run.

## Building and Validating the Dependency Graph

### Registering Jobs in the Scheduler

Before execution begins, Probe creates a `JobScheduler` in [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go) to manage the workflow state. The `AddJob` method registers each job, generating an ID if one is missing and initializing tracking maps for status, results, and repeat counters:

```go
func (js *JobScheduler) AddJob(job Job) string {
    if job.ID == "" {
        job.ID = generateJobID()
    }
    js.jobs[job.ID] = job
    js.status[job.ID] = JobPending
    // … initialize repeat counters …
    return job.ID
}

```

### Validating Dependencies and Detecting Cycles

Once all jobs are registered, `ValidateDependencies` performs two critical checks:

1. **Existence validation**: Every name in `Needs` must correspond to a registered job ID.
2. **Cycle detection**: The scheduler uses a DAG utility (`dag.DetectCycleFn`) to ensure no circular dependencies exist (e.g., `A` needs `B`, `B` needs `A`).

```go
func (js *JobScheduler) ValidateDependencies() error {
    // 1️⃣ Ensure every `need` points to an existing job
    for _, job := range js.jobs {
        for _, dep := range job.Needs {
            if _, ok := js.jobs[dep]; !ok {
                return fmt.Errorf("job '%s' depends on non-existent job '%s'", job.ID, dep)
            }
        }
    }
    // 2️⃣ Detect cycles using the DAG helper
    return js.checkCircularDependencies()
}

```

If validation fails, Probe aborts the workflow immediately with an error message.

## Runtime Execution and Dependency Resolution

### Checking if a Job Can Run

During the execution phase, the scheduler determines readiness via `CanRunJob`. A job is runnable only when **all** jobs listed in its `Needs` array have fully completed all their repeat iterations successfully:

```go
func (js *JobScheduler) CanRunJob(jobID string) bool {
    job := js.jobs[jobID]
    if js.status[jobID] != JobPending {
        return false
    }
    // All dependencies must be fully completed (including repeats)
    for _, dep := range job.Needs {
        if !js.isJobFullyCompleted(dep) {
            return false
        }
    }
    return true
}

```

The helper `isJobFullyCompleted` verifies both the job status and that the success result is true:

```go
func (js *JobScheduler) isJobFullyCompleted(jobID string) bool {
    if js.status[jobID] != JobCompleted {
        return false
    }
    target := js.repeatTargets[jobID]
    counter := js.repeatCounters[jobID]
    return counter >= target && js.results[jobID]
}

```

### The Main Execution Loop

The workflow engine in [`main/workflow.go`](https://github.com/linyows/probe/blob/main/main/workflow.go) orchestrates execution through `startJobsWithDependencies`. It repeatedly queries the scheduler for runnable jobs and launches them concurrently:

```go
func (w *Workflow) startJobsWithDependencies(ctx JobContext) error {
    for !ctx.JobScheduler.AllJobsCompleted() {
        runnable := ctx.JobScheduler.GetRunnableJobs()
        if len(runnable) == 0 {
            // No job can proceed → mark jobs whose dependencies failed as skipped
            w.handleNoRunnableJobs(ctx)
            continue
        }
        w.processRunnableJobs(runnable, ctx)
        ctx.JobScheduler.wg.Wait()
    }
    return nil
}

```

`GetRunnableJobs` iterates over all pending jobs and returns those where `CanRunJob` returns true.

### Handling Failed Dependencies and Skipped Jobs

When a dependency fails, Probe prevents downstream execution. The `MarkJobsWithFailedDependencies` method in [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go) identifies jobs whose dependencies have failed or completed unsuccessfully, marking them as failed:

```go
func (js *JobScheduler) MarkJobsWithFailedDependencies() []string {
    var skipped []string
    for id, job := range js.jobs {
        if js.status[id] != JobPending {
            continue
        }
        for _, dep := range job.Needs {
            if js.status[dep] == JobFailed || (js.status[dep] == JobCompleted && !js.results[dep]) {
                js.status[id] = JobFailed
                js.results[id] = false
                skipped = append(skipped, id)
                break
            }
        }
    }
    return skipped
}

```

Finally, [`main/workflow.go`](https://github.com/linyows/probe/blob/main/main/workflow.go) updates the output for these skipped jobs via `updateSkippedJobsOutput`, setting their status to `"skipped"` in the final report:

```go
func (w *Workflow) updateSkippedJobsOutput(skippedJobs []string, rs *Result) {
    for _, jobID := range skippedJobs {
        if jr, ok := rs.Jobs[jobID]; ok {
            jr.mutex.Lock()
            jr.EndTime = jr.StartTime
            jr.Status = "skipped"
            jr.Success = true // treated as successful, like a skipif
            jr.mutex.Unlock()
        }
    }
}

```

## Key Implementation Files

| File | Purpose | Link |
|------|---------|------|
| [`main/job.go`](https://github.com/linyows/probe/blob/main/main/job.go) | Definition of `Job` (including `Needs`) and per-job execution logic | [job.go](https://github.com/linyows/probe/blob/main/main/job.go) |
| [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go) | Job scheduler that registers jobs, validates dependencies, and decides runnable jobs | [scheduler.go](https://github.com/linyows/probe/blob/main/main/scheduler.go) |
| [`main/workflow.go`](https://github.com/linyows/probe/blob/main/main/workflow.go) | Orchestrates the whole workflow, repeatedly invoking the scheduler, handling dead-locks & skipped jobs | [workflow.go](https://github.com/linyows/probe/blob/main/main/workflow.go) |
| [`main/executor.go`](https://github.com/linyows/probe/blob/main/main/executor.go) | Executes a single job (including repeat & async handling) once the scheduler marks it runnable | [executor.go](https://github.com/linyows/probe/blob/main/main/executor.go) |
| `dag/*` | Utility for detecting dependency cycles used by the scheduler | [dag package](https://github.com/linyows/probe/tree/main/main/dag) |

## Summary

- **Declaration**: Probe uses the `Needs []string` field in [`main/job.go`](https://github.com/linyows/probe/blob/main/main/job.go) to capture dependency lists from YAML workflow files.
- **Validation**: The `JobScheduler` in [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go) validates that all referenced jobs exist and uses a DAG utility to detect circular dependencies before execution starts.
- **Execution**: The scheduler only marks a job as runnable when `CanRunJob` confirms all dependencies have fully completed (including all repeat iterations) successfully.
- **Failure handling**: If a dependency fails, `MarkJobsWithFailedDependencies` marks downstream jobs as failed, and [`main/workflow.go`](https://github.com/linyows/probe/blob/main/main/workflow.go) reports them as *skipped* in the final output.

## Frequently Asked Questions

### What happens if a job lists a dependency that does not exist?

Probe validates the dependency graph before running any jobs. In [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go), the `ValidateDependencies` function checks that every name in a job's `Needs` array corresponds to a registered job ID. If a dependency is missing, Probe returns an error immediately and aborts the workflow.

### Does Probe support circular dependencies between jobs?

No. Probe explicitly prevents circular dependencies. The scheduler uses the `dag.DetectCycleFn` utility in [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go) to check for cycles during validation. If a cycle is detected (for example, Job A needs Job B, and Job B needs Job A), the workflow fails to start with a validation error.

### How does Probe handle jobs that need to run multiple times (repeats)?

Probe respects the `repeat` configuration when evaluating dependencies. The `isJobFullyCompleted` method in [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go) checks that a dependency has not only reached `JobCompleted` status but also that its repeat counter meets or exceeds the target and the result is successful. A dependent job will not start until all repeats of its dependencies finish successfully.

### What is the difference between a failed job and a skipped job in Probe?

A **failed** job is one that executed but returned an error or non-zero exit code. A **skipped** job is one that never ran because one of its dependencies failed. In [`main/scheduler.go`](https://github.com/linyows/probe/blob/main/main/scheduler.go), `MarkJobsWithFailedDependencies` marks dependent jobs as failed internally. Then in [`main/workflow.go`](https://github.com/linyows/probe/blob/main/main/workflow.go), `updateSkippedJobsOutput` updates the final result to show these jobs with a `"skipped"` status, treating them as successful for flow-control purposes but indicating they did not execute.