How Probe Handles Job Dependencies with the `needs` Keyword

Probe implements job dependencies through a Needs []string field on the Job struct, validated by a JobScheduler that checks for circular dependencies and ensures a job only runs after all declared upstream jobs complete successfully.

Probe is a lightweight workflow runner inspired by GitHub Actions. When you define jobs in a workflow file, the needs keyword lets you specify which jobs must finish before another can start. This article explains how Probe parses these dependencies, validates them at startup, and enforces them during execution according to the source code in linyows/probe.

How the needs Field Declares Dependencies

The Job Struct Definition

In main/job.go, the Job struct contains a Needs field that captures the dependency list from your YAML workflow:

type Job struct {
    ID      string            `yaml:"id,omitempty"`
    Name    string            `yaml:"name,omitempty"`
    Needs   []string          `yaml:"needs,omitempty"`   // ← dependency names
    Steps   []Step            `yaml:"steps,omitempty"`
    // … other fields …
}

When Probe loads a workflow, it unmarshals the YAML into these structs. The needs array holds the id or name values of jobs that must complete before this job can run.

Building and Validating the Dependency Graph

Registering Jobs in the Scheduler

Before execution begins, Probe creates a JobScheduler in main/scheduler.go to manage the workflow state. The AddJob method registers each job, generating an ID if one is missing and initializing tracking maps for status, results, and repeat counters:

func (js *JobScheduler) AddJob(job Job) string {
    if job.ID == "" {
        job.ID = generateJobID()
    }
    js.jobs[job.ID] = job
    js.status[job.ID] = JobPending
    // … initialize repeat counters …
    return job.ID
}

Validating Dependencies and Detecting Cycles

Once all jobs are registered, ValidateDependencies performs two critical checks:

1. Existence validation: Every name in Needs must correspond to a registered job ID.
2. Cycle detection: The scheduler uses a DAG utility (dag.DetectCycleFn) to ensure no circular dependencies exist (e.g., A needs B, B needs A).

func (js *JobScheduler) ValidateDependencies() error {
    // 1️⃣ Ensure every `need` points to an existing job
    for _, job := range js.jobs {
        for _, dep := range job.Needs {
            if _, ok := js.jobs[dep]; !ok {
                return fmt.Errorf("job '%s' depends on non-existent job '%s'", job.ID, dep)
            }
        }
    }
    // 2️⃣ Detect cycles using the DAG helper
    return js.checkCircularDependencies()
}

If validation fails, Probe aborts the workflow immediately with an error message.

Runtime Execution and Dependency Resolution

Checking if a Job Can Run

During the execution phase, the scheduler determines readiness via CanRunJob. A job is runnable only when all jobs listed in its Needs array have fully completed all their repeat iterations successfully:

func (js *JobScheduler) CanRunJob(jobID string) bool {
    job := js.jobs[jobID]
    if js.status[jobID] != JobPending {
        return false
    }
    // All dependencies must be fully completed (including repeats)
    for _, dep := range job.Needs {
        if !js.isJobFullyCompleted(dep) {
            return false
        }
    }
    return true
}

The helper isJobFullyCompleted verifies both the job status and that the success result is true:

func (js *JobScheduler) isJobFullyCompleted(jobID string) bool {
    if js.status[jobID] != JobCompleted {
        return false
    }
    target := js.repeatTargets[jobID]
    counter := js.repeatCounters[jobID]
    return counter >= target && js.results[jobID]
}

The Main Execution Loop

The workflow engine in main/workflow.go orchestrates execution through startJobsWithDependencies. It repeatedly queries the scheduler for runnable jobs and launches them concurrently:

func (w *Workflow) startJobsWithDependencies(ctx JobContext) error {
    for !ctx.JobScheduler.AllJobsCompleted() {
        runnable := ctx.JobScheduler.GetRunnableJobs()
        if len(runnable) == 0 {
            // No job can proceed → mark jobs whose dependencies failed as skipped
            w.handleNoRunnableJobs(ctx)
            continue
        }
        w.processRunnableJobs(runnable, ctx)
        ctx.JobScheduler.wg.Wait()
    }
    return nil
}

GetRunnableJobs iterates over all pending jobs and returns those where CanRunJob returns true.

Handling Failed Dependencies and Skipped Jobs

When a dependency fails, Probe prevents downstream execution. The MarkJobsWithFailedDependencies method in main/scheduler.go identifies jobs whose dependencies have failed or completed unsuccessfully, marking them as failed:

func (js *JobScheduler) MarkJobsWithFailedDependencies() []string {
    var skipped []string
    for id, job := range js.jobs {
        if js.status[id] != JobPending {
            continue
        }
        for _, dep := range job.Needs {
            if js.status[dep] == JobFailed || (js.status[dep] == JobCompleted && !js.results[dep]) {
                js.status[id] = JobFailed
                js.results[id] = false
                skipped = append(skipped, id)
                break
            }
        }
    }
    return skipped
}

Finally, main/workflow.go updates the output for these skipped jobs via updateSkippedJobsOutput, setting their status to "skipped" in the final report:

func (w *Workflow) updateSkippedJobsOutput(skippedJobs []string, rs *Result) {
    for _, jobID := range skippedJobs {
        if jr, ok := rs.Jobs[jobID]; ok {
            jr.mutex.Lock()
            jr.EndTime = jr.StartTime
            jr.Status = "skipped"
            jr.Success = true // treated as successful, like a skipif
            jr.mutex.Unlock()
        }
    }
}

Key Implementation Files

File	Purpose	Link
main/job.go	Definition of Job (including Needs) and per-job execution logic	job.go
main/scheduler.go	Job scheduler that registers jobs, validates dependencies, and decides runnable jobs	scheduler.go
main/workflow.go	Orchestrates the whole workflow, repeatedly invoking the scheduler, handling dead-locks & skipped jobs	workflow.go
main/executor.go	Executes a single job (including repeat & async handling) once the scheduler marks it runnable	executor.go
dag/*	Utility for detecting dependency cycles used by the scheduler	dag package

Summary

• Declaration: Probe uses the Needs []string field in main/job.go to capture dependency lists from YAML workflow files.
• Validation: The JobScheduler in main/scheduler.go validates that all referenced jobs exist and uses a DAG utility to detect circular dependencies before execution starts.
• Execution: The scheduler only marks a job as runnable when CanRunJob confirms all dependencies have fully completed (including all repeat iterations) successfully.
• Failure handling: If a dependency fails, MarkJobsWithFailedDependencies marks downstream jobs as failed, and main/workflow.go reports them as skipped in the final output.

Frequently Asked Questions

What happens if a job lists a dependency that does not exist?

Probe validates the dependency graph before running any jobs. In main/scheduler.go, the ValidateDependencies function checks that every name in a job's Needs array corresponds to a registered job ID. If a dependency is missing, Probe returns an error immediately and aborts the workflow.

Does Probe support circular dependencies between jobs?

No. Probe explicitly prevents circular dependencies. The scheduler uses the dag.DetectCycleFn utility in main/scheduler.go to check for cycles during validation. If a cycle is detected (for example, Job A needs Job B, and Job B needs Job A), the workflow fails to start with a validation error.

How does Probe handle jobs that need to run multiple times (repeats)?

Probe respects the repeat configuration when evaluating dependencies. The isJobFullyCompleted method in main/scheduler.go checks that a dependency has not only reached JobCompleted status but also that its repeat counter meets or exceeds the target and the result is successful. A dependent job will not start until all repeats of its dependencies finish successfully.

What is the difference between a failed job and a skipped job in Probe?

A failed job is one that executed but returned an error or non-zero exit code. A skipped job is one that never ran because one of its dependencies failed. In main/scheduler.go, MarkJobsWithFailedDependencies marks dependent jobs as failed internally. Then in main/workflow.go, updateSkippedJobsOutput updates the final result to show these jobs with a "skipped" status, treating them as successful for flow-control purposes but indicating they did not execute.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}