How Probe Handles Embedded Jobs and Composite Workflows: A Technical Deep Dive

Probe implements composite workflows by treating steps with uses: embedded as references to external workflow files, loading and executing them independently while sharing variable context and output handling with the parent job.

The linyows/probe repository provides a sophisticated mechanism for embedded jobs (also called composite workflows) that enables reusable, modular workflow design. This capability allows you to decompose complex automation into smaller, testable sub-workflows that execute within the context of a parent job. Understanding how Probe handles these composite structures requires examining the path resolution, independent execution, and rendering logic implemented across the codebase.

How Embedded Jobs Work in Probe

Probe's composite workflow system operates on a simple but powerful abstraction: any step specifying uses: embedded triggers a specialized execution path that loads and runs a separate workflow file. This approach differs from simple action plugins because the embedded file is parsed as a complete Job struct with its own steps, defaults, and execution context.

The mechanism relies on three core principles:

• Relative Path Resolution: Embedded workflows are resolved relative to the parent workflow's directory using the basePath established during initial parsing
• Independent Execution: Each embedded job runs in its own printer context via job.RunIndependently, isolating output while preserving the parent's variable scope
• Flattened Visualization: DAG renderers expand embedded steps inline, showing the complete composite structure in both Mermaid and ASCII formats

Three-Phase Execution Architecture

Probe's handling of embedded jobs follows a distinct three-phase lifecycle implemented across dag.go, embedded/client.go, and the rendering engines.

Phase 1: Path Resolution and Workflow Loading

When Probe parses the initial workflow in probe.go:91-97, it stores the directory of the first YAML file in workflow.basePath. This path serves as the anchor for all relative lookups of embedded jobs.

The resolution process occurs in dag.go:60-95 within DagRendererBase.ResolvePath. This method:

1. Expands variables in the step's with.path using workflow context
2. Resolves the absolute path relative to basePath using filepath.Abs
3. Validates file existence before attempting to load

The LoadEmbeddedJob function (dag.go:98-117) then reads the referenced file and unmarshals it into a probe.Job struct, preparing it for execution.

Phase 2: Independent Job Execution

The embedded action plugin (actions/embedded/main.go:30-38) creates an embedded.Request and invokes embedded.Execute. The actual execution logic in embedded/client.go:65-100 handles several critical tasks:

• Defaults Propagation: applyDefaultsToSteps merges any defaults defined in the embedded job into its individual steps
• Isolated Output: A fresh printer instance (probe.NewPrinter(true, []string{jobID})) ensures the embedded job's output streams are captured separately
• Error Classification: System-level failures (file not found, YAML parse errors) return as Go error types and abort the workflow, while job-level failures populate Result.Res.Error allowing the parent workflow to continue

The embedded job runs via job.RunIndependently, returning success status, outputs, and reports that merge into the parent step's result map through standard Step.finalize handling.

Phase 3: DAG Rendering and Visualization

Both DagMermaidRenderer (dag_mermaid.go:65-84) and DagAsciiRenderer (dag_ascii.go) implement expansion logic for composite workflows. When rendering:

1. The renderer detects step.Uses == "embedded"
2. Calls ExpandPath and ResolvePath to locate the embedded file
3. Loads the embedded job via LoadEmbeddedJob
4. Iterates over embeddedJob.Steps to emit sub-nodes inline

This produces a flattened visual representation showing the full composite workflow hierarchy without requiring manual documentation of the sub-workflow structure.

Code Example: Creating a Composite Workflow

The following example demonstrates a parent workflow that embeds an integration testing suite as a reusable sub-workflow:

# parent.yml

jobs:
  - name: Deploy
    steps:
      - name: Run unit tests
        uses: shell
        with:
          cmd: go test ./...
      - name: Run integration suite (embedded)
        uses: embedded
        with:
          path: ./embedded/integration.yml
          vars:
            env: staging

# embedded/integration.yml

jobs:
  - name: Integration
    steps:
      - name: Start containers
        uses: docker
        with:
          compose: docker-compose.yml
      - name: Run tests
        uses: shell
        with:
          cmd: go test -tags=integration ./...

When executed, Probe loads embedded/integration.yml relative to parent.yml's directory, passes the vars.env variable into the embedded context, and executes the integration steps sequentially. The outputs from both "Start containers" and "Run tests" become available to subsequent steps in the parent workflow. Running probe render --format mermaid parent.yml displays all four steps in a single composite graph.

Key Implementation Files

Understanding Probe's embedded job architecture requires familiarity with these specific source files:

• probe.go – Establishes workflow.basePath for relative path resolution during initial workflow loading
• dag.go – Contains DagRendererBase.ResolvePath (lines 60-95) and LoadEmbeddedJob (lines 98-117) for path handling and workflow loading
• embedded/client.go – Implements the request lifecycle including parsing, defaults application, and independent execution via job.RunIndependently
• actions/embedded/main.go – Provides the plugin glue that invokes the embedded client from the action system
• dag_mermaid.go and dag_ascii.go – Contain the rendering logic (lines 65-84 in the Mermaid implementation) that expands embedded steps for visualization
• step.go – Executes the embedded action and handles result aggregation through the standard step output mechanism

Summary

Probe handles embedded jobs and composite workflows through a robust three-phase system:

• Path resolution uses workflow.basePath and DagRendererBase.ResolvePath to locate embedded workflow files relative to the parent
• Independent execution via embedded/client.go runs sub-workflows in isolated printer contexts while sharing variable scope, with distinct handling for system-level versus job-level errors
• Visual expansion in DAG renderers produces complete workflow diagrams by inline-expanding embedded steps during Mermaid or ASCII generation
• Result aggregation merges embedded outputs into the parent job's step results, enabling complex workflow-of-workflows patterns without custom orchestration

Frequently Asked Questions

How does Probe resolve paths for embedded workflow files?

Probe resolves embedded workflow paths relative to the parent workflow's directory using the basePath variable set during initial parsing in probe.go:91-97. The DagRendererBase.ResolvePath method in dag.go:60-95 expands workflow variables in the path string, then uses filepath.Abs to resolve the absolute path relative to basePath, verifying file existence before loading.

What happens when an embedded job fails?

Probe distinguishes between system-level failures and job-level failures. System errors (file not found, YAML parse errors) return as Go error types and abort the entire workflow. Job execution failures (failed steps within the embedded workflow) populate the Result.Res.Error field, allowing the parent workflow to continue execution and handle the failure according to its own error handling logic.

Can embedded jobs access variables from the parent workflow?

Yes, embedded jobs share the same variable context as the parent workflow. When defining an embedded step, the with.vars map passes variables into the embedded job's evaluation context. These variables become available for expansion within the embedded workflow file's step definitions, enabling dynamic configuration of reusable sub-workflows.

How does Probe visualize composite workflows in DAG renders?

Both DagMermaidRenderer (dag_mermaid.go:65-84) and DagAsciiRenderer detect steps with uses: embedded, load the referenced workflow file via LoadEmbeddedJob, and iterate over the embedded job's steps to emit them as sub-nodes inline. This produces a flattened visual representation showing the complete composite workflow hierarchy rather than treating the embedded job as an opaque single node.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}