How gh-aw Compiles Markdown Workflows into GitHub Actions YAML: The 4-Phase Pipeline Explained

The gh-aw CLI compiles markdown workflows into GitHub Actions YAML through a four-phase pipeline—parse and load, validate, generate YAML, and final validation—implemented across compiler.go and compiler_yaml.go in the github/gh-aw repository.

When you run gh aw compile, you trigger a sophisticated compilation process that transforms human-readable markdown into the machine-readable .lock.yml files that GitHub Actions executes. This article breaks down exactly how gh-aw compiles markdown workflows into GitHub Actions YAML, referencing the actual source implementation in the github/gh-aw repository.

The Four-Phase Compilation Pipeline

The compilation process is logically divided into four distinct phases, each handled by specific functions in the pkg/workflow directory.

Phase 1: Parse and Load

The pipeline begins in pkg/workflow/compiler.go with the CompileWorkflow method. This function orchestrates the initial parsing by calling ParseWorkflowFile, which reads the markdown file from disk and extracts the YAML front-matter from the markdown body.

The parser builds a WorkflowData structure that holds everything the compiler needs, including workflow metadata, permissions, and the raw markdown content. The parser also computes a front-matter hash via parser.ComputeFrontmatterHashFromFile to ensure reproducibility—this hash is later embedded as a comment in the generated YAML.

Phase 2: Validate

Once loaded, the workflow data passes through validateWorkflowData in compiler.go (lines 14-63). This phase performs comprehensive safety checks before any YAML generation occurs:

• Expression safety – validateExpressionSafety ensures only whitelisted ${{ … }} patterns are used
• Runtime-import validation – validateRuntimeImportFiles guarantees imported markdown files can be loaded at runtime
• Permission and feature-flag checks – Validates dangerous permissions, safe-output settings, network firewall configurations, and concurrency expressions
• Tool-set consistency – Ensures all referenced tools are compatible

Errors and warnings are emitted using the console package via formatCompilerError and formatCompilerMessage, providing formatted console output with consistent styling.

Phase 3: Generate YAML

The core transformation happens in generateYAML (located in pkg/workflow/compiler_yaml.go, lines 59-92). This phase constructs the actual GitHub Actions YAML through several sub-components:

Job Construction – buildJobs creates job objects, resolves inter-job dependencies, and validates against duplicate step IDs using the internal JobManager.

Header Generation – generateWorkflowHeader writes a structured comment block containing:

• The gh-aw logo and version
• Source file reference
• Description metadata
• Import/include manifest
• Front-matter hash for reproducibility

Body Generation – generateWorkflowBody writes top-level workflow fields (name, on, permissions, concurrency, env) using deterministic ordering via yaml.MarshalWithFieldOrder from pkg/workflow/yaml.go.

Prompt Chunking – The markdown body is split by splitContentIntoChunks (max 20KB per chunk, max 5 chunks) to obey GitHub Actions step size limits. Each chunk becomes a heredoc block in a unified "prompt generation" step created by generateUnifiedPromptCreationStep. The ExpressionExtractor replaces ${{ }} expressions with environment variable placeholders for safe runtime interpolation.

Post-Steps – Additional steps are appended for safe-output collection (generateOutputCollectionStep), artifact upload, and workflow overview generation (generateWorkflowOverviewStep).

Phase 4: Final Validation and Output

The final phase occurs in generateAndValidateYAML and writeWorkflowOutput (back in compiler.go). Before writing to disk, the compiler performs rigorous validation:

• Expression size limits – validateExpressionSizes ensures no expression exceeds 21KB
• Template injection prevention – validateNoTemplateInjection guards against malicious template patterns
• Schema validation – validateGitHubActionsSchema validates against the embedded schemas/github-workflow.json (unless --no-validation is used)
• Container and runtime checks – Validates container images, runtime packages, and repository features

If validation passes, writeWorkflowOutput writes the content to <filename>.lock.yml. This function:

• Skips writing if the content is unchanged (preserving timestamps)
• Warns if the file exceeds the 500KB recommended limit (MaxLockFileSize)
• Prints a success message with the relative path and file size

If validation fails, an *.invalid.yml file is written for debugging, and a formatted console error is returned.

Key Implementation Files in github/gh-aw

Understanding the compilation pipeline requires familiarity with these specific files in the repository:

File	Role	Key Functions
pkg/workflow/compiler.go	Core orchestration and validation	CompileWorkflow, validateWorkflowData, generateAndValidateYAML, writeWorkflowOutput
pkg/workflow/compiler_yaml.go	YAML generation and prompt chunking	generateYAML, buildJobs, splitContentIntoChunks, generateUnifiedPromptCreationStep
pkg/workflow/yaml.go	Deterministic YAML marshaling	MarshalWithFieldOrder, UnquoteYAMLKey, CleanYAMLNullValues
pkg/parser/frontmatter.go	Front-matter parsing and hashing	ParseWorkflowFile, ComputeFrontmatterHashFromFile
pkg/workflow/job_manager.go	Job dependency resolution	JobManager, buildJobs
pkg/workflow/expressions.go	Expression safety and extraction	validateExpressionSafety, ExpressionExtractor
schemas/github-workflow.json	Embedded validation schema	Used by validateGitHubActionsSchema

Practical Usage Examples

Compiling from the CLI

The most common way to compile markdown workflows into GitHub Actions YAML is using the gh aw compile command:

$ gh aw compile .github/workflows/example.md
⚙️  Parsing workflow file
✅  Successfully generated workflow (example.md) (12 KB)

This creates .github/workflows/example.lock.yml containing the compiled GitHub Actions workflow. The CLI automatically handles all four compilation phases, including validation against the embedded JSON schema.

Programmatic Compilation in Go

You can also invoke the compiler programmatically within your own Go applications:

package main

import (
    "log"

    "github.com/github/gh-aw/pkg/workflow"
)

func main() {
    comp := workflow.NewCompiler()
    // optional: comp.SetStrictMode(true) // treat missing permissions as errors
    
    if err := comp.CompileWorkflow(".github/workflows/example.md"); err != nil {
        log.Fatalf("Compilation failed: %v", err)
    }
}

The CompileWorkflow method triggers the full pipeline: parsing the markdown, validating safety constraints, generating the YAML with chunked prompts, and writing the .lock.yml file.

Inspecting the Generated Lock File

After compilation, examine the generated file to verify the transformation:

$ cat .github/workflows/example.lock.yml | head -n 20

# Generated by gh-aw (github.com/github/gh-aw)

# Source: .github/workflows/example.md

name: "example"
on:
  push:
    branches:
      - main

permissions: {}

concurrency:
  group: "example-${{ github.run_id }}"
  cancel-in-progress: true

env:
  GH_AW_PROMPT: /tmp/gh-aw/aw_prompt.txt

The header includes the front-matter hash for reproducibility, while the body uses deterministic field ordering guaranteed by yaml.MarshalWithFieldOrder in pkg/workflow/yaml.go.

Summary

• Four-phase pipeline: gh-aw compiles markdown workflows into GitHub Actions YAML through Parse & Load, Validate, Generate YAML, and Final Validation phases.
• Entry point: The CompileWorkflow function in pkg/workflow/compiler.go orchestrates the entire process, from reading the markdown file to writing the .lock.yml output.
• Safety first: The validation phase checks expression safety, permissions, template injection risks, and validates against the embedded schemas/github-workflow.json schema.
• Prompt chunking: Large markdown bodies are automatically split into 20KB chunks (max 5) using splitContentIntoChunks to comply with GitHub Actions step size limits.
• Deterministic output: The yaml.MarshalWithFieldOrder function ensures consistent field ordering in the generated YAML, while the front-matter hash guarantees reproducibility.

Frequently Asked Questions

What is the maximum size for a compiled workflow file?

The gh-aw compiler warns when the generated .lock.yml file exceeds 500KB (defined as MaxLockFileSize in pkg/workflow/compiler.go). While GitHub Actions does not enforce a strict limit, files larger than this threshold may hit repository or performance constraints. If your workflow exceeds this limit, consider splitting it into multiple workflows or reducing embedded content.

How does gh-aw handle large markdown prompts that exceed GitHub Actions step limits?

When the markdown body exceeds 20KB, gh-aw automatically chunks the content using splitContentIntoChunks in pkg/workflow/compiler_yaml.go. The compiler splits the content into a maximum of 5 chunks, each wrapped in a heredoc block within the generated YAML. This ensures compliance with GitHub Actions step size constraints while preserving the full prompt content for runtime AI processing.

Can I skip schema validation when compiling a workflow?

Yes. The compiler supports skipping the final JSON schema validation step through the --no-validation CLI flag or programmatically by setting the appropriate compiler option. When enabled, the validateGitHubActionsSchema function in pkg/workflow/compiler_yaml.go is bypassed, though expression size checks (21KB limit) and template injection prevention (validateNoTemplateInjection) still run for safety.

What happens if validation fails during compilation?

If any validation check fails during the generateAndValidateYAML phase, gh-aw writes the invalid content to an *.invalid.yml file for debugging purposes instead of the standard .lock.yml file. The compiler returns a formatted error message using formatCompilerError in pkg/workflow/compiler.go, which provides detailed context about which validation rule failed (expression safety, permission checks, or schema validation) and the specific line or file causing the issue.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}