How gh-aw Compiles Markdown Workflows to GitHub Actions YAML Internally

The gh-aw CLI compiles markdown workflows to GitHub Actions YAML through a four-phase pipeline—parse, validate, generate, and finalize—implemented across compiler.go and compiler_yaml.go in the pkg/workflow package.

The gh-aw extension transforms human-readable Markdown workflow definitions into executable GitHub Actions YAML files. Understanding how it compiles markdown workflows to GitHub Actions YAML internally helps developers debug compilation errors and optimize their workflow definitions for the AI-driven execution model.

The Four-Phase Compilation Pipeline

The compilation process in gh-aw follows a strict sequence implemented in pkg/workflow/compiler.go. Each phase handles a specific transformation from the source markdown to the final .lock.yml output.

Phase 1: Parse and Load

The pipeline begins with (*Compiler).CompileWorkflow, which orchestrates the entire process. It first invokes ParseWorkflowFile to read the markdown from disk and extract the YAML front-matter.

During this phase, the compiler:

• Reads the source file from the path provided (e.g., .github/workflows/example.md)
• Splits the front-matter block from the markdown body using the parser package
• Computes a front-matter hash via parser.ComputeFrontmatterHashFromFile for reproducibility
• Constructs a WorkflowData structure that holds all metadata needed for subsequent phases

This phase is located at compiler.go lines 81-100.

Phase 2: Validate

Before generating any YAML, the compiler runs validateWorkflowData to ensure the workflow is safe and compliant. This phase performs comprehensive checks defined in compiler.go lines 14-63.

Validation includes:

• Expression safety: validateExpressionSafety ensures only whitelisted ${{ … }} patterns are used to prevent injection attacks
• Runtime imports: validateRuntimeImportFiles guarantees that any imported markdown can be loaded at runtime
• Permission and security checks: Validates dangerous permissions, safe-output configurations, network firewall settings, and concurrency group expressions
• Tool-set consistency: Ensures all referenced tools and features are compatible

Errors and warnings are emitted through formatCompilerError or formatCompilerMessage, which use console.FormatError for uniform CLI styling.

Phase 3: Generate YAML

The core transformation happens in generateYAML (located in compiler_yaml.go lines 59-92). This phase constructs the actual GitHub Actions YAML structure through several sub-components:

Job Construction

The buildJobs function creates job objects, resolves inter-job dependencies, and validates against duplicate step IDs using the JobManager from pkg/workflow/job_manager.go.

Header Generation

generateWorkflowHeader writes:

• A logo comment identifying the file as generated by gh-aw
• Source file attribution
• Description and import manifest
• The front-matter hash for traceability

Body Generation

generateWorkflowBody serializes top-level fields in deterministic order using MarshalWithFieldOrder from pkg/workflow/yaml.go:

• name
• on (triggers)
• permissions
• concurrency
• run-name
• env
• Rendered jobs from JobManager

Prompt Chunking

For AI-driven workflows, the markdown body is split into chunks of maximum 20 KB, with a hard limit of 5 chunks total, via splitContentIntoChunks. Each chunk becomes a heredoc block in a unified "prompt creation" step generated by generateUnifiedPromptCreationStep. Expressions inside the markdown are extracted using NewExpressionExtractor and replaced with environment variable placeholders, enabling runtime interpolation while keeping the AI prompt clean.

Post-Steps

Additional steps are appended for:

• Creating aw_info.json (generateCreateAwInfo)
• Workflow overview (generateWorkflowOverviewStep)
• Safe output collection (generateOutputCollectionStep)
• Artifact upload

Phase 4: Finalize and Write Output

The final phase, implemented in generateAndValidateYAML (compiler.go lines 67-88), ensures the generated YAML is safe and compliant before writing to disk.

Validation steps include:

• Expression size limits: validateExpressionSizes ensures no expression exceeds 21 KB
• Template injection prevention: validateNoTemplateInjection guards against unsafe template patterns
• Schema validation: validateGitHubActionsSchema validates against the embedded schemas/github-workflow.json (unless skipValidation is true)
• Container and runtime checks: Validates container images, runtime packages, firewall rules, and repository features

If validation passes, writeWorkflowOutput (compiler.go lines 51-78) writes the generated YAML to a .lock.yml file next to the source markdown. This function:

• Skips writing if the content is unchanged (preserving timestamps)
• Warns when the file exceeds the 500 KB recommended limit (MaxLockFileSize)
• Prints a success message with the relative path and file size

If validation fails, an *.invalid.yml file is written for debugging, and a formatted console error is returned.

Key Implementation Files

The compilation pipeline spans several files in the pkg/workflow directory:

• compiler.go: Core orchestration including CompileWorkflow, validation dispatcher, YAML generation, and file output handling
• compiler_yaml.go: YAML generation logic including job building, header generation, prompt chunking, and post-steps
• yaml.go: Deterministic YAML marshaling utilities (MarshalWithFieldOrder, UnquoteYAMLKey, CleanYAMLNullValues) using goccy/go-yaml
• job_manager.go: Job dependency resolution and step validation
• expressions.go: Expression extraction and safety validation
• parser/frontmatter.go: Front-matter parsing and hash computation

The embedded JSON schema at schemas/github-workflow.json provides the validation rules for the final output.

How Prompt Generation and Chunking Works

A distinctive feature of gh-aw is its AI-driven execution model, which requires special handling of the markdown body:

1. Content Splitting: The splitContentIntoChunks function in compiler_yaml.go divides the markdown body into chunks of maximum 20 KB, with a hard limit of 5 chunks total. This prevents exceeding GitHub Actions step size limits.

2. Expression Extraction: Before chunking, NewExpressionExtractor identifies ${{ … }} expressions and replaces them with environment variable placeholders. This allows the AI engine to receive clean text while preserving the ability to interpolate values at runtime.

3. Heredoc Generation: Each chunk is wrapped in a shell heredoc within a unified prompt creation step generated by generateUnifiedPromptCreationStep. These steps write the chunks to temporary files (e.g., /tmp/gh-aw/prompt-0.txt) that the AI runtime can access.

This approach ensures that large markdown workflows can be processed by GitHub Actions without hitting the 48 KB step limit or 21 KB expression limit, while maintaining the dynamic capabilities of the workflow expression syntax.

Example: Compiling a Workflow

CLI Compilation

To compile a markdown workflow from the command line:

$ gh aw compile .github/workflows/example.md
⚙️  Parsing workflow file
✅  Successfully generated workflow (example.md) (12 KB)

This command creates a Compiler instance, invokes CompileWorkflow, and writes the output to .github/workflows/example.lock.yml.

Programmatic Compilation

You can also compile workflows programmatically using the Go API:

package main

import (
    "log"

    "github.com/github/gh-aw/pkg/workflow"
)

func main() {
    comp := workflow.NewCompiler()
    // optional: comp.SetStrictMode(true) // treat missing permissions as errors
    if err := comp.CompileWorkflow(".github/workflows/example.md"); err != nil {
        log.Fatalf("Compilation failed: %v", err)
    }
}

The CompileWorkflow method triggers the full four-phase pipeline described above.

Inspecting the Generated Output

The generated .lock.yml file includes metadata comments and deterministic field ordering:

$ cat .github/workflows/example.lock.yml | head -n 20

# Generated by gh-aw (github.com/github/gh-aw)

# Source: .github/workflows/example.md

name: "example"
on:
  push:
    branches:
      - main

permissions: {}

concurrency:
  group: "example-${{ github.run_id }}"
  cancel-in-progress: true

env:
  GH_AW_PROMPT: /tmp/gh-aw/aw_prompt.txt

The top-level field ordering (name, on, permissions, …) is guaranteed by MarshalWithFieldOrder in pkg/workflow/yaml.go.

Chunked Prompt Example

When the markdown body exceeds 20 KB, the compiler generates multiple heredoc steps:

- name: Build prompt – chunk 1
  run: |
    cat <<'EOF' > /tmp/gh-aw/prompt-0.txt
    ...first 20 KB of cleaned markdown...
    EOF

- name: Build prompt – chunk 2
  run: |
    cat <<'EOF' > /tmp/gh-aw/prompt-1.txt
    ...next 20 KB...
    EOF

These steps are generated by splitContentIntoChunks (see compiler_yaml.go lines 89-115).

Summary

• Four-phase pipeline: gh-aw compiles markdown workflows to GitHub Actions YAML through parse, validate, generate, and finalize phases implemented in pkg/workflow/compiler.go and pkg/workflow/compiler_yaml.go.
• Safety-first validation: The compiler checks expression safety, permissions, template injection, and schema compliance against schemas/github-workflow.json before writing output.
• AI-ready chunking: Large markdown bodies are split into 20 KB heredoc chunks with expression extraction to bypass GitHub Actions step limits while preserving runtime interpolation.
• Deterministic output: The MarshalWithFieldOrder utility ensures consistent YAML field ordering, and front-matter hashes provide reproducibility.
• Lock file generation: Valid workflows are written to .lock.yml files next to their source markdown, with automatic change detection and size warnings for files exceeding 500 KB.

Frequently Asked Questions

What is the entry point for compiling a markdown workflow in gh-aw?

The entry point is the CompileWorkflow method in pkg/workflow/compiler.go. When you run gh aw compile <path>.md, the CLI creates a Compiler instance and calls this method, which orchestrates the four-phase pipeline: parsing the markdown, validating the workflow data, generating the YAML structure, and writing the final lock file.

How does gh-aw handle large markdown files that exceed GitHub Actions step size limits?

The compiler uses the splitContentIntoChunks function in pkg/workflow/compiler_yaml.go to divide the markdown body into chunks of maximum 20 KB, with a hard limit of 5 chunks total. Each chunk becomes a heredoc block in a unified prompt creation step generated by generateUnifiedPromptCreationStep. Before chunking, NewExpressionExtractor replaces ${{ … }} expressions with environment variable placeholders to preserve runtime interpolation capabilities.

What validation checks does gh-aw perform before generating the final YAML?

Before writing the .lock.yml file, generateAndValidateYAML in compiler.go executes several safety checks: validateExpressionSizes ensures no expression exceeds 21 KB; validateNoTemplateInjection guards against unsafe template patterns; and validateGitHubActionsSchema validates the output against the embedded schemas/github-workflow.json. The compiler also checks container images, runtime packages, firewall rules, and repository features during the validation phase.

Why does gh-aw generate a .lock.yml file instead of overwriting the original markdown?

The .lock.yml file serves as the compiled, machine-readable artifact that GitHub Actions executes, while preserving the human-readable .md source for editing. This separation allows gh-aw to perform change detection (skipping writes when content is unchanged to preserve timestamps), embed metadata like front-matter hashes for reproducibility, and warn when the compiled output exceeds the 500 KB recommended limit (MaxLockFileSize). If validation fails, the compiler writes an *.invalid.yml file for debugging instead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}