# How Quarkdown Compiles .qd Files to HTML and PDF: A Deep Dive into the Pipeline

> Deep dive into the Quarkdown compilation pipeline. Discover how .qd files transform into HTML and PDF via lexing, parsing, AST rendering, and optional PDF generation with Puppeteer.

- Repository: [Giorgio Garofalo/quarkdown](https://github.com/iamgio/quarkdown)
- Tags: deep-dive
- Published: 2026-04-29

---

**Quarkdown transforms .qd source files into HTML or PDF through a 10-stage pipeline that lexes and parses the source into an AST, expands function calls, renders the tree to HTML, and optionally delegates to Puppeteer for PDF generation.**

The Quarkdown compilation flow converts lightweight markup documents into production-ready HTML or PDF artifacts through a modular, extensible pipeline. Whether invoked via the CLI or embedded as a library, the process follows a strict sequence of transformation stages defined in `quarkdown-core` and `quarkdown-html`. Each stage is a pure, composable component that passes data to the next via the `PipelineChainFactory`.

## Stage 1: CLI Entry and Pipeline Initialization

Compilation begins in [`CompileCommand.kt`](https://github.com/iamgio/quarkdown/blob/main/CompileCommand.kt), which builds a `Pipeline` instance based on user flags like `--pdf` or `--pipe`.

```kotlin
// quarkdown-cli/src/main/kotlin/com/quarkdown/cli/exec/CompileCommand.kt
class CompileCommand : ExecuteCommand("compile") {
    …
    override fun createExecutionStrategy(cliOptions: CliOptions) = FileExecutionStrategy(source)
}

```

`FileExecutionStrategy` constructs a `Pipeline` object from [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/Pipeline.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/Pipeline.kt) and calls `pipeline.execute(source)`. The `Pipeline` class stores the mutable context, registered libraries, and rendering components before triggering the stage chain.

## Stage 2: Building the Compilation Stage Chain

The `Pipeline.execute` method delegates to `executeUnwrapped`, which builds the complete stage chain via `PipelineChainFactory.fullChain` in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/PipelineChainFactory.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/PipelineChainFactory.kt).

```kotlin
object PipelineChainFactory {
    fun fullChain(
        source: CharSequence,
        renderingComponents: RenderingComponents,
        options: PipelineOptions,
    ): PipelineStage<Unit, Set<OutputResource>> =
        AttachmentStage then
        LibrariesRegistrationStage then
        LexingStage(source) then
        ParsingStage then
        AttributesUpdateStage(renderingComponents.postRenderer.preferredMediaStorageOptions) then
        FunctionCallExpansionStage then
        TreeTraversalStage then
        RenderingStage(renderingComponents.nodeRenderer) thenOptionally
        PostRenderingStage(renderingComponents.postRenderer).takeIf { options.wrapOutput } then
        AfterAllRenderingPeek then
        ResourceGenerationStage(renderingComponents.postRenderer)
}

```

The `thenOptionally` operator skips **PostRenderingStage** when `wrapOutput` is disabled, allowing raw HTML generation without document templates.

## Stage 3: Core Processing Stages

Each stage implements `PipelineStage<I, O>` and performs a single responsibility:

- **AttachmentStage** – Attaches the `Pipeline` instance to the document's `Context`, enabling sub-document support. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/AttachmentStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/AttachmentStage.kt).

- **LibrariesRegistrationStage** – Registers standard-library `Library` sets into the context so that `.function` calls can be resolved later. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/LibrariesRegistrationStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/LibrariesRegistrationStage.kt).

- **LexingStage** – Tokenizes the raw `.qd` source text using the flavor's lexer. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/LexingStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/LexingStage.kt).

- **ParsingStage** – Transforms the token stream into an **AST** rooted at `AstRoot`. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/ParsingStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/ParsingStage.kt).

- **AttributesUpdateStage** – Stores the root AST in the context, registers sub-documents in the graph, and merges media-storage options required by the renderer (HTML requires local media storage). Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/AttributesUpdateStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/AttributesUpdateStage.kt).

- **FunctionCallExpansionStage** – Walks the AST and expands every Quarkdown function call (e.g., `.foo {…}`) via `FunctionCallNodeExpander`. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/FunctionCallExpansionStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/FunctionCallExpansionStage.kt).

- **TreeTraversalStage** – Executes the flavor's tree iterator to visit every node, allowing hooks to modify the tree before rendering. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/TreeTraversalStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/TreeTraversalStage.kt).

## Stage 4: Rendering and Resource Generation

After AST processing, the pipeline enters the output generation phase:

- **RenderingStage** – Calls the flavor-specific `NodeRenderer` (HTML, plain text, etc.) to produce the main content string. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/RenderingStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/RenderingStage.kt).

- **PostRenderingStage** – Wraps the rendered content into a final document template including HTML `<head>`, CSS, and scripts. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/PostRenderingStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/PostRenderingStage.kt).

- **ResourceGenerationStage** – Uses the `PostRenderer` to convert the final string into concrete `OutputResource` objects (HTML files, media assets, sub-document files). It recursively processes sub-documents. Located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/ResourceGenerationStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/ResourceGenerationStage.kt).

The HTML renderer is instantiated via [`HtmlPostRenderer.kt`](https://github.com/iamgio/quarkdown/blob/main/HtmlPostRenderer.kt) in the `quarkdown-html` module, producing [`main.html`](https://github.com/iamgio/quarkdown/blob/main/main.html) and associated assets like CSS and fonts.

## Stage 5: PDF Generation via Puppeteer

When the `--pdf` flag is set, the compilation flow extends the pipeline with a decorator pattern. The `HtmlRendererExtension` wraps the standard HTML post-renderer:

```kotlin
// quarkdown-html/src/main/kotlin/com/quarkdown/rendering/html/extension/HtmlRendererExtension.kt
if (options.exportPdf) {
    postRenderer = PdfHtmlPostRendererDecorator(
        postRenderer = postRenderer,
        options = pdfOptions,
    )
}

```

The `PdfHtmlPostRendererDecorator` (located in [`quarkdown-html/src/main/kotlin/com/quarkdown/rendering/html/pdf/PdfHtmlPostRendererDecorator.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-html/src/main/kotlin/com/quarkdown/rendering/html/pdf/PdfHtmlPostRendererDecorator.kt)) performs three operations:

1. Generates standard HTML resources via the wrapped post-renderer.
2. Copies resources to a temporary directory.
3. Invokes `HtmlPdfExporter` (which uses **Puppeteer**) to render the temporary HTML into a single PDF file.

```kotlin
override fun generateResources(rendered: CharSequence): Set<OutputResource> {
    val resources = postRenderer.generateResources(rendered)
    // …create temp dir, copy resources, export PDF…
    return out?.let(BinaryOutputArtifact::fromFile)
               ?.also { tempDirectory.deleteRecursively() }
               ?.let(::setOf) ?: emptySet()
}

```

The `wrapResources` method in the pipeline finalizes the output naming, producing either a single `<document-name>.pdf` file or an `OutputResourceGroup` when sub-documents exist.

## End-to-End CLI Example

```bash

# Compile to HTML only

quarkdown compile example.qd --output out

# Compile and produce PDF (requires Chrome/Puppeteer)

quarkdown compile example.qd --pdf --output out

```

Internally, the first command runs the full stage chain ending at `ResourceGenerationStage` with HTML output. The second command sets `options.exportPdf = true`, triggering the `PdfHtmlPostRendererDecorator` to intercept resources and return a `BinaryOutputArtifact` containing the PDF data.

## Programmatic Pipeline Usage

You can trigger the same compilation flow directly in Kotlin without the CLI:

```kotlin
val context = MutableContext(flavor = MyFlavor())
val libraries = setOf(Stdlib.Library)
val rendererFactory = MyFlavor().rendererFactory

val pipeline = Pipeline(
    context = context,
    options = PipelineOptions(exportPdf = true, wrapOutput = true),
    libraries = libraries,
    renderer = { factory, ctx -> RenderingComponents.fromFlavor(factory, ctx) }
)

val resources: OutputResource? = pipeline.execute(File("doc.qd").readText())

```

This follows the identical flow as `CompileCommand`: the `Pipeline` executes `PipelineChainFactory.fullChain`, processes all stages, and returns the output resources (HTML or PDF) based on the `PipelineOptions` configuration.

## Summary

- **Entry Point**: [`CompileCommand.kt`](https://github.com/iamgio/quarkdown/blob/main/CompileCommand.kt) initializes the pipeline and delegates to `FileExecutionStrategy`.
- **Stage Chain**: `PipelineChainFactory.fullChain` orchestrates 10 sequential stages from lexing to resource generation.
- **AST Processing**: The pipeline tokenizes source into tokens, parses into an AST, expands function calls via `FunctionCallExpansionStage`, and traverses the tree for flavor-specific hooks.
- **HTML Output**: [`HtmlPostRenderer.kt`](https://github.com/iamgio/quarkdown/blob/main/HtmlPostRenderer.kt) generates the final document template and assets.
- **PDF Output**: [`PdfHtmlPostRendererDecorator.kt`](https://github.com/iamgio/quarkdown/blob/main/PdfHtmlPostRendererDecorator.kt) intercepts HTML resources, creates a temporary runtime environment, and uses Puppeteer to export PDF files.
- **Extensibility**: Each stage is a pure function implementing `PipelineStage<I, O>`, allowing custom stages or renderers to be injected without modifying core logic.

## Frequently Asked Questions

### What is the difference between LexingStage and ParsingStage?

**LexingStage** converts the raw `.qd` character sequence into a stream of tokens using the flavor's lexer rules, while **ParsingStage** consumes that token stream to build an Abstract Syntax Tree (AST) rooted at `AstRoot`. The lexer handles lexical analysis (identifying keywords, symbols, and literals), and the parser handles syntactic analysis (building the hierarchical structure of the document).

### How does Quarkdown expand function calls during compilation?

The **FunctionCallExpansionStage** traverses the AST and identifies `FunctionCallNode` instances (representing `.functionName {…}` syntax). It uses `FunctionCallNodeExpander` to resolve these calls against registered libraries from `LibrariesRegistrationStage`, replacing the call nodes with their expanded results before rendering occurs.

### Can I skip the HTML wrapping stage when compiling?

Yes. The `thenOptionally` operator in `PipelineChainFactory.fullChain` skips **PostRenderingStage** when `PipelineOptions.wrapOutput` is set to `false`. This produces raw rendered content without the HTML `<head>`, CSS, or script wrappers, useful when piping output to other tools or generating fragments rather than complete documents.

### What dependencies are required for PDF generation?

PDF generation requires a Chrome or Chromium installation accessible to Puppeteer. When using the `--pdf` flag, the `PdfHtmlPostRendererDecorator` launches a headless browser via `HtmlPdfExporter` to render the temporary HTML files. If Chrome is not installed in the system path, the PDF export stage will fail with a runtime exception.