How Quarkdown Compiles .qd Files to HTML and PDF: A Deep Dive into the Pipeline

Quarkdown transforms .qd source files into HTML or PDF through a 10-stage pipeline that lexes and parses the source into an AST, expands function calls, renders the tree to HTML, and optionally delegates to Puppeteer for PDF generation.

The Quarkdown compilation flow converts lightweight markup documents into production-ready HTML or PDF artifacts through a modular, extensible pipeline. Whether invoked via the CLI or embedded as a library, the process follows a strict sequence of transformation stages defined in quarkdown-core and quarkdown-html. Each stage is a pure, composable component that passes data to the next via the PipelineChainFactory.

Stage 1: CLI Entry and Pipeline Initialization

Compilation begins in CompileCommand.kt, which builds a Pipeline instance based on user flags like --pdf or --pipe.

// quarkdown-cli/src/main/kotlin/com/quarkdown/cli/exec/CompileCommand.kt
class CompileCommand : ExecuteCommand("compile") {
    …
    override fun createExecutionStrategy(cliOptions: CliOptions) = FileExecutionStrategy(source)
}

FileExecutionStrategy constructs a Pipeline object from quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/Pipeline.kt and calls pipeline.execute(source). The Pipeline class stores the mutable context, registered libraries, and rendering components before triggering the stage chain.

Stage 2: Building the Compilation Stage Chain

The Pipeline.execute method delegates to executeUnwrapped, which builds the complete stage chain via PipelineChainFactory.fullChain in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/PipelineChainFactory.kt.

object PipelineChainFactory {
    fun fullChain(
        source: CharSequence,
        renderingComponents: RenderingComponents,
        options: PipelineOptions,
    ): PipelineStage<Unit, Set<OutputResource>> =
        AttachmentStage then
        LibrariesRegistrationStage then
        LexingStage(source) then
        ParsingStage then
        AttributesUpdateStage(renderingComponents.postRenderer.preferredMediaStorageOptions) then
        FunctionCallExpansionStage then
        TreeTraversalStage then
        RenderingStage(renderingComponents.nodeRenderer) thenOptionally
        PostRenderingStage(renderingComponents.postRenderer).takeIf { options.wrapOutput } then
        AfterAllRenderingPeek then
        ResourceGenerationStage(renderingComponents.postRenderer)
}

The thenOptionally operator skips PostRenderingStage when wrapOutput is disabled, allowing raw HTML generation without document templates.

Stage 3: Core Processing Stages

Each stage implements PipelineStage<I, O> and performs a single responsibility:

• AttachmentStage – Attaches the Pipeline instance to the document's Context, enabling sub-document support. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/AttachmentStage.kt.

• LibrariesRegistrationStage – Registers standard-library Library sets into the context so that .function calls can be resolved later. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/LibrariesRegistrationStage.kt.

• LexingStage – Tokenizes the raw .qd source text using the flavor's lexer. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/LexingStage.kt.

• ParsingStage – Transforms the token stream into an AST rooted at AstRoot. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/ParsingStage.kt.

• AttributesUpdateStage – Stores the root AST in the context, registers sub-documents in the graph, and merges media-storage options required by the renderer (HTML requires local media storage). Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/AttributesUpdateStage.kt.

• FunctionCallExpansionStage – Walks the AST and expands every Quarkdown function call (e.g., .foo {…}) via FunctionCallNodeExpander. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/FunctionCallExpansionStage.kt.

• TreeTraversalStage – Executes the flavor's tree iterator to visit every node, allowing hooks to modify the tree before rendering. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/TreeTraversalStage.kt.

Stage 4: Rendering and Resource Generation

After AST processing, the pipeline enters the output generation phase:

• RenderingStage – Calls the flavor-specific NodeRenderer (HTML, plain text, etc.) to produce the main content string. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/RenderingStage.kt.

• PostRenderingStage – Wraps the rendered content into a final document template including HTML <head>, CSS, and scripts. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/PostRenderingStage.kt.

• ResourceGenerationStage – Uses the PostRenderer to convert the final string into concrete OutputResource objects (HTML files, media assets, sub-document files). It recursively processes sub-documents. Located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/ResourceGenerationStage.kt.

The HTML renderer is instantiated via HtmlPostRenderer.kt in the quarkdown-html module, producing main.html and associated assets like CSS and fonts.

Stage 5: PDF Generation via Puppeteer

When the --pdf flag is set, the compilation flow extends the pipeline with a decorator pattern. The HtmlRendererExtension wraps the standard HTML post-renderer:

// quarkdown-html/src/main/kotlin/com/quarkdown/rendering/html/extension/HtmlRendererExtension.kt
if (options.exportPdf) {
    postRenderer = PdfHtmlPostRendererDecorator(
        postRenderer = postRenderer,
        options = pdfOptions,
    )
}

The PdfHtmlPostRendererDecorator (located in quarkdown-html/src/main/kotlin/com/quarkdown/rendering/html/pdf/PdfHtmlPostRendererDecorator.kt) performs three operations:

1. Generates standard HTML resources via the wrapped post-renderer.
2. Copies resources to a temporary directory.
3. Invokes HtmlPdfExporter (which uses Puppeteer) to render the temporary HTML into a single PDF file.

override fun generateResources(rendered: CharSequence): Set<OutputResource> {
    val resources = postRenderer.generateResources(rendered)
    // …create temp dir, copy resources, export PDF…
    return out?.let(BinaryOutputArtifact::fromFile)
               ?.also { tempDirectory.deleteRecursively() }
               ?.let(::setOf) ?: emptySet()
}

The wrapResources method in the pipeline finalizes the output naming, producing either a single <document-name>.pdf file or an OutputResourceGroup when sub-documents exist.

End-to-End CLI Example

# Compile to HTML only

quarkdown compile example.qd --output out

# Compile and produce PDF (requires Chrome/Puppeteer)

quarkdown compile example.qd --pdf --output out

Internally, the first command runs the full stage chain ending at ResourceGenerationStage with HTML output. The second command sets options.exportPdf = true, triggering the PdfHtmlPostRendererDecorator to intercept resources and return a BinaryOutputArtifact containing the PDF data.

Programmatic Pipeline Usage

You can trigger the same compilation flow directly in Kotlin without the CLI:

val context = MutableContext(flavor = MyFlavor())
val libraries = setOf(Stdlib.Library)
val rendererFactory = MyFlavor().rendererFactory

val pipeline = Pipeline(
    context = context,
    options = PipelineOptions(exportPdf = true, wrapOutput = true),
    libraries = libraries,
    renderer = { factory, ctx -> RenderingComponents.fromFlavor(factory, ctx) }
)

val resources: OutputResource? = pipeline.execute(File("doc.qd").readText())

This follows the identical flow as CompileCommand: the Pipeline executes PipelineChainFactory.fullChain, processes all stages, and returns the output resources (HTML or PDF) based on the PipelineOptions configuration.

Summary

• Entry Point: CompileCommand.kt initializes the pipeline and delegates to FileExecutionStrategy.
• Stage Chain: PipelineChainFactory.fullChain orchestrates 10 sequential stages from lexing to resource generation.
• AST Processing: The pipeline tokenizes source into tokens, parses into an AST, expands function calls via FunctionCallExpansionStage, and traverses the tree for flavor-specific hooks.
• HTML Output: HtmlPostRenderer.kt generates the final document template and assets.
• PDF Output: PdfHtmlPostRendererDecorator.kt intercepts HTML resources, creates a temporary runtime environment, and uses Puppeteer to export PDF files.
• Extensibility: Each stage is a pure function implementing PipelineStage<I, O>, allowing custom stages or renderers to be injected without modifying core logic.

Frequently Asked Questions

What is the difference between LexingStage and ParsingStage?

LexingStage converts the raw .qd character sequence into a stream of tokens using the flavor's lexer rules, while ParsingStage consumes that token stream to build an Abstract Syntax Tree (AST) rooted at AstRoot. The lexer handles lexical analysis (identifying keywords, symbols, and literals), and the parser handles syntactic analysis (building the hierarchical structure of the document).

How does Quarkdown expand function calls during compilation?

The FunctionCallExpansionStage traverses the AST and identifies FunctionCallNode instances (representing .functionName {…} syntax). It uses FunctionCallNodeExpander to resolve these calls against registered libraries from LibrariesRegistrationStage, replacing the call nodes with their expanded results before rendering occurs.

Can I skip the HTML wrapping stage when compiling?

Yes. The thenOptionally operator in PipelineChainFactory.fullChain skips PostRenderingStage when PipelineOptions.wrapOutput is set to false. This produces raw rendered content without the HTML <head>, CSS, or script wrappers, useful when piping output to other tools or generating fragments rather than complete documents.

What dependencies are required for PDF generation?

PDF generation requires a Chrome or Chromium installation accessible to Puppeteer. When using the --pdf flag, the PdfHtmlPostRendererDecorator launches a headless browser via HtmlPdfExporter to render the temporary HTML files. If Chrome is not installed in the system path, the PDF export stage will fail with a runtime exception.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}