How Quarkdown's Pipeline Handles Function Call Expansion and AST Transformation

Quarkdown resolves function calls through a dedicated pipeline stage that converts raw syntax into concrete AST nodes by resolving, executing, and mapping function outputs before rendering.

The iamgio/quarkdown project processes documents through a multi-stage pipeline where function call expansion and AST transformation serve as the bridge between parsing and rendering. This architecture ensures that dynamic content generated by functions becomes static AST nodes that downstream stages can process without awareness of the original function syntax.

The Pipeline Stage Architecture

The expansion process centers on FunctionCallExpansionStage, located in quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/FunctionCallExpansionStage.kt. This stage implements the peek method, which instantiates a FunctionCallNodeExpander and triggers expandAll() to process all queued function calls.

In quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/Pipeline.kt, the stage chain is constructed to position function call expansion immediately after parsing. The stage hooks into PipelineHooks.afterExpanding, ensuring it executes automatically once the ParsingStage produces an AstRoot containing FunctionCallNode objects.

The Expansion Process

Resolving and Executing Function Calls

The core driver resides in quarkdown-core/src/main/kotlin/com/quarkdown/core/function/call/FunctionCallNodeExpander.kt. For each FunctionCallNode, the expander performs four critical steps:

1. Resolution: Calls node.context.resolveUnchecked(node) to locate the function definition in the node's context.
2. Execution: Invokes call.execute() to run the function logic.
3. Mapping: Uses OutputValueVisitorFactory to convert the returned OutputValue into concrete AST nodes.
4. Attachment: Stores the resulting node as a child of the original FunctionCallNode.

Converting Output Values to AST Nodes

The OutputValueVisitorFactory interface, implemented by NodeOutputValueVisitorFactory in quarkdown-core/src/main/kotlin/com/quarkdown/core/function/value/output/node/NodeOutputValueVisitorFactory.kt, creates specialized visitors for different node types:

• BlockNodeOutputValueVisitor (BlockNodeOutputValueVisitor.kt): Handles block-level values (String, Number, Boolean, Object, None). It delegates to the inline visitor and wraps results in a Paragraph block when necessary.
• InlineNodeOutputValueVisitor (InlineNodeOutputValueVisitor.kt): Creates inline elements like Text, CheckBox, and CodeSpan nodes.

Recursive Markdown Processing

When functions return raw markdown strings, the system must parse them into AST sub-trees. The ValueFactory class in quarkdown-core/src/main/kotlin/com/quarkdown/core/function/value/factory/ValueFactory.kt provides two critical methods:

• blockMarkdown: Parses block-level markdown, expanding any nested function calls encountered.
• inlineMarkdown: Parses inline markdown with recursive expansion.

This recursive capability enables functions to return dynamic content that itself contains additional function calls, which the pipeline expands in subsequent iterations.

Queue Management and Safe Expansion

During the ParsingStage, each encountered .functionCall node is queued in the mutable context via MutableContext.dequeueAllFunctionCalls. When FunctionCallExpansionStage runs, it calls context.dequeueAllFunctionCalls() to obtain a snapshot of the queue.

This snapshot pattern prevents ConcurrentModificationException by allowing the expander to safely add new function calls to the queue while iterating over the snapshot. After processing, any newly enqueued calls remain for subsequent expandAll() invocations, guaranteeing correct expansion order for nested dependencies.

Error Handling and Diagnostics

The expander wraps all resolution and execution exceptions as PipelineException instances via e.asPipelineException(). These errors are stored in node.error along with the pipeline's errorHandler, preserving diagnostic information for the rendering stage.

When expansion fails, the original FunctionCallNode retains the error state rather than its expected child content, allowing renderers to display meaningful error messages or fallback content.

Practical Pipeline Flow

The following Kotlin code illustrates how the pipeline coordinates expansion:

import com.quarkdown.core.pipeline.Pipeline
import com.quarkdown.core.pipeline.stages.FunctionCallExpansionStage
import com.quarkdown.core.function.call.FunctionCallNodeExpander

// Pipeline construction
val pipeline = Pipeline(
    // ... other stages
    FunctionCallExpansionStage(),
    // ... rendering stages
)

// The expansion stage internally executes:
val expander = FunctionCallNodeExpander(rootContext)
expander.expandAll() // Processes all queued FunctionCallNodes

During execution, the pipeline follows this sequence:

1. Lexing: Raw text converts to token streams.
2. Parsing: Tokens become an AstRoot with FunctionCallNode objects queued in context.
3. Expansion: FunctionCallNodeExpander resolves, executes, and maps each call to AST nodes.
4. Rendering: The fully expanded AST transforms into target formats (HTML, PDF, plain text).

Summary

• FunctionCallExpansionStage sits between parsing and rendering in Pipeline.kt, triggered via PipelineHooks.afterExpanding.
• FunctionCallNodeExpander resolves functions via node.context.resolveUnchecked() and maps outputs using OutputValueVisitorFactory visitors.
• ValueFactory methods blockMarkdown and inlineMarkdown enable recursive expansion of markdown returned by functions.
• The queue snapshot mechanism in MutableContext.dequeueAllFunctionCalls prevents concurrent modification during expansion.
• Errors are captured as PipelineException instances in node.error for diagnostic rendering.

Frequently Asked Questions

How does Quarkdown prevent concurrent modification during function call expansion?

The FunctionCallNodeExpander calls context.dequeueAllFunctionCalls() to create a snapshot of the pending function queue before processing begins. This allows the expander to safely enqueue new function calls returned by executing functions while iterating over the snapshot, eliminating ConcurrentModificationException risks.

Can functions return markdown that contains additional function calls?

Yes. When functions return raw markdown strings, the expander uses ValueFactory.blockMarkdown or ValueFactory.inlineMarkdown to parse the content. These methods recursively process the markdown, queuing any new function calls encountered for expansion in subsequent iterations until the AST contains no unexpanded function nodes.

What happens when a function execution fails during the expansion stage?

The expander catches exceptions during resolution or execution, wraps them as PipelineException objects via e.asPipelineException(), and stores them in the FunctionCallNode.error property. The error handler from the pipeline context is preserved alongside the exception, allowing renderers to display diagnostic information rather than crashing the entire pipeline.

Where does function call expansion occur in the overall pipeline lifecycle?

Expansion occurs in the third major phase, positioned between lexing/parsing and rendering. According to the source in Pipeline.kt, raw text flows through: Lexing → ParsingStage (which queues calls) → FunctionCallExpansionStage (which resolves and expands calls) → Rendering stages. This positioning ensures all downstream stages work with a concrete, function-free AST.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}