# How Quarkdown's Pipeline Handles Function Call Expansion and AST Transformation

> Discover how Quarkdown's pipeline expands function calls converting syntax to AST nodes by resolving executing and mapping outputs for efficient rendering.

- Repository: [Giorgio Garofalo/quarkdown](https://github.com/iamgio/quarkdown)
- Tags: internals
- Published: 2026-04-29

---

**Quarkdown resolves function calls through a dedicated pipeline stage that converts raw syntax into concrete AST nodes by resolving, executing, and mapping function outputs before rendering.**

The iamgio/quarkdown project processes documents through a multi-stage pipeline where **function call expansion and AST transformation** serve as the bridge between parsing and rendering. This architecture ensures that dynamic content generated by functions becomes static AST nodes that downstream stages can process without awareness of the original function syntax.

## The Pipeline Stage Architecture

The expansion process centers on `FunctionCallExpansionStage`, located in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/FunctionCallExpansionStage.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/stages/FunctionCallExpansionStage.kt). This stage implements the `peek` method, which instantiates a `FunctionCallNodeExpander` and triggers `expandAll()` to process all queued function calls.

In [`quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/Pipeline.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/pipeline/Pipeline.kt), the stage chain is constructed to position function call expansion immediately after parsing. The stage hooks into `PipelineHooks.afterExpanding`, ensuring it executes automatically once the `ParsingStage` produces an `AstRoot` containing `FunctionCallNode` objects.

## The Expansion Process

### Resolving and Executing Function Calls

The core driver resides in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/function/call/FunctionCallNodeExpander.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/function/call/FunctionCallNodeExpander.kt). For each `FunctionCallNode`, the expander performs four critical steps:

1. **Resolution**: Calls `node.context.resolveUnchecked(node)` to locate the function definition in the node's context.
2. **Execution**: Invokes `call.execute()` to run the function logic.
3. **Mapping**: Uses `OutputValueVisitorFactory` to convert the returned `OutputValue` into concrete AST nodes.
4. **Attachment**: Stores the resulting node as a child of the original `FunctionCallNode`.

### Converting Output Values to AST Nodes

The `OutputValueVisitorFactory` interface, implemented by `NodeOutputValueVisitorFactory` in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/function/value/output/node/NodeOutputValueVisitorFactory.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/function/value/output/node/NodeOutputValueVisitorFactory.kt), creates specialized visitors for different node types:

- **BlockNodeOutputValueVisitor** ([`BlockNodeOutputValueVisitor.kt`](https://github.com/iamgio/quarkdown/blob/main/BlockNodeOutputValueVisitor.kt)): Handles block-level values (String, Number, Boolean, Object, None). It delegates to the inline visitor and wraps results in a `Paragraph` block when necessary.
- **InlineNodeOutputValueVisitor** ([`InlineNodeOutputValueVisitor.kt`](https://github.com/iamgio/quarkdown/blob/main/InlineNodeOutputValueVisitor.kt)): Creates inline elements like `Text`, `CheckBox`, and `CodeSpan` nodes.

### Recursive Markdown Processing

When functions return raw markdown strings, the system must parse them into AST sub-trees. The `ValueFactory` class in [`quarkdown-core/src/main/kotlin/com/quarkdown/core/function/value/factory/ValueFactory.kt`](https://github.com/iamgio/quarkdown/blob/main/quarkdown-core/src/main/kotlin/com/quarkdown/core/function/value/factory/ValueFactory.kt) provides two critical methods:

- `blockMarkdown`: Parses block-level markdown, expanding any nested function calls encountered.
- `inlineMarkdown`: Parses inline markdown with recursive expansion.

This recursive capability enables functions to return dynamic content that itself contains additional function calls, which the pipeline expands in subsequent iterations.

## Queue Management and Safe Expansion

During the `ParsingStage`, each encountered `.functionCall` node is queued in the **mutable context** via `MutableContext.dequeueAllFunctionCalls`. When `FunctionCallExpansionStage` runs, it calls `context.dequeueAllFunctionCalls()` to obtain a **snapshot** of the queue.

This snapshot pattern prevents `ConcurrentModificationException` by allowing the expander to safely add new function calls to the queue while iterating over the snapshot. After processing, any newly enqueued calls remain for subsequent `expandAll()` invocations, guaranteeing correct expansion order for nested dependencies.

## Error Handling and Diagnostics

The expander wraps all resolution and execution exceptions as `PipelineException` instances via `e.asPipelineException()`. These errors are stored in `node.error` along with the pipeline's `errorHandler`, preserving diagnostic information for the rendering stage.

When expansion fails, the original `FunctionCallNode` retains the error state rather than its expected child content, allowing renderers to display meaningful error messages or fallback content.

## Practical Pipeline Flow

The following Kotlin code illustrates how the pipeline coordinates expansion:

```kotlin
import com.quarkdown.core.pipeline.Pipeline
import com.quarkdown.core.pipeline.stages.FunctionCallExpansionStage
import com.quarkdown.core.function.call.FunctionCallNodeExpander

// Pipeline construction
val pipeline = Pipeline(
    // ... other stages
    FunctionCallExpansionStage(),
    // ... rendering stages
)

// The expansion stage internally executes:
val expander = FunctionCallNodeExpander(rootContext)
expander.expandAll() // Processes all queued FunctionCallNodes

```

During execution, the pipeline follows this sequence:

1. **Lexing**: Raw text converts to token streams.
2. **Parsing**: Tokens become an `AstRoot` with `FunctionCallNode` objects queued in context.
3. **Expansion**: `FunctionCallNodeExpander` resolves, executes, and maps each call to AST nodes.
4. **Rendering**: The fully expanded AST transforms into target formats (HTML, PDF, plain text).

## Summary

- **FunctionCallExpansionStage** sits between parsing and rendering in [`Pipeline.kt`](https://github.com/iamgio/quarkdown/blob/main/Pipeline.kt), triggered via `PipelineHooks.afterExpanding`.
- **FunctionCallNodeExpander** resolves functions via `node.context.resolveUnchecked()` and maps outputs using `OutputValueVisitorFactory` visitors.
- **ValueFactory** methods `blockMarkdown` and `inlineMarkdown` enable recursive expansion of markdown returned by functions.
- The **queue snapshot** mechanism in `MutableContext.dequeueAllFunctionCalls` prevents concurrent modification during expansion.
- Errors are captured as **PipelineException** instances in `node.error` for diagnostic rendering.

## Frequently Asked Questions

### How does Quarkdown prevent concurrent modification during function call expansion?

The `FunctionCallNodeExpander` calls `context.dequeueAllFunctionCalls()` to create a snapshot of the pending function queue before processing begins. This allows the expander to safely enqueue new function calls returned by executing functions while iterating over the snapshot, eliminating `ConcurrentModificationException` risks.

### Can functions return markdown that contains additional function calls?

Yes. When functions return raw markdown strings, the expander uses `ValueFactory.blockMarkdown` or `ValueFactory.inlineMarkdown` to parse the content. These methods recursively process the markdown, queuing any new function calls encountered for expansion in subsequent iterations until the AST contains no unexpanded function nodes.

### What happens when a function execution fails during the expansion stage?

The expander catches exceptions during resolution or execution, wraps them as `PipelineException` objects via `e.asPipelineException()`, and stores them in the `FunctionCallNode.error` property. The error handler from the pipeline context is preserved alongside the exception, allowing renderers to display diagnostic information rather than crashing the entire pipeline.

### Where does function call expansion occur in the overall pipeline lifecycle?

Expansion occurs in the **third major phase**, positioned between lexing/parsing and rendering. According to the source in [`Pipeline.kt`](https://github.com/iamgio/quarkdown/blob/main/Pipeline.kt), raw text flows through: Lexing → `ParsingStage` (which queues calls) → `FunctionCallExpansionStage` (which resolves and expands calls) → Rendering stages. This positioning ensures all downstream stages work with a concrete, function-free AST.