How Lemon AI's Markdown Utility Parses AI-Generated Markdown Content

Lemon AI's markdown utility converts raw LLM output into structured task arrays by tokenizing Markdown with the marked lexer and mapping headings to content blocks.

The hexdocom/lemonai repository implements a lightweight processing pipeline to transform unstructured AI responses into actionable task lists. At the core of this system lies a specialized markdown utility that parses generated Markdown without rendering HTML, instead extracting semantic structure for downstream planning agents.

Tokenizing LLM Output with the marked Lexer

The utility relies on the marked library's lexer to break text into a flat array of tokens without generating HTML. This approach provides deterministic, fast document traversal.

In src/utils/markdown.js, the implementation uses:

const marked = require('marked');          // ← third-party lexer
const tokens = marked.lexer(markdown);      // tokenises the whole Markdown string

The lexer produces tokens such as heading, paragraph, and list types. By using marked.lexer() instead of the full parser, the utility avoids HTML generation overhead while maintaining precise structural awareness of the document.

Extracting Title-Content Mappings from Tokens

After tokenization, the utility walks the array sequentially to build structured task objects. The algorithm distinguishes between heading tokens and content tokens using strict type checking.

Processing Heading Tokens

When encountering a token where token.type === 'heading', the utility executes two operations:

1. If a previous title exists, the current accumulated content buffer is stored as an object {title, content}
2. The new heading's text becomes the next title and the content buffer resets to empty

Accumulating Content Blocks

For any non-heading token, the utility appends the raw Markdown source to the current content buffer:

// Logic for non-heading tokens
contentBuffer += token.raw;

When the loop completes, the final pending title-content pair pushes into the results list, ensuring no trailing content drops from the output.

Generating Human-Readable Task Descriptions

After constructing the title-content map, the utility enriches each entry with a concatenated description field. This transformation occurs in the mapping phase:

item.description = item.title + '\n' + item.content;

The description field combines the heading and its associated Markdown block into a single string. Downstream components consume this field directly for display purposes, eliminating the need for additional formatting logic in the presentation layer.

Integration with the Planning Agent

The markdown utility exposes the resolveMarkdown function for consumption by the planning system. In src/agent/planning/index.js, the planning agent invokes the utility after receiving LLM output:

const { resolveMarkdown } = require("@src/utils/markdown");

// … inside processResult …
const tasks = await resolveMarkdown(markdown);

The function returns a clean array of task objects structured as:

[
  { title: 'Step 1', content: '…raw markdown…', description: 'Step 1\n…raw markdown…' },
  { title: 'Step 2', content: '…raw markdown…', description: 'Step 2\n…raw markdown…' }
]

This structured output enables the planning agent to iterate over discrete tasks, track progress, and execute sequential operations based on the LLM's generated plan.

Summary

• Tokenization strategy: Uses marked.lexer() for fast, deterministic Markdown parsing without HTML generation
• Structural extraction: Implements sequential token walking to map headings to content blocks using token.type checks
• Data enrichment: Concatenates titles and content into human-readable description fields for direct UI consumption
• System integration: Exports resolveMarkdown from src/utils/markdown.js for use by the planning agent in src/agent/planning/index.js
• Output format: Returns standardized arrays of objects containing title, content, and description properties

Frequently Asked Questions

Why does Lemon AI use marked's lexer instead of the full parser?

The marked lexer provides a flat token array without HTML generation overhead. According to the hexdocom/lemonai source code, this approach offers deterministic document traversal and faster processing when the system only needs structural extraction rather than rendered output.

What happens to Markdown content that appears before the first heading?

The utility initializes an empty content buffer before processing begins. Any tokens encountered before the first heading token accumulate in this buffer but only persist if a subsequent heading triggers the storage logic. Content preceding the first heading typically remains unassociated unless the document structure specifically accounts for it.

How does the planning agent handle the structured task array?

The planning agent in src/agent/planning/index.js awaits the resolveMarkdown promise and receives the standardized task array. Each task object's description field provides immediately displayable content, while the separate title and content fields enable granular control over task execution and status tracking.

Can the markdown utility handle nested Markdown structures?

The utility processes all non-heading tokens by appending token.raw to the content buffer, which preserves nested Markdown syntax within content blocks. However, the structural extraction logic operates at the heading level, treating nested lists or code blocks within sections as part of the parent heading's content rather than separate task entities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}