# How Lemon AI's Markdown Utility Parses AI-Generated Markdown Content

> Discover how Lemon AI's markdown utility tokenizes LLM output with the marked lexer, parsing AI-generated markdown into structured task arrays and mapping headings to content blocks.

- Repository: [hexdocom/lemonai](https://github.com/hexdocom/lemonai)
- Tags: how-to-guide
- Published: 2026-03-03

---

**Lemon AI's markdown utility converts raw LLM output into structured task arrays by tokenizing Markdown with the `marked` lexer and mapping headings to content blocks.**

The `hexdocom/lemonai` repository implements a lightweight processing pipeline to transform unstructured AI responses into actionable task lists. At the core of this system lies a specialized **markdown utility** that parses generated Markdown without rendering HTML, instead extracting semantic structure for downstream planning agents.

## Tokenizing LLM Output with the marked Lexer

The utility relies on the **`marked`** library's lexer to break text into a flat array of tokens without generating HTML. This approach provides deterministic, fast document traversal.

In [`src/utils/markdown.js`](https://github.com/hexdocom/lemonai/blob/main/src/utils/markdown.js), the implementation uses:

```javascript
const marked = require('marked');          // ← third-party lexer
const tokens = marked.lexer(markdown);      // tokenises the whole Markdown string

```

The lexer produces tokens such as **heading**, **paragraph**, and **list** types. By using `marked.lexer()` instead of the full parser, the utility avoids HTML generation overhead while maintaining precise structural awareness of the document.

## Extracting Title-Content Mappings from Tokens

After tokenization, the utility walks the array sequentially to build structured task objects. The algorithm distinguishes between heading tokens and content tokens using strict type checking.

### Processing Heading Tokens

When encountering a token where `token.type === 'heading'`, the utility executes two operations:

1. If a previous `title` exists, the current accumulated `content` buffer is stored as an object `{title, content}`
2. The new heading's text becomes the next `title` and the `content` buffer resets to empty

### Accumulating Content Blocks

For any non-heading token, the utility appends the raw Markdown source to the current content buffer:

```javascript
// Logic for non-heading tokens
contentBuffer += token.raw;

```

When the loop completes, the final pending title-content pair pushes into the results list, ensuring no trailing content drops from the output.

## Generating Human-Readable Task Descriptions

After constructing the title-content map, the utility enriches each entry with a concatenated description field. This transformation occurs in the mapping phase:

```javascript
item.description = item.title + '\n' + item.content;

```

The `description` field combines the heading and its associated Markdown block into a single string. Downstream components consume this field directly for display purposes, eliminating the need for additional formatting logic in the presentation layer.

## Integration with the Planning Agent

The markdown utility exposes the `resolveMarkdown` function for consumption by the planning system. In [`src/agent/planning/index.js`](https://github.com/hexdocom/lemonai/blob/main/src/agent/planning/index.js), the planning agent invokes the utility after receiving LLM output:

```javascript
const { resolveMarkdown } = require("@src/utils/markdown");

// … inside processResult …
const tasks = await resolveMarkdown(markdown);

```

The function returns a clean array of task objects structured as:

```javascript
[
  { title: 'Step 1', content: '…raw markdown…', description: 'Step 1\n…raw markdown…' },
  { title: 'Step 2', content: '…raw markdown…', description: 'Step 2\n…raw markdown…' }
]

```

This structured output enables the planning agent to iterate over discrete tasks, track progress, and execute sequential operations based on the LLM's generated plan.

## Summary

- **Tokenization strategy**: Uses `marked.lexer()` for fast, deterministic Markdown parsing without HTML generation
- **Structural extraction**: Implements sequential token walking to map headings to content blocks using `token.type` checks
- **Data enrichment**: Concatenates titles and content into human-readable `description` fields for direct UI consumption
- **System integration**: Exports `resolveMarkdown` from [`src/utils/markdown.js`](https://github.com/hexdocom/lemonai/blob/main/src/utils/markdown.js) for use by the planning agent in [`src/agent/planning/index.js`](https://github.com/hexdocom/lemonai/blob/main/src/agent/planning/index.js)
- **Output format**: Returns standardized arrays of objects containing `title`, `content`, and `description` properties

## Frequently Asked Questions

### Why does Lemon AI use marked's lexer instead of the full parser?

The **marked** lexer provides a flat token array without HTML generation overhead. According to the `hexdocom/lemonai` source code, this approach offers deterministic document traversal and faster processing when the system only needs structural extraction rather than rendered output.

### What happens to Markdown content that appears before the first heading?

The utility initializes an empty content buffer before processing begins. Any tokens encountered before the first heading token accumulate in this buffer but only persist if a subsequent heading triggers the storage logic. Content preceding the first heading typically remains unassociated unless the document structure specifically accounts for it.

### How does the planning agent handle the structured task array?

The planning agent in [`src/agent/planning/index.js`](https://github.com/hexdocom/lemonai/blob/main/src/agent/planning/index.js) awaits the `resolveMarkdown` promise and receives the standardized task array. Each task object's `description` field provides immediately displayable content, while the separate `title` and `content` fields enable granular control over task execution and status tracking.

### Can the markdown utility handle nested Markdown structures?

The utility processes all non-heading tokens by appending `token.raw` to the content buffer, which preserves nested Markdown syntax within content blocks. However, the structural extraction logic operates at the heading level, treating nested lists or code blocks within sections as part of the parent heading's content rather than separate task entities.