ReportEngine Architecture and Report Generation Pipeline: How the Bettafish Codebase Converts Agent Outputs into Structured Reports

The ReportEngine in the 666ghj/bettafish repository implements a modular, four-layer pipeline that transforms raw outputs from Query, Media, and Insight agents into polished HTML, Markdown, or PDF documents through orchestrated LLM-driven processing nodes and intermediate representation stitching.

The ReportEngine serves as the central reporting service within the Bettafish project, designed to synthesize heterogeneous data sources into publication-ready documents. Its architecture separates concerns between orchestration, processing, utilities, and rendering to maintain stateless, retry-safe operation while supporting real-time streaming feedback.

Four-Layer Architecture Overview

The ReportEngine architecture divides functionality into four logical layers that process data sequentially from raw inputs to final output files.

Orchestration Layer

The ReportAgent class in ReportEngine/agent.py acts as the central coordinator, maintaining workflow state and driving each processing stage. It manages the end-to-end lifecycle from input normalization through final persistence.

Processing Nodes

Individual LLM-driven steps handle specific transformation tasks:

• TemplateSelectionNode → Selects appropriate Markdown skeletons
• DocumentLayoutNode → Designs document-wide structure and themes
• WordBudgetNode → Calculates per-chapter word targets
• ChapterGenerationNode → Synthesizes content via LLM calls
• GraphRAGQueryNode → Enhances chapters with knowledge graph context

Core Utilities

Helper modules in ReportEngine/core/ and ReportEngine/utils/ manage template parsing, intermediate representation (IR) stitching, chapter storage, configuration loading, and JSON validation.

Renderers and State

The rendering layer in ReportEngine/renderers/ converts IR into target formats (HTML, Markdown, PDF), while ReportEngine/state/state.py and ReportEngine/flask_interface.py provide runtime state management and HTTP API access.

Orchestration Layer: The ReportAgent Workflow

The ReportAgent class implements the primary generate_report() method, which executes an 11-step pipeline to convert agent outputs into structured documents.

Input Normalization and Template Selection

First, _normalize_reports() converts the three agent outputs (Query, Media, Insight) plus optional forum logs into plain strings. Then _select_template() invokes TemplateSelectionNode to analyze the query and content, selecting an appropriate Markdown template from the local repository.

Template Parsing and Layout Design

The _slice_template() method calls parse_template_sections from ReportEngine/core/template_parser.py to decompose the template into TemplateSection objects, each assigned a stable chapter_id. Subsequently, DocumentLayoutNode generates the document-wide design including titles, hero images, table of contents entries, and theme tokens.

Budget Planning and Context Building

WordBudgetNode computes per-chapter word targets and global writing guidelines. The agent then aggregates all inputs via _build_generation_context(), creating a single dictionary containing the query, normalized reports, layout specifications, and budget constraints for downstream consumption.

Optional GraphRAG Enhancement

When GRAPHRAG_ENABLED is true, _build_knowledge_graph() constructs a graph from the engine states and forum logs. During chapter generation, GraphRAGQueryNode executes multi-turn queries against this graph to enrich content with contextual relationships.

Chapter Generation and IR Stitching

For each TemplateSection, the agent invokes ChapterGenerationNode.run(), which may retry on JSON parsing errors or content-filter failures using fallback LLM chains (json_rescue_clients). Streaming callbacks (stream_handler) emit real-time progress events to UI clients.

After all chapters complete, DocumentComposer.build_document() in ReportEngine/core/stitcher.py performs IR stitching: sorting chapters, injecting unique anchors, merging metadata, and versioning the document structure.

Rendering and Persistence

Finally, HTMLRenderer (or MarkdownRenderer) consumes the IR via self.renderer.render(document_ir) to produce the output string. If save_report=True, _save_report() writes the HTML, IR JSON, and status files to self.config.OUTPUT_DIR/<report_id>/.

Processing Nodes: LLM-Driven Pipeline Steps

Each node inherits from BaseNode in ReportEngine/nodes/base_node.py and implements specific transformation logic.

TemplateSelectionNode

Located in ReportEngine/nodes/template_selection_node.py, this node calls the LLM with the query, engine reports, forum logs, and available templates. It returns a dictionary containing template_name, template_content, and selection_reason.

DocumentLayoutNode

This node in ReportEngine/nodes/document_layout_node.py generates the high-level document structure, determining title placement, hero imagery, navigation structure, and color theming.

WordBudgetNode

Implemented in ReportEngine/nodes/word_budget_node.py, this component analyzes template complexity and content depth to allocate word counts per chapter while establishing global writing constraints.

ChapterGenerationNode

Theheavy lifting occurs in ReportEngine/nodes/chapter_generation_node.py, which takes a TemplateSection and generation context to produce chapter JSON. It utilizes utils/json_parser.RobustJSONParser to handle malformed LLM outputs and supports configurable retry logic.

GraphRAGQueryNode

When enabled, ReportEngine/nodes/graphrag_query_node.py queries the knowledge graph built from agent states, returning graph_results and graph_enhancement_prompt to augment chapter content with relationship-aware context.

Core Utilities and IR Management

The utility layer ensures data integrity and transformation consistency throughout the pipeline.

Template Parsing

ReportEngine/core/template_parser.py defines the TemplateSection dataclass and parse_template_sections() function, which extracts hierarchical section numbers, URL slugs, and content outlines from Markdown templates.

IR Stitching

ReportEngine/core/stitcher.py contains DocumentComposer, which guarantees unique HTML anchors, assigns default chapterId values, and creates the final IR version stamped with IR_VERSION constants.

Chapter Storage and Validation

ReportEngine/core/chapter_storage.py creates per-run directories for debugging intermediate outputs. Supporting validation occurs in ReportEngine/utils/chart_validator.py and chart_review_service.py, which verify chart block well-formedness before rendering.

Configuration

ReportEngine/utils/config.py centralizes settings including API keys, model names, output directories, and feature toggles like GRAPHRAG_ENABLED.

Rendering Pipeline

The renderer layer converts the intermediate representation into human-readable formats.

HTMLRenderer

ReportEngine/renderers/html_renderer.py transforms IR into complete HTML pages, injecting CSS assets, anchor links, and dynamic table of contents navigation.

MarkdownRenderer

ReportEngine/renderers/markdown_renderer.py provides a fallback conversion that degrades unsupported block types (charts, word-clouds, widgets) into tables or textual placeholders, ensuring content accessibility even when advanced visualizations cannot render.

PDFRenderer

ReportEngine/renderers/pdf_renderer.py generates PDF documents by first rendering to HTML, then using weasyprint or headless browser engines to produce the final binary output.

State Management and Flask API

ReportEngine/state/state.py maintains the ReportState object, tracking metadata, generation timestamps, success/failure flags, and HTML content throughout the lifecycle.

The ReportEngine/flask_interface.py exposes REST endpoints including /run to initiate generation and /status/<task_id> for polling progress, enabling integration with external microservices and frontend applications.

End-to-End Workflow Example

Basic Synchronous Usage

from ReportEngine import ReportAgent
from config import settings

# Initialise the agent

agent = ReportAgent()

# Inputs from the three sub-engines

query_output   = "### Query engine summary …"

media_output   = "### Media analysis …"

insight_output = "### Insight findings …"

forum_logs     = "User A: …\nUser B: …"

# Generate report

result = agent.generate_report(
    query="武汉大学舆情分析",
    reports=[query_output, media_output, insight_output],
    forum_logs=forum_logs,
    custom_template="",   # Auto-select template

    save_report=True,
)

print("HTML length:", len(result["html_content"]))
print("Files:", result.get("html_path"), result.get("ir_path"))

Streaming Progress Events

def stream_handler(event_type, payload):
    # Forward to WebSocket or SSE client

    print(f"[{event_type}] {payload}")

html_result = agent.generate_report(
    query="武汉大学舆情分析",
    reports=[q, m, i],
    forum_logs=logs,
    stream_handler=stream_handler,
    save_report=False,
)

The stream_handler receives stage, progress, chapter_chunk, and error events emitted during the generation loop.

Disabling GraphRAG for Lightweight Runs

# Override at runtime

agent.config.GRAPHRAG_ENABLED = False

report = agent.generate_report(
    query="疫情舆情报告",
    reports=[q, m, i],
    forum_logs=logs,
    save_report=True,
)

When disabled, the engine skips graph building (lines ~600-630 of agent.py) and proceeds directly to chapter generation.

Summary

• The ReportEngine architecture comprises four layers: Orchestration (ReportAgent), Processing Nodes (LLM-driven transformers), Core Utilities (IR management), and Renderers (format converters).
• The pipeline is stateless between runs, persisting all intermediate data to per-run directories under OUTPUT_DIR/<report_id>/.
• ReportAgent.generate_report() implements an 11-step workflow from input normalization through template selection, chapter generation, IR stitching, and final rendering.
• Processing nodes inherit from BaseNode and handle specific tasks like template selection, layout design, word budgeting, and GraphRAG enhancement.
• IR stitching via DocumentComposer guarantees unique anchors and versioned metadata before HTML/Markdown rendering.
• Flask interface endpoints (/run, /status/<task_id>) enable asynchronous integration with external services.

Frequently Asked Questions

How does ReportEngine handle malformed LLM outputs during chapter generation?

The ChapterGenerationNode in ReportEngine/nodes/chapter_generation_node.py implements retry logic wrapped around utils/json_parser.RobustJSONParser. When JSON parsing fails, content filters trigger, or outputs are too sparse, the node automatically attempts regeneration using fallback LLM clients (json_rescue_clients) to ensure valid chapter structure without manual intervention.

What file formats can the ReportEngine generate?

According to the renderer implementations in ReportEngine/renderers/, the engine supports HTML via HTMLRenderer, Markdown via MarkdownRenderer, and PDF via PDFRenderer (which uses weasyprint or headless browser conversion). The Markdown renderer serves as a robust fallback that converts unsupported visual elements like charts into textual tables.

Is the ReportEngine architecture stateless, and where are outputs stored?

Yes, the architecture is designed to be stateless between runs. All intermediate chapter JSON files, final HTML, IR documents, and status logs are written to a unique per-run directory structure under self.config.OUTPUT_DIR/<report_id>/ as implemented in ReportEngine/core/chapter_storage.py and the _save_report() method of ReportAgent.

How does the ReportEngine integrate with the three analysis agents (Query, Media, Insight)?

The engine receives raw string outputs from these agents via the reports parameter of generate_report(), which expects a list containing the Query, Media, and Insight outputs in sequence. The _normalize_reports() method standardizes these inputs before the TemplateSelectionNode and subsequent processing nodes synthesize them into coherent chapters, optionally enhanced by GraphRAGQueryNode when knowledge graph features are enabled.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}