# BettaFish Data Pipeline: Complete Flow from Data Collection to Final Report Generation

> Explore the BettaFish Data Pipeline: Discover the complete flow from data collection with MindSpider to LLM-enhanced narratives from QueryEngine and final HTML reports via ReportEngine.

- Repository: [BaiFu/bettafish](https://github.com/666ghj/bettafish)
- Tags: how-to-guide
- Published: 2026-02-23

---

**BettaFish transforms raw social media data into polished HTML reports through a three-stage pipeline: MindSpider crawls multi-platform data into a relational database, QueryEngine generates LLM-enhanced narratives through iterative reflection, and ReportEngine renders the final output into structured HTML using dynamic templates.**

The BettaFish open-source project automates the transformation of scattered social media intelligence into publication-ready reports. Understanding the complete flow from data collection to final report generation reveals how the system orchestrates crawling, analysis, and rendering engines to deliver actionable insights.

## Stage 1: Multi-Platform Data Collection (MindSpider)

### Keyword-Driven Crawling Architecture

The data collection phase begins in [`MindSpider/DeepSentimentCrawling/platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/platform_crawler.py), where the `PlatformCrawler` class orchestrates multi-platform crawling through the `run_multi_platform_crawl_by_keywords` method (lines 63-71). This method constructs CLI commands for the external MediaCrawler tool and executes them as subprocesses with a one-hour timeout.

```python

# From MindSpider/DeepSentimentCrawling/platform_crawler.py

result = self.run_crawler(platform, keywords, login_type, max_notes_per_keyword)

```

The crawler supports platforms including XiaoHongShu (xhs) and Weibo, collecting notes, comments, and metadata based on keyword relevance. The method builds a command for the external MediaCrawler CLI using `sys.executable main.py`, runs the command in a subprocess, and parses the output to produce per-platform statistics (lines 63-71).

### Database Persistence and Statistics

Before crawling begins, `configure_mediacrawler_db()` establishes the database connection (lines 45-48). Crawled data persists into relational tables defined in `MindSpider/schema/models_*`, while runtime statistics accumulate in `self.crawl_stats[platform]` (lines 78-89), tracking note counts, comment volumes, and error rates per platform.

## Stage 2: Insight Generation and Narrative Construction (QueryEngine)

### Report Structure Initialization

The analysis phase centers in [`QueryEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/QueryEngine/agent.py), where the `QueryEngineAgent.research` method (lines 41-50) initiates the transformation of raw data into structured narratives. The engine first generates a report skeleton through `_generate_report_structure(query)` (lines 82-92), creating discrete paragraph objects that define the narrative architecture.

### Iterative Paragraph Processing

The core intelligence operates within `_process_paragraphs()` (lines 97-108), which iterates over each paragraph to perform initial search and summarization followed by reflection-based refinement.

**Initial Search and Summarization**: The `_initial_search_and_summary(i)` method (lines 24-40) constructs search queries, selects appropriate search tools, queries the previously crawled database, and feeds results to an LLM for first-pass summarization.

**Reflection and Refinement**: The `_reflection_loop(i)` repeatedly invokes the LLM to refine paragraph quality, with iteration limits defined by `config.MAX_REFLECTIONS`.

```python

# Conceptual flow from QueryEngine/agent.py

from QueryEngine.agent import QueryEngineAgent

agent = QueryEngineAgent()
final_report = agent.research(
    query="2025 年中国 AI 产业趋势分析",
    save_report=False,
)
print(final_report[:500])   # preview

```

The output is a cohesive markdown narrative integrating crawled data, search results, and domain-specific insights.

## Stage 3: Report Rendering and HTML Generation (ReportEngine)

### Template Selection and Normalization

The final rendering stage resides in [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py), where `ReportEngineAgent.generate_report` (lines 24-34) orchestrates the transformation of narrative content into polished HTML. The process begins with `_normalize_reports(reports)` (line 77), which merges outputs from QueryEngine, MediaEngine, and InsightEngine into a uniform structure.

### Document Architecture and Layout

The rendering pipeline proceeds through several specialized nodes:

**Template Selection**: `_select_template(...)` (lines 93-99) chooses an appropriate Markdown template based on content type and query context.

**Template Slicing**: `_slice_template(...)` (lines 106-108) decomposes the template into discrete sections for chapter-wise processing.

**Document Layout**: `document_layout_node.run(...)` (lines 14-24) establishes global design elements including title, hero section, and table of contents.

**Word Budget Planning**: `word_budget_node.run(...)` (lines 34-45) allocates target word counts per chapter based on content complexity and template constraints.

### Chapter Generation and Persistence

For each section, the engine invokes `ChapterGenerationNode` through `_run_stage_with_retry`, producing a JSON Intermediate Representation (IR) of the chapter content. The accumulated IR passes to an HTML renderer, generating the final styled document.

When `save_report=True`, the system persists outputs via `self.chapter_storage.start_session(...)` (line 96), writing HTML, IR JSON, and manifest files to a unique report directory.

```python

# From ReportEngine/agent.py usage pattern

from ReportEngine.agent import ReportEngineAgent

report_agent = ReportEngineAgent()
html_output = report_agent.generate_report(
    query="2025 年中国 AI 产业趋势分析",
    reports=[final_report],
    forum_logs="",
    custom_template="",
    save_report=True,
)
print("HTML written to:", html_output["html_path"])

```

## Summary

- **MindSpider** handles multi-platform data collection through `PlatformCrawler.run_multi_platform_crawl_by_keywords`, persisting raw social media data and metadata to a relational database.
- **QueryEngine** transforms crawled data into structured narratives via `QueryEngineAgent.research`, using iterative LLM reflection to refine paragraph-level insights.
- **ReportEngine** renders final outputs through `ReportEngineAgent.generate_report`, applying templates, layout algorithms, and word-budget planning to produce polished HTML reports.
- The pipeline uses loose coupling through Python objects and JSON structures, enabling independent scaling and customization of each stage.

## Frequently Asked Questions

### How does BettaFish handle authentication for social media crawling?

BettaFish delegates authentication to the external MediaCrawler tool, which supports multiple login types including QR code authentication (`login_type="qrcode"`). The `PlatformCrawler` class in [`MindSpider/DeepSentimentCrawling/platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/platform_crawler.py) passes these credentials through CLI arguments when spawning the subprocess, with timeout handling to prevent hanging on authentication prompts.

### What database schema does BettaFish use for storing crawled data?

The project uses relational tables defined in `MindSpider/schema/models_*` files to persist crawled content. The `configure_mediacrawler_db()` function in [`MindSpider/DeepSentimentCrawling/platform_crawler.py`](https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/platform_crawler.py) (lines 45-48) establishes the database connection before crawling begins, while the external MediaCrawler handles the actual INSERT operations into these schema-defined tables.

### Can I customize the report templates in BettaFish?

Yes, the `ReportEngine` supports custom templates through the `custom_template` parameter in `ReportEngineAgent.generate_report`. When provided, the engine uses `_select_template()` (lines 93-99 in [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py)) to load your specific Markdown template; otherwise, it selects automatically based on content type. The template is then sliced into sections via `_slice_template()` for chapter-wise processing.

### How does the QueryEngine ensure report quality through reflection?

The `QueryEngineAgent` implements an iterative refinement loop through `_reflection_loop(i)` called within `_process_paragraphs()` (lines 97-108 in [`QueryEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/QueryEngine/agent.py)). After the initial LLM summary generated by `_initial_search_and_summary()`, the reflection loop repeatedly critiques and improves each paragraph, with the maximum iteration count controlled by `config.MAX_REFLECTIONS`. This ensures factual accuracy and narrative coherence before final assembly.