BettaFish Data Pipeline: Complete Flow from Data Collection to Final Report Generation

BettaFish transforms raw social media data into polished HTML reports through a three-stage pipeline: MindSpider crawls multi-platform data into a relational database, QueryEngine generates LLM-enhanced narratives through iterative reflection, and ReportEngine renders the final output into structured HTML using dynamic templates.

The BettaFish open-source project automates the transformation of scattered social media intelligence into publication-ready reports. Understanding the complete flow from data collection to final report generation reveals how the system orchestrates crawling, analysis, and rendering engines to deliver actionable insights.

Stage 1: Multi-Platform Data Collection (MindSpider)

Keyword-Driven Crawling Architecture

The data collection phase begins in MindSpider/DeepSentimentCrawling/platform_crawler.py, where the PlatformCrawler class orchestrates multi-platform crawling through the run_multi_platform_crawl_by_keywords method (lines 63-71). This method constructs CLI commands for the external MediaCrawler tool and executes them as subprocesses with a one-hour timeout.

# From MindSpider/DeepSentimentCrawling/platform_crawler.py

result = self.run_crawler(platform, keywords, login_type, max_notes_per_keyword)

The crawler supports platforms including XiaoHongShu (xhs) and Weibo, collecting notes, comments, and metadata based on keyword relevance. The method builds a command for the external MediaCrawler CLI using sys.executable main.py, runs the command in a subprocess, and parses the output to produce per-platform statistics (lines 63-71).

Database Persistence and Statistics

Before crawling begins, configure_mediacrawler_db() establishes the database connection (lines 45-48). Crawled data persists into relational tables defined in MindSpider/schema/models_*, while runtime statistics accumulate in self.crawl_stats[platform] (lines 78-89), tracking note counts, comment volumes, and error rates per platform.

Stage 2: Insight Generation and Narrative Construction (QueryEngine)

Report Structure Initialization

The analysis phase centers in QueryEngine/agent.py, where the QueryEngineAgent.research method (lines 41-50) initiates the transformation of raw data into structured narratives. The engine first generates a report skeleton through _generate_report_structure(query) (lines 82-92), creating discrete paragraph objects that define the narrative architecture.

Iterative Paragraph Processing

The core intelligence operates within _process_paragraphs() (lines 97-108), which iterates over each paragraph to perform initial search and summarization followed by reflection-based refinement.

Initial Search and Summarization: The _initial_search_and_summary(i) method (lines 24-40) constructs search queries, selects appropriate search tools, queries the previously crawled database, and feeds results to an LLM for first-pass summarization.

Reflection and Refinement: The _reflection_loop(i) repeatedly invokes the LLM to refine paragraph quality, with iteration limits defined by config.MAX_REFLECTIONS.

# Conceptual flow from QueryEngine/agent.py

from QueryEngine.agent import QueryEngineAgent

agent = QueryEngineAgent()
final_report = agent.research(
    query="2025 年中国 AI 产业趋势分析",
    save_report=False,
)
print(final_report[:500])   # preview

The output is a cohesive markdown narrative integrating crawled data, search results, and domain-specific insights.

Stage 3: Report Rendering and HTML Generation (ReportEngine)

Template Selection and Normalization

The final rendering stage resides in ReportEngine/agent.py, where ReportEngineAgent.generate_report (lines 24-34) orchestrates the transformation of narrative content into polished HTML. The process begins with _normalize_reports(reports) (line 77), which merges outputs from QueryEngine, MediaEngine, and InsightEngine into a uniform structure.

Document Architecture and Layout

The rendering pipeline proceeds through several specialized nodes:

Template Selection: _select_template(...) (lines 93-99) chooses an appropriate Markdown template based on content type and query context.

Template Slicing: _slice_template(...) (lines 106-108) decomposes the template into discrete sections for chapter-wise processing.

Document Layout: document_layout_node.run(...) (lines 14-24) establishes global design elements including title, hero section, and table of contents.

Word Budget Planning: word_budget_node.run(...) (lines 34-45) allocates target word counts per chapter based on content complexity and template constraints.

Chapter Generation and Persistence

For each section, the engine invokes ChapterGenerationNode through _run_stage_with_retry, producing a JSON Intermediate Representation (IR) of the chapter content. The accumulated IR passes to an HTML renderer, generating the final styled document.

When save_report=True, the system persists outputs via self.chapter_storage.start_session(...) (line 96), writing HTML, IR JSON, and manifest files to a unique report directory.

# From ReportEngine/agent.py usage pattern

from ReportEngine.agent import ReportEngineAgent

report_agent = ReportEngineAgent()
html_output = report_agent.generate_report(
    query="2025 年中国 AI 产业趋势分析",
    reports=[final_report],
    forum_logs="",
    custom_template="",
    save_report=True,
)
print("HTML written to:", html_output["html_path"])

Summary

• MindSpider handles multi-platform data collection through PlatformCrawler.run_multi_platform_crawl_by_keywords, persisting raw social media data and metadata to a relational database.
• QueryEngine transforms crawled data into structured narratives via QueryEngineAgent.research, using iterative LLM reflection to refine paragraph-level insights.
• ReportEngine renders final outputs through ReportEngineAgent.generate_report, applying templates, layout algorithms, and word-budget planning to produce polished HTML reports.
• The pipeline uses loose coupling through Python objects and JSON structures, enabling independent scaling and customization of each stage.

Frequently Asked Questions

How does BettaFish handle authentication for social media crawling?

BettaFish delegates authentication to the external MediaCrawler tool, which supports multiple login types including QR code authentication (login_type="qrcode"). The PlatformCrawler class in MindSpider/DeepSentimentCrawling/platform_crawler.py passes these credentials through CLI arguments when spawning the subprocess, with timeout handling to prevent hanging on authentication prompts.

What database schema does BettaFish use for storing crawled data?

The project uses relational tables defined in MindSpider/schema/models_* files to persist crawled content. The configure_mediacrawler_db() function in MindSpider/DeepSentimentCrawling/platform_crawler.py (lines 45-48) establishes the database connection before crawling begins, while the external MediaCrawler handles the actual INSERT operations into these schema-defined tables.

Can I customize the report templates in BettaFish?

Yes, the ReportEngine supports custom templates through the custom_template parameter in ReportEngineAgent.generate_report. When provided, the engine uses _select_template() (lines 93-99 in ReportEngine/agent.py) to load your specific Markdown template; otherwise, it selects automatically based on content type. The template is then sliced into sections via _slice_template() for chapter-wise processing.

How does the QueryEngine ensure report quality through reflection?

The QueryEngineAgent implements an iterative refinement loop through _reflection_loop(i) called within _process_paragraphs() (lines 97-108 in QueryEngine/agent.py). After the initial LLM summary generated by _initial_search_and_summary(), the reflection loop repeatedly critiques and improves each paragraph, with the maximum iteration count controlled by config.MAX_REFLECTIONS. This ensures factual accuracy and narrative coherence before final assembly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}