MediaCrawler Output Format: Configuration Guide for JSONL, CSV, Excel, and Databases

MediaCrawler defaults to JSONL (JSON Lines) output but supports seven additional formats including CSV, Excel, SQLite, MySQL, PostgreSQL, MongoDB, and standard JSON, configurable via the SAVE_DATA_OPTION setting or --save_data_option CLI flag.

The open-source MediaCrawler project (NanmiCoder/MediaCrawler) provides flexible data persistence for scraped content across platforms like Zhihu, XHS, and Weibo. Understanding the MediaCrawler output format options is essential for integrating scraped data into your analytics pipeline or database architecture.

Default JSONL Output Format

By default, MediaCrawler writes all scraped data as JSONL (JSON Lines), where each line represents a single JSON object. This configuration is defined in config/base_config.py where SAVE_DATA_OPTION = "jsonl".

The JSONL format offers significant advantages for web scraping: it enables fast append operations and maintains low memory usage since the writer does not need to hold the entire dataset in memory. The AsyncFileWriter class in tools/async_file_writer.py handles these writes asynchronously, ensuring non-blocking I/O operations during high-volume crawling.

Supported MediaCrawler Output Formats

MediaCrawler implements a pluggable storage architecture supporting multiple output formats. Each format uses a dedicated implementation class within the store/ directory:

• JSONL (default): Line-delimited JSON using write_to_jsonl in AsyncFileWriter
• CSV: Tabular data written via AsyncFileWriter.write_to_csv, implemented in store/*/CsvStoreImplement
• JSON: Complete document files using write_single_item_to_json (e.g., store/zhihu/_store_impl.py)
• Excel: Structured .xlsx files with multiple sheets via ExcelStoreBase singleton
• SQLite: Local relational database using SqliteStoreImplement and SQLAlchemy sessions
• MySQL/PostgreSQL: Production database support via DbStoreImplement with SQLAlchemy ORM
• MongoDB: NoSQL document storage using MongoDBStoreBase and MongoStoreImplement

All file-based formats utilize the centralized AsyncFileWriter in tools/async_file_writer.py to abstract file handling and ensure consistent asynchronous I/O across different output types.

How Output Format Selection Works

The MediaCrawler output format propagates through a three-stage factory pattern:

1. Configuration Reading: During startup, MediaCrawler reads SAVE_DATA_OPTION from config/base_config.py or CLI arguments
2. Factory Mapping: Each platform package (Zhihu, XHS, Weibo, etc.) contains a StoreFactory in its __init__.py that maps the option string to concrete implementation classes (e.g., "jsonl" → ZhihuJsonlStoreImplement)
3. Writer Execution: The selected store class calls AsyncFileWriter methods that write to the appropriate file type or database connection

This architecture ensures that changing the output format requires modifying only the configuration, while the underlying crawler logic remains unchanged.

Changing MediaCrawler Output Format via CLI

You can override the default output format using the --save_data_option flag when running the crawler:

# Default JSONL output

uv run main.py --platform xhs --lt qrcode --type search

# Explicit CSV output

uv run main.py --platform xhs --lt qrcode --type search --save_data_option csv

# Excel output for business analysis

uv run main.py --platform xhs --lt qrcode --type search --save_data_option excel

# Database output (PostgreSQL)

uv run main.py --platform zhihu --lt qrcode --type search --save_data_option db

All commands respect the --save_data_option flag, which takes precedence over the config/base_config.py setting.

Programmatic Control of Output Formats

For custom implementations, you can interact directly with the AsyncFileWriter class:

from tools.async_file_writer import AsyncFileWriter

# Initialize writer for specific platform and crawler type

writer = AsyncFileWriter(platform="zhihu", crawler_type="search")

# Write content as JSONL (default behavior)

await writer.write_to_jsonl(
    item_type="contents", 
    item={"content_id": "123", "title": "Example Post"}
)

# Write comments as CSV

await writer.write_to_csv(
    item_type="comments", 
    item={"comment_id": "c1", "text": "Sample comment"}
)

The writer automatically creates directories under data/ and names files using the pattern <run_index>_<type>.jsonl or <run_index>_<type>.csv based on the method called.

Alternatively, use the factory pattern for database storage:

from store.zhihu import ZhihuStoreFactory
import config

# Set format before creating store

config.SAVE_DATA_OPTION = "excel"  # or "jsonl", "csv", "db"

store = ZhihuStoreFactory.create_store()

# Store will use Excel implementation automatically

await store.store_content({"content_id": "123", "title": "Demo"})

Summary

• MediaCrawler defaults to JSONL output format configured in config/base_config.py via SAVE_DATA_OPTION
• Seven additional formats are supported: CSV, JSON, Excel, SQLite, MySQL, PostgreSQL, and MongoDB
• Format selection uses a Factory pattern mapping configuration strings to implementation classes in store/*/
• The AsyncFileWriter in tools/async_file_writer.py handles all asynchronous file I/O operations
• Change formats via the --save_data_option CLI flag or by modifying SAVE_DATA_OPTION before runtime
• Output files are automatically organized in data/ with naming patterns like <run_index>_<type>.jsonl

Frequently Asked Questions

What is the default output format of MediaCrawler?

MediaCrawler defaults to JSONL (JSON Lines), where each scraped item is written as a single JSON object per line. This is defined by SAVE_DATA_OPTION = "jsonl" in config/base_config.py. JSONL provides efficient append operations and minimal memory overhead during large crawling operations.

How do I change MediaCrawler output to CSV or Excel?

Use the --save_data_option CLI flag followed by the format name: csv for comma-separated values or excel for .xlsx files. For example: uv run main.py --platform xhs --save_data_option csv. You can also permanently change the default by editing SAVE_DATA_OPTION in config/base_config.py.

Where does MediaCrawler save output files?

File-based outputs (JSONL, CSV, JSON, Excel) are saved in the data/ directory, automatically created at runtime. Files follow the naming pattern <run_index>_<item_type>.<ext> (e.g., 1_contents.jsonl or 1_comments.csv). The AsyncFileWriter class manages directory creation and file path generation based on the platform and crawler type parameters.

Can MediaCrawler write to multiple databases simultaneously?

While MediaCrawler supports SQLite, MySQL, PostgreSQL, and MongoDB through DbStoreImplement and MongoStoreImplement, the architecture uses a single SAVE_DATA_OPTION value per crawl session. To write to multiple databases simultaneously, you would need to modify the store implementation to instantiate multiple writers or run separate crawler instances with different configuration files targeting different storage backends.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}