# MediaCrawler Output Format: Configuration Guide for JSONL, CSV, Excel, and Databases

> Explore MediaCrawler output formats. Learn to configure JSONL, CSV, Excel, and databases like SQL & MongoDB for your data needs. Get your data your way.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: configuration-guide
- Published: 2026-06-28

---

**MediaCrawler defaults to JSONL (JSON Lines) output but supports seven additional formats including CSV, Excel, SQLite, MySQL, PostgreSQL, MongoDB, and standard JSON, configurable via the `SAVE_DATA_OPTION` setting or `--save_data_option` CLI flag.**

The open-source MediaCrawler project (NanmiCoder/MediaCrawler) provides flexible data persistence for scraped content across platforms like Zhihu, XHS, and Weibo. Understanding the MediaCrawler output format options is essential for integrating scraped data into your analytics pipeline or database architecture.

## Default JSONL Output Format

By default, MediaCrawler writes all scraped data as **JSONL** (JSON Lines), where each line represents a single JSON object. This configuration is defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) where `SAVE_DATA_OPTION = "jsonl"`.

The JSONL format offers significant advantages for web scraping: it enables fast append operations and maintains low memory usage since the writer does not need to hold the entire dataset in memory. The `AsyncFileWriter` class in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) handles these writes asynchronously, ensuring non-blocking I/O operations during high-volume crawling.

## Supported MediaCrawler Output Formats

MediaCrawler implements a pluggable storage architecture supporting multiple output formats. Each format uses a dedicated implementation class within the `store/` directory:

- **JSONL** (default): Line-delimited JSON using `write_to_jsonl` in `AsyncFileWriter`
- **CSV**: Tabular data written via `AsyncFileWriter.write_to_csv`, implemented in `store/*/CsvStoreImplement`
- **JSON**: Complete document files using `write_single_item_to_json` (e.g., [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py))
- **Excel**: Structured `.xlsx` files with multiple sheets via `ExcelStoreBase` singleton
- **SQLite**: Local relational database using `SqliteStoreImplement` and SQLAlchemy sessions
- **MySQL/PostgreSQL**: Production database support via `DbStoreImplement` with SQLAlchemy ORM
- **MongoDB**: NoSQL document storage using `MongoDBStoreBase` and `MongoStoreImplement`

All file-based formats utilize the centralized `AsyncFileWriter` in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) to abstract file handling and ensure consistent asynchronous I/O across different output types.

## How Output Format Selection Works

The MediaCrawler output format propagates through a three-stage factory pattern:

1. **Configuration Reading**: During startup, MediaCrawler reads `SAVE_DATA_OPTION` from [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) or CLI arguments
2. **Factory Mapping**: Each platform package (Zhihu, XHS, Weibo, etc.) contains a `StoreFactory` in its [`__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/__init__.py) that maps the option string to concrete implementation classes (e.g., `"jsonl"` → `ZhihuJsonlStoreImplement`)
3. **Writer Execution**: The selected store class calls `AsyncFileWriter` methods that write to the appropriate file type or database connection

This architecture ensures that changing the output format requires modifying only the configuration, while the underlying crawler logic remains unchanged.

## Changing MediaCrawler Output Format via CLI

You can override the default output format using the `--save_data_option` flag when running the crawler:

```bash

# Default JSONL output

uv run main.py --platform xhs --lt qrcode --type search

# Explicit CSV output

uv run main.py --platform xhs --lt qrcode --type search --save_data_option csv

# Excel output for business analysis

uv run main.py --platform xhs --lt qrcode --type search --save_data_option excel

# Database output (PostgreSQL)

uv run main.py --platform zhihu --lt qrcode --type search --save_data_option db

```

All commands respect the `--save_data_option` flag, which takes precedence over the [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) setting.

## Programmatic Control of Output Formats

For custom implementations, you can interact directly with the `AsyncFileWriter` class:

```python
from tools.async_file_writer import AsyncFileWriter

# Initialize writer for specific platform and crawler type

writer = AsyncFileWriter(platform="zhihu", crawler_type="search")

# Write content as JSONL (default behavior)

await writer.write_to_jsonl(
    item_type="contents", 
    item={"content_id": "123", "title": "Example Post"}
)

# Write comments as CSV

await writer.write_to_csv(
    item_type="comments", 
    item={"comment_id": "c1", "text": "Sample comment"}
)

```

The writer automatically creates directories under `data/` and names files using the pattern `<run_index>_<type>.jsonl` or `<run_index>_<type>.csv` based on the method called.

Alternatively, use the factory pattern for database storage:

```python
from store.zhihu import ZhihuStoreFactory
import config

# Set format before creating store

config.SAVE_DATA_OPTION = "excel"  # or "jsonl", "csv", "db"

store = ZhihuStoreFactory.create_store()

# Store will use Excel implementation automatically

await store.store_content({"content_id": "123", "title": "Demo"})

```

## Summary

- MediaCrawler defaults to **JSONL** output format configured in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) via `SAVE_DATA_OPTION`
- Seven additional formats are supported: **CSV, JSON, Excel, SQLite, MySQL, PostgreSQL, and MongoDB**
- Format selection uses a **Factory pattern** mapping configuration strings to implementation classes in `store/*/`
- The `AsyncFileWriter` in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) handles all asynchronous file I/O operations
- Change formats via the `--save_data_option` CLI flag or by modifying `SAVE_DATA_OPTION` before runtime
- Output files are automatically organized in `data/` with naming patterns like `<run_index>_<type>.jsonl`

## Frequently Asked Questions

### What is the default output format of MediaCrawler?

MediaCrawler defaults to **JSONL** (JSON Lines), where each scraped item is written as a single JSON object per line. This is defined by `SAVE_DATA_OPTION = "jsonl"` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). JSONL provides efficient append operations and minimal memory overhead during large crawling operations.

### How do I change MediaCrawler output to CSV or Excel?

Use the `--save_data_option` CLI flag followed by the format name: `csv` for comma-separated values or `excel` for `.xlsx` files. For example: `uv run main.py --platform xhs --save_data_option csv`. You can also permanently change the default by editing `SAVE_DATA_OPTION` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py).

### Where does MediaCrawler save output files?

File-based outputs (JSONL, CSV, JSON, Excel) are saved in the `data/` directory, automatically created at runtime. Files follow the naming pattern `<run_index>_<item_type>.<ext>` (e.g., `1_contents.jsonl` or `1_comments.csv`). The `AsyncFileWriter` class manages directory creation and file path generation based on the platform and crawler type parameters.

### Can MediaCrawler write to multiple databases simultaneously?

While MediaCrawler supports SQLite, MySQL, PostgreSQL, and MongoDB through `DbStoreImplement` and `MongoStoreImplement`, the architecture uses a single `SAVE_DATA_OPTION` value per crawl session. To write to multiple databases simultaneously, you would need to modify the store implementation to instantiate multiple writers or run separate crawler instances with different configuration files targeting different storage backends.