# Configuring MediaCrawler Data Storage Backends: MySQL, PostgreSQL, SQLite, Excel, and JSONL

> Easily configure MediaCrawler data storage for MySQL, PostgreSQL, SQLite, Excel, and JSONL. Switch backends effortlessly without changing crawler code using config.SAVE_DATA_OPTION.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-07-31

---

**MediaCrawler uses a factory‑style store system that routes all data persistence through a single configuration variable `config.SAVE_DATA_OPTION`, allowing you to switch between relational databases, file formats, and NoSQL without modifying crawler logic.**

The open‑source **NanmiCoder/MediaCrawler** repository provides a unified abstraction layer for saving scraped content from platforms like XiaoHongShu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. By decoupling storage implementation from crawling logic, developers can toggle between JSONL, Excel, SQLite, MySQL, PostgreSQL, or MongoDB by changing one setting in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) or passing a command‑line flag.

## Understanding the Storage Architecture

MediaCrawler’s persistence layer follows a three‑tier flow: **configuration → factory → concrete store**. This design ensures that platform‑specific crawlers remain agnostic to how data is saved.

### The Configuration Layer

The entry point for backend selection resides in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), which defines the default `SAVE_DATA_OPTION`:

```python

# config/base_config.py

SAVE_DATA_OPTION = "jsonl"  # Options: csv | db | json | jsonl | sqlite | excel | postgres

```

When launching a crawl, [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) parses the `--save_data_option` CLI flag and populates the global configuration. The validator uses `SaveDataOptionEnum` to ensure only supported backends (CSV, DB, JSON, JSONL, SQLITE, MONGODB, EXCEL, POSTGRES) are accepted. This value becomes the single source of truth for the entire application.

### The Factory Pattern

Each platform implements a store factory (e.g., [`store/xhs/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/__init__.py) for XiaoHongShu) that maps the `SAVE_DATA_OPTION` string to a concrete implementation class:

```python

# store/xhs/__init__.py (excerpt)

STORES = {
    "csv": XhsCsvStoreImplement,
    "json": XhsJsonStoreImplement,
    "jsonl": XhsJsonlStoreImplement,
    "db": XhsDbStoreImplement,
    "sqlite": XhsSqliteStoreImplement,
    "mongodb": XhsMongoStoreImplement,
    "excel": XhsExcelStoreImplement,
    "postgres": XhsDbStoreImplement,   # PostgreSQL reuses the generic DB implement

}
store_class = XhsStoreFactory.STORES.get(config.SAVE_DATA_OPTION)

```

When a crawler retrieves content or comments, it calls the factory to obtain the appropriate writer, ensuring consistent data handling across all backends.

## Supported Storage Backends

MediaCrawler supports eight distinct persistence mechanisms, categorized into relational databases and file‑based formats.

### Relational Databases

**MySQL, PostgreSQL, and SQLite** share a common async SQLAlchemy foundation defined in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py). The `get_async_engine(db_type)` function instantiates the correct engine based on `SAVE_DATA_OPTION`:

```python

# database/db_session.py (excerpt)

db_type = config.SAVE_DATA_OPTION
engine = get_async_engine(db_type)   # Handles mysql, sqlite, postgres

```

Implementation classes such as `XhsDbStoreImplement` or `ZhihuDbStoreImplement` perform upserts using `select`, `update`, and `delete` statements against ORM models located in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py). Notably, PostgreSQL uses the same `XhsDbStoreImplement` class as MySQL, differing only in the connection string and dialect handled by the engine factory.

### File‑Based Formats

**JSONL, Excel, CSV, and raw JSON** bypass the database layer and write directly to the filesystem:

- **JSONL** (`XhsJsonlStoreImplement`, `BiliJsonlStoreImplement`, etc.) streams each record as a line‑delimited JSON object, enabling incremental processing and easy parsing by Unix tools.
- **Excel** output utilizes a singleton pattern via [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py), allowing all platforms to append data to a single shared workbook rather than spawning multiple file handles.
- **CSV** and **JSON** implementations use [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) for non‑blocking I/O operations.

## Implementation Details

### Database Session Management

When `SAVE_DATA_OPTION` is set to `db` (MySQL), `postgres`, or `sqlite`, the main entry point [`main/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main/main.py) initializes the schema before crawling begins:

```python

# main/main.py (excerpt)

await db.init_db(config.SAVE_DATA_OPTION)

```

This call ensures that tables defined in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) are created via SQLAlchemy’s `create_all()` method before any write operations occur. The async engine supports connection pooling for MySQL and PostgreSQL while using a simple file‑based connection for SQLite.

### Store Implementation Classes

Concrete implementations reside in platform‑specific files such as [`store/xhs/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/_store_impl.py). Each class implements an async `save()` method that accepts scraped data dictionaries. For database backends, this method wraps SQLAlchemy sessions; for file backends, it delegates to [`async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/async_file_writer.py) or the Excel singleton.

## Configuration Examples

Switching backends requires no code changes beyond altering the `SAVE_DATA_OPTION` value. Below are runnable commands for each major backend.

### Run with MySQL

```bash
python -m MediaCrawler.main --platform xhs \
    --save_data_option db \
    --init_db mysql \
    --keywords "python, data science"

```

The `--save_data_option db` flag directs the factory to instantiate `XhsDbStoreImplement`. The `--init_db mysql` argument triggers `db.init_db("mysql")`, creating the required MySQL tables via SQLAlchemy migrations.

### Run with PostgreSQL

```bash
python -m MediaCrawler.main --platform zhihu \
    --save_data_option postgres \
    --init_db postgres \
    --keywords "machine learning"

```

PostgreSQL reuses the generic DB implementation classes (e.g., `ZhihuDbStoreImplement`) while [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py) configures the async engine for the PostgreSQL dialect.

### Run with SQLite

```bash
python -m MediaCrawler.main --platform douyin \
    --save_data_option sqlite \
    --init_db sqlite \
    --keywords "AI"

```

SQLite operates as a file‑based relational database, requiring no external server. The data is stored in a local `.db` file defined by the connection string in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py).

### Run with Excel Output

```bash
python -m MediaCrawler.main --platform ks \
    --save_data_option excel \
    --keywords "short video"

```

The factory returns `KuaishouExcelStoreImplement`, which appends records to a centralized workbook managed by [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py). This singleton pattern prevents memory exhaustion when crawling large datasets.

### Run with JSONL

```bash
python -m MediaCrawler.main --platform bili \
    --save_data_option jsonl \
    --keywords "anime"

```

`BiliJsonlStoreImplement` writes each content item as a separate JSON line, producing newline‑delimited JSON files ideal for downstream ETL pipelines or streaming analytics.

## Summary

- **Single Configuration Point:** All storage routing flows through [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and the `SAVE_DATA_OPTION` variable.
- **Factory Abstraction:** Platform‑specific factories in `store/{platform}/__init__.py` map configuration strings to concrete implementation classes.
- **Unified Database Layer:** MySQL, PostgreSQL, and SQLite share SQLAlchemy async engines defined in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py), with PostgreSQL reusing the standard DB implement classes.
- **File‑Based Flexibility:** JSONL, CSV, and Excel writers use [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) or the Excel singleton in [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py) for efficient disk I/O.
- **Zero Code Changes:** Switching from SQLite to PostgreSQL or from JSONL to Excel requires only editing `SAVE_DATA_OPTION` or passing `--save_data_option` at runtime.

## Frequently Asked Questions

### How do I switch from JSONL to MySQL without changing Python code?

Set the `--save_data_option db` flag when running the crawler, or modify `SAVE_DATA_OPTION = "db"` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py). Ensure you also pass `--init_db mysql` so [`main/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main/main.py) executes `db.init_db()` to create the necessary tables before crawling begins.

### Does PostgreSQL use the same implementation classes as MySQL?

Yes. According to the source code in [`store/xhs/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/__init__.py), the `"postgres"` key maps to `XhsDbStoreImplement`, identical to the `"db"` (MySQL) mapping. The distinction occurs in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py), where `get_async_engine()` instantiates a PostgreSQL‑specific async engine based on the `SAVE_DATA_OPTION` value.

### Where is the Excel workbook file created when using `--save_data_option excel`?

The Excel singleton defined in [`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py) manages the workbook location. By default, it writes to a timestamped `.xlsx` file in the project root or a configured output directory. All platforms share this single instance, ensuring concurrent crawls append to the same spreadsheet rather than corrupting multiple file handles.

### What is the difference between `json` and `jsonl` storage options?

The `json` option typically writes the entire dataset as a single JSON array to a file, while `jsonl` (JSON Lines) writes each scraped item as an individual JSON object per line. The `XhsJsonlStoreImplement` class streams records incrementally, making it suitable for large crawls where holding the entire dataset in memory would be impractical.