Configuring MediaCrawler Data Storage Backends: MySQL, PostgreSQL, SQLite, Excel, and JSONL

MediaCrawler uses a factory‑style store system that routes all data persistence through a single configuration variable config.SAVE_DATA_OPTION, allowing you to switch between relational databases, file formats, and NoSQL without modifying crawler logic.

The open‑source NanmiCoder/MediaCrawler repository provides a unified abstraction layer for saving scraped content from platforms like XiaoHongShu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. By decoupling storage implementation from crawling logic, developers can toggle between JSONL, Excel, SQLite, MySQL, PostgreSQL, or MongoDB by changing one setting in config/base_config.py or passing a command‑line flag.

Understanding the Storage Architecture

MediaCrawler’s persistence layer follows a three‑tier flow: configuration → factory → concrete store. This design ensures that platform‑specific crawlers remain agnostic to how data is saved.

The Configuration Layer

The entry point for backend selection resides in config/base_config.py, which defines the default SAVE_DATA_OPTION:

# config/base_config.py

SAVE_DATA_OPTION = "jsonl"  # Options: csv | db | json | jsonl | sqlite | excel | postgres

When launching a crawl, cmd_arg/arg.py parses the --save_data_option CLI flag and populates the global configuration. The validator uses SaveDataOptionEnum to ensure only supported backends (CSV, DB, JSON, JSONL, SQLITE, MONGODB, EXCEL, POSTGRES) are accepted. This value becomes the single source of truth for the entire application.

The Factory Pattern

Each platform implements a store factory (e.g., store/xhs/__init__.py for XiaoHongShu) that maps the SAVE_DATA_OPTION string to a concrete implementation class:

# store/xhs/__init__.py (excerpt)

STORES = {
    "csv": XhsCsvStoreImplement,
    "json": XhsJsonStoreImplement,
    "jsonl": XhsJsonlStoreImplement,
    "db": XhsDbStoreImplement,
    "sqlite": XhsSqliteStoreImplement,
    "mongodb": XhsMongoStoreImplement,
    "excel": XhsExcelStoreImplement,
    "postgres": XhsDbStoreImplement,   # PostgreSQL reuses the generic DB implement

}
store_class = XhsStoreFactory.STORES.get(config.SAVE_DATA_OPTION)

When a crawler retrieves content or comments, it calls the factory to obtain the appropriate writer, ensuring consistent data handling across all backends.

Supported Storage Backends

MediaCrawler supports eight distinct persistence mechanisms, categorized into relational databases and file‑based formats.

Relational Databases

MySQL, PostgreSQL, and SQLite share a common async SQLAlchemy foundation defined in database/db_session.py. The get_async_engine(db_type) function instantiates the correct engine based on SAVE_DATA_OPTION:

# database/db_session.py (excerpt)

db_type = config.SAVE_DATA_OPTION
engine = get_async_engine(db_type)   # Handles mysql, sqlite, postgres

Implementation classes such as XhsDbStoreImplement or ZhihuDbStoreImplement perform upserts using select, update, and delete statements against ORM models located in database/models.py. Notably, PostgreSQL uses the same XhsDbStoreImplement class as MySQL, differing only in the connection string and dialect handled by the engine factory.

File‑Based Formats

JSONL, Excel, CSV, and raw JSON bypass the database layer and write directly to the filesystem:

• JSONL (XhsJsonlStoreImplement, BiliJsonlStoreImplement, etc.) streams each record as a line‑delimited JSON object, enabling incremental processing and easy parsing by Unix tools.
• Excel output utilizes a singleton pattern via store/excel_store_base.py, allowing all platforms to append data to a single shared workbook rather than spawning multiple file handles.
• CSV and JSON implementations use tools/async_file_writer.py for non‑blocking I/O operations.

Implementation Details

Database Session Management

When SAVE_DATA_OPTION is set to db (MySQL), postgres, or sqlite, the main entry point main/main.py initializes the schema before crawling begins:

# main/main.py (excerpt)

await db.init_db(config.SAVE_DATA_OPTION)

This call ensures that tables defined in database/models.py are created via SQLAlchemy’s create_all() method before any write operations occur. The async engine supports connection pooling for MySQL and PostgreSQL while using a simple file‑based connection for SQLite.

Store Implementation Classes

Concrete implementations reside in platform‑specific files such as store/xhs/_store_impl.py. Each class implements an async save() method that accepts scraped data dictionaries. For database backends, this method wraps SQLAlchemy sessions; for file backends, it delegates to async_file_writer.py or the Excel singleton.

Configuration Examples

Switching backends requires no code changes beyond altering the SAVE_DATA_OPTION value. Below are runnable commands for each major backend.

Run with MySQL

python -m MediaCrawler.main --platform xhs \
    --save_data_option db \
    --init_db mysql \
    --keywords "python, data science"

The --save_data_option db flag directs the factory to instantiate XhsDbStoreImplement. The --init_db mysql argument triggers db.init_db("mysql"), creating the required MySQL tables via SQLAlchemy migrations.

Run with PostgreSQL

python -m MediaCrawler.main --platform zhihu \
    --save_data_option postgres \
    --init_db postgres \
    --keywords "machine learning"

PostgreSQL reuses the generic DB implementation classes (e.g., ZhihuDbStoreImplement) while database/db_session.py configures the async engine for the PostgreSQL dialect.

Run with SQLite

python -m MediaCrawler.main --platform douyin \
    --save_data_option sqlite \
    --init_db sqlite \
    --keywords "AI"

SQLite operates as a file‑based relational database, requiring no external server. The data is stored in a local .db file defined by the connection string in database/db_session.py.

Run with Excel Output

python -m MediaCrawler.main --platform ks \
    --save_data_option excel \
    --keywords "short video"

The factory returns KuaishouExcelStoreImplement, which appends records to a centralized workbook managed by store/excel_store_base.py. This singleton pattern prevents memory exhaustion when crawling large datasets.

Run with JSONL

python -m MediaCrawler.main --platform bili \
    --save_data_option jsonl \
    --keywords "anime"

BiliJsonlStoreImplement writes each content item as a separate JSON line, producing newline‑delimited JSON files ideal for downstream ETL pipelines or streaming analytics.

Summary

• Single Configuration Point: All storage routing flows through config/base_config.py and the SAVE_DATA_OPTION variable.
• Factory Abstraction: Platform‑specific factories in store/{platform}/__init__.py map configuration strings to concrete implementation classes.
• Unified Database Layer: MySQL, PostgreSQL, and SQLite share SQLAlchemy async engines defined in database/db_session.py, with PostgreSQL reusing the standard DB implement classes.
• File‑Based Flexibility: JSONL, CSV, and Excel writers use tools/async_file_writer.py or the Excel singleton in store/excel_store_base.py for efficient disk I/O.
• Zero Code Changes: Switching from SQLite to PostgreSQL or from JSONL to Excel requires only editing SAVE_DATA_OPTION or passing --save_data_option at runtime.

Frequently Asked Questions

How do I switch from JSONL to MySQL without changing Python code?

Set the --save_data_option db flag when running the crawler, or modify SAVE_DATA_OPTION = "db" in config/base_config.py. Ensure you also pass --init_db mysql so main/main.py executes db.init_db() to create the necessary tables before crawling begins.

Does PostgreSQL use the same implementation classes as MySQL?

Yes. According to the source code in store/xhs/__init__.py, the "postgres" key maps to XhsDbStoreImplement, identical to the "db" (MySQL) mapping. The distinction occurs in database/db_session.py, where get_async_engine() instantiates a PostgreSQL‑specific async engine based on the SAVE_DATA_OPTION value.

Where is the Excel workbook file created when using --save_data_option excel?

The Excel singleton defined in store/excel_store_base.py manages the workbook location. By default, it writes to a timestamped .xlsx file in the project root or a configured output directory. All platforms share this single instance, ensuring concurrent crawls append to the same spreadsheet rather than corrupting multiple file handles.

What is the difference between json and jsonl storage options?

The json option typically writes the entire dataset as a single JSON array to a file, while jsonl (JSON Lines) writes each scraped item as an individual JSON object per line. The XhsJsonlStoreImplement class streams records incrementally, making it suitable for large crawls where holding the entire dataset in memory would be impractical.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}