How to Store Crawled Data in MySQL with MediaCrawler: Complete Configuration Guide

MediaCrawler persists scraped platform data to a MySQL database by setting the storage option to "db", which triggers an async SQLAlchemy engine with the asyncmy driver to insert records via ORM models.

The NanmiCoder/MediaCrawler repository provides a flexible storage abstraction that supports multiple backends. When configured for database persistence, the crawler automatically manages connection pooling, schema initialization, and asynchronous inserts for every extracted item across platforms like Zhihu, Tieba, and Weibo.

Configure MySQL Credentials

All connection parameters are centralized in config/db_config.py. The mysql_db_config dictionary defines the host, port, credentials, and database name used to construct the async connection URL.

# config/db_config.py

mysql_db_config = {
    "host": "127.0.0.1",
    "port": 3306,
    "user": "mc_user",
    "password": "mc_password",
    "db_name": "media_crawler",
}

You can override these values via environment variables or by modifying the dictionary directly. The database/db_session.py file consumes this configuration to instantiate the engine.

Select the Database Storage Option

To enable MySQL persistence, signal the application to use the database backend rather than JSON or CSV output. This is controlled by the SAVE_DATA_OPTION flag.

Method 1: Edit config/base_config.py:

# config/base_config.py

SAVE_DATA_OPTION = "db"  # Options: "json", "csv", "db"

Method 2: Pass the flag at runtime:

python -m main --platform zhihu --save_option db --init_db mysql

The --init_db mysql argument triggers schema creation if tables do not exist, ensuring the ORM models in database/models.py are synchronized with your database.

How the Async Storage Layer Works

When SAVE_DATA_OPTION equals "db", MediaCrawler initializes the pipeline through three core components:

Engine Creation in database/db_session.py

The session module constructs the database URL and engine dynamically. For MySQL, it uses the asyncmy dialect to enable non-blocking I/O.

# database/db_session.py (conceptual flow)

from sqlalchemy.ext.asyncio import create_async_engine

if db_type == "mysql":
    db_url = (
        f"mysql+asyncmy://{mysql_db_config['user']}:"
        f"{mysql_db_config['password']}@{mysql_db_config['host']}:"
        f"{mysql_db_config['port']}/{mysql_db_config['db_name']}"
    )
    engine = create_async_engine(db_url, echo=False)

ORM Models in database/models.py

Platform-specific crawlers define data classes that inherit from SQLAlchemy's declarative base. These map Python objects to MySQL tables, with columns representing scraped fields like content, author, and publish_time.

Store Implementation Selection

The entry point in main.py inspects the SAVE_DATA_OPTION configuration. When the value is "db", it delegates to StoreFactory.create_store(), which returns an instance of SqlStoreImplement. This class handles asynchronous insertion of each crawled item into the MySQL tables defined by the ORM models.

Running the Crawler with MySQL Backend

To store Zhihu search results in MySQL, execute:

python -m main \
    --platform zhihu \
    --save_option db \
    --init_db mysql

For custom implementations, the storage flow follows this pattern:

# Conceptual usage from main.py

import config
from store.factory import StoreFactory

if config.SAVE_DATA_OPTION == "db":
    store = StoreFactory.create_store(save_option="db")
    # store.save(item) is called automatically for each scraped item

The SqlStoreImplement class serializes each item according to the schema in database/models.py and commits it to the connection pool managed by the engine in database/db_session.py.

Summary

• Configuration: Set MySQL credentials in config/db_config.py and enable database mode via SAVE_DATA_OPTION = "db" in config/base_config.py or the --save_option db CLI flag.
• Engine: database/db_session.py constructs an async SQLAlchemy engine using the mysql+asyncmy dialect for non-blocking operations.
• Schema: ORM models in database/models.py define table structures; use --init_db mysql to create tables on first run.
• Persistence: main.py routes crawled items to SqlStoreImplement via StoreFactory.create_store() when database mode is active.

Frequently Asked Questions

What MySQL driver does MediaCrawler use?

MediaCrawler uses the asyncmy driver. This is specified in the SQLAlchemy connection URL as mysql+asyncmy:// inside database/db_session.py, enabling fully asynchronous database operations that do not block the crawler's event loop.

Can I use environment variables instead of editing db_config.py?

Yes. While the default configuration reads from the mysql_db_config dictionary in config/db_config.py, you can modify the file to read from environment variables (e.g., os.getenv("MYSQL_PASSWORD")) or inject configuration at runtime before the engine instantiation in database/db_session.py.

Does MediaCrawler automatically create the MySQL tables?

Yes. When you pass the --init_db mysql flag, MediaCrawler executes schema creation logic that ensures all tables defined in database/models.py exist. You only need to run this once or when upgrading to a version with new model definitions.

Which platforms support MySQL storage?

All platforms in the repository—including Zhihu, Tieba, Weibo, and others—support MySQL storage. Each platform's store implementation (e.g., store/zhihu/__init__.py) registers the "db" option, allowing SqlStoreImplement to handle the persistence layer generically across different content types.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}