# How to Store Crawled Data in MySQL with MediaCrawler: Complete Configuration Guide

> Learn how to store crawled data in MySQL with MediaCrawler. Configure the storage option to db for efficient data persistence using SQLAlchemy and asyncmy.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**MediaCrawler persists scraped platform data to a MySQL database by setting the storage option to `"db"`, which triggers an async SQLAlchemy engine with the `asyncmy` driver to insert records via ORM models.**

The **NanmiCoder/MediaCrawler** repository provides a flexible storage abstraction that supports multiple backends. When configured for database persistence, the crawler automatically manages connection pooling, schema initialization, and asynchronous inserts for every extracted item across platforms like Zhihu, Tieba, and Weibo.

## Configure MySQL Credentials

All connection parameters are centralized in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py). The `mysql_db_config` dictionary defines the host, port, credentials, and database name used to construct the async connection URL.

```python

# config/db_config.py

mysql_db_config = {
    "host": "127.0.0.1",
    "port": 3306,
    "user": "mc_user",
    "password": "mc_password",
    "db_name": "media_crawler",
}

```

You can override these values via environment variables or by modifying the dictionary directly. The [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py) file consumes this configuration to instantiate the engine.

## Select the Database Storage Option

To enable MySQL persistence, signal the application to use the database backend rather than JSON or CSV output. This is controlled by the `SAVE_DATA_OPTION` flag.

**Method 1:** Edit [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py):

```python

# config/base_config.py

SAVE_DATA_OPTION = "db"  # Options: "json", "csv", "db"

```

**Method 2:** Pass the flag at runtime:

```bash
python -m main --platform zhihu --save_option db --init_db mysql

```

The `--init_db mysql` argument triggers schema creation if tables do not exist, ensuring the ORM models in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) are synchronized with your database.

## How the Async Storage Layer Works

When `SAVE_DATA_OPTION` equals `"db"`, MediaCrawler initializes the pipeline through three core components:

**Engine Creation in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py)**

The session module constructs the database URL and engine dynamically. For MySQL, it uses the `asyncmy` dialect to enable non-blocking I/O.

```python

# database/db_session.py (conceptual flow)

from sqlalchemy.ext.asyncio import create_async_engine

if db_type == "mysql":
    db_url = (
        f"mysql+asyncmy://{mysql_db_config['user']}:"
        f"{mysql_db_config['password']}@{mysql_db_config['host']}:"
        f"{mysql_db_config['port']}/{mysql_db_config['db_name']}"
    )
    engine = create_async_engine(db_url, echo=False)

```

**ORM Models in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py)**

Platform-specific crawlers define data classes that inherit from SQLAlchemy's declarative base. These map Python objects to MySQL tables, with columns representing scraped fields like `content`, `author`, and `publish_time`.

**Store Implementation Selection**

The entry point in [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) inspects the `SAVE_DATA_OPTION` configuration. When the value is `"db"`, it delegates to `StoreFactory.create_store()`, which returns an instance of `SqlStoreImplement`. This class handles asynchronous insertion of each crawled item into the MySQL tables defined by the ORM models.

## Running the Crawler with MySQL Backend

To store Zhihu search results in MySQL, execute:

```bash
python -m main \
    --platform zhihu \
    --save_option db \
    --init_db mysql

```

For custom implementations, the storage flow follows this pattern:

```python

# Conceptual usage from main.py

import config
from store.factory import StoreFactory

if config.SAVE_DATA_OPTION == "db":
    store = StoreFactory.create_store(save_option="db")
    # store.save(item) is called automatically for each scraped item

```

The `SqlStoreImplement` class serializes each item according to the schema in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) and commits it to the connection pool managed by the engine in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py).

## Summary

- **Configuration**: Set MySQL credentials in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py) and enable database mode via `SAVE_DATA_OPTION = "db"` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) or the `--save_option db` CLI flag.
- **Engine**: [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py) constructs an async SQLAlchemy engine using the `mysql+asyncmy` dialect for non-blocking operations.
- **Schema**: ORM models in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) define table structures; use `--init_db mysql` to create tables on first run.
- **Persistence**: [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) routes crawled items to `SqlStoreImplement` via `StoreFactory.create_store()` when database mode is active.

## Frequently Asked Questions

### What MySQL driver does MediaCrawler use?

MediaCrawler uses the **`asyncmy`** driver. This is specified in the SQLAlchemy connection URL as `mysql+asyncmy://` inside [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py), enabling fully asynchronous database operations that do not block the crawler's event loop.

### Can I use environment variables instead of editing [`db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/db_config.py)?

Yes. While the default configuration reads from the `mysql_db_config` dictionary in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py), you can modify the file to read from environment variables (e.g., `os.getenv("MYSQL_PASSWORD")`) or inject configuration at runtime before the engine instantiation in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py).

### Does MediaCrawler automatically create the MySQL tables?

Yes. When you pass the `--init_db mysql` flag, MediaCrawler executes schema creation logic that ensures all tables defined in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) exist. You only need to run this once or when upgrading to a version with new model definitions.

### Which platforms support MySQL storage?

All platforms in the repository—including **Zhihu**, **Tieba**, **Weibo**, and others—support MySQL storage. Each platform's store implementation (e.g., [`store/zhihu/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/__init__.py)) registers the `"db"` option, allowing `SqlStoreImplement` to handle the persistence layer generically across different content types.