# How to Store Crawled Data in MongoDB with MediaCrawler: A Complete Guide

> Learn to store crawled data in MongoDB using MediaCrawler. This guide details configuring your mongodb_config for seamless data persistence with Motor async connections and automatic upserts.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**MediaCrawler provides a reusable MongoDB abstraction layer that handles async connections via Motor, automatic upserts, and platform-specific collections, allowing you to persist crawled data by configuring `mongodb_config` in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py) and using the `MongoDBStoreBase` class or platform-specific store implementations.**

MediaCrawler is an open-source multi-platform content crawler that ships with built-in MongoDB persistence. This guide explains how to store crawled data in MongoDB with MediaCrawler using the async storage layer implemented in [`database/mongodb_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/mongodb_store_base.py) and platform-specific store classes that handle data normalization and collection management.

## Configure MongoDB Connection Parameters

All MongoDB connection settings are centralized in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py). The `mongodb_config` dictionary defines the parameters for the async Motor client to establish a connection.

The configuration supports optional authentication. When `user` and `password` are provided, the `MongoDBConnection` singleton automatically constructs a connection URL with credentials; otherwise, it connects without authentication.

```python

# config/db_config.py

mongodb_config = {
    "host": "localhost",
    "port": 27017,
    "user": "",           # Optional: leave empty for no auth

    "password": "",       # Optional: leave empty for no auth

    "db_name": "media_crawler",
}

```

The `MongoDBConnection` class reads this dictionary and creates an async Motor client instance that is shared across the application, ensuring efficient connection pooling and non-blocking I/O operations.

## Core MongoDB Storage Abstraction

The [`database/mongodb_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/mongodb_store_base.py) file defines `MongoDBStoreBase`, the foundational class for all MongoDB operations in MediaCrawler. This class encapsulates the Motor client interaction and provides CRUD operations with automatic BSON-to-dict conversion.

Key methods include:

- **`save_or_update`** – Performs an upsert operation using a query filter and data dictionary, creating or updating documents atomically to prevent duplicates.
- **`find_one`** – Retrieves a single document matching the specified filter.
- **`find_many`** – Returns multiple documents with optional limit and skip parameters.
- **`create_index`** – Builds indexes on specified fields for optimized queries.

The constructor requires a `collection_prefix` parameter (e.g., `"xhs"` for Xiaohongshu, `"zhihu"` for Zhihu) that prefixes all collection names, ensuring data isolation between platforms in the same database.

## Platform-Specific Store Implementations

Each supported platform implements a concrete store class that instantiates `MongoDBStoreBase` with the appropriate prefix and provides semantic methods for different content types.

For example, in [`store/xhs/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/_store_impl.py), the `XhsMongoStoreImplement` class initializes the base store with `collection_prefix="xhs"` and exposes methods like `store_content`, `store_comment`, and `store_creator`. These methods internally call `save_or_update` with specific collection suffixes:

- **Contents**: `collection_suffix="contents"` for posts and notes
- **Comments**: `collection_suffix="comments"` for user comments
- **Creators**: `collection_suffix="creators"` for user profile data

Similarly, [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py) contains `ZhihuMongoStoreImplement` using the prefix `"zhihu"`.

## Storing Crawled Data: Code Examples

### Using the Base Class Directly

For custom storage logic, instantiate `MongoDBStoreBase` and call the upsert method directly:

```python
from database.mongodb_store_base import MongoDBStoreBase

# Initialize store for Xiaohongshu data

mongo_store = MongoDBStoreBase(collection_prefix="xhs")

# Prepare crawled note data

note_data = {
    "note_id": "abc123",
    "user_id": "u001",
    "title": "My travel vlog",
    "desc": "A short description",
    "liked_count": "150",
    "image_list": ["https://example.com/img1.jpg"],
}

# Upsert the document (creates or updates)

await mongo_store.save_or_update(
    collection_suffix="contents",
    query={"note_id": note_data["note_id"]},
    data=note_data,
)

```

### Using the High-Level Platform Store

For standard platforms, use the provided implementation class to handle boilerplate:

```python
from store.xhs._store_impl import XhsMongoStoreImplement

xhs_store = XhsMongoStoreImplement()

# Store a note - automatically handles collection selection and upsert logic

await xhs_store.store_content(note_data)

# Store a comment

comment_data = {
    "comment_id": "cmt001",
    "note_id": "abc123",
    "text": "Nice post!"
}
await xhs_store.store_comment(comment_data)

```

Both approaches ensure that repeated crawls update existing records rather than creating duplicates, thanks to the query-based upsert mechanism in `save_or_update`.

## Querying Stored Data

Retrieve data using the base class methods, which return native Python dictionaries:

```python

# Fetch a specific note by ID

note = await mongo_store.find_one("contents", {"note_id": "abc123"})
print(note["title"])  # Output: My travel vlog

# Fetch the latest 100 comments

comments = await mongo_store.find_many("comments", {}, limit=100)

```

The async methods return Motor cursors that are automatically converted to lists of dictionaries, making integration with the rest of your Python application seamless.

## Summary

- **Configuration**: Define connection parameters in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py) under `mongodb_config`; the `MongoDBConnection` singleton handles async Motor client initialization with optional authentication.
- **Storage Layer**: `MongoDBStoreBase` in [`database/mongodb_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/mongodb_store_base.py) provides the core CRUD operations including `save_or_update` for atomic upserts.
- **Platform Isolation**: Use `collection_prefix` (e.g., `"xhs"`, `"zhihu"`) to namespace collections, and `collection_suffix` to separate content types (contents, comments, creators).
- **Implementation Pattern**: Platform stores like `XhsMongoStoreImplement` wrap the base class and provide semantic methods for different data models.
- **Duplicate Prevention**: The upsert logic in `save_or_update` ensures that re-crawling the same content updates existing documents rather than creating duplicates.

## Frequently Asked Questions

### How does MediaCrawler handle MongoDB authentication?

MediaCrawler supports authenticated connections through the `user` and `password` fields in `mongodb_config`. When these values are populated in [`config/db_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/db_config.py), the `MongoDBConnection` class constructs a MongoDB connection URL with credentials. If left empty, it connects to the MongoDB instance without authentication, suitable for local development environments.

### What is the difference between MongoDBStoreBase and platform-specific implementations?

`MongoDBStoreBase` is the generic abstraction layer providing raw CRUD operations against MongoDB collections. Platform-specific implementations like `XhsMongoStoreImplement` in [`store/xhs/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/_store_impl.py) instantiate the base class with a specific prefix (e.g., `"xhs"`) and provide convenience methods like `store_content` that automatically set the correct `collection_suffix` and normalize data before calling `save_or_update`.

### How does MediaCrawler prevent duplicate entries when re-crawling the same content?

The `save_or_update` method in `MongoDBStoreBase` uses MongoDB's upsert functionality. You provide a `query` dictionary (typically containing unique identifiers like `note_id` or `comment_id`) and a `data` dictionary. If a document matching the query exists, it updates that document; otherwise, it inserts a new one. This ensures that successive crawls synchronize the database state without creating duplicate records.

### Can I extend MediaCrawler to store data from new platforms in MongoDB?

Yes. To add MongoDB support for a new platform, create a new file in the `store/` directory (e.g., [`store/new_platform/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/new_platform/_store_impl.py)), define a class that instantiates `MongoDBStoreBase` with a unique `collection_prefix`, and implement methods that call `save_or_update` with appropriate collection suffixes for your data types (contents, users, etc.). The architecture is designed to be extensible without modifying the core storage logic.