How to Store Crawled Data in MongoDB with MediaCrawler: A Complete Guide

MediaCrawler provides a reusable MongoDB abstraction layer that handles async connections via Motor, automatic upserts, and platform-specific collections, allowing you to persist crawled data by configuring mongodb_config in config/db_config.py and using the MongoDBStoreBase class or platform-specific store implementations.

MediaCrawler is an open-source multi-platform content crawler that ships with built-in MongoDB persistence. This guide explains how to store crawled data in MongoDB with MediaCrawler using the async storage layer implemented in database/mongodb_store_base.py and platform-specific store classes that handle data normalization and collection management.

Configure MongoDB Connection Parameters

All MongoDB connection settings are centralized in config/db_config.py. The mongodb_config dictionary defines the parameters for the async Motor client to establish a connection.

The configuration supports optional authentication. When user and password are provided, the MongoDBConnection singleton automatically constructs a connection URL with credentials; otherwise, it connects without authentication.

# config/db_config.py

mongodb_config = {
    "host": "localhost",
    "port": 27017,
    "user": "",           # Optional: leave empty for no auth

    "password": "",       # Optional: leave empty for no auth

    "db_name": "media_crawler",
}

The MongoDBConnection class reads this dictionary and creates an async Motor client instance that is shared across the application, ensuring efficient connection pooling and non-blocking I/O operations.

Core MongoDB Storage Abstraction

The database/mongodb_store_base.py file defines MongoDBStoreBase, the foundational class for all MongoDB operations in MediaCrawler. This class encapsulates the Motor client interaction and provides CRUD operations with automatic BSON-to-dict conversion.

Key methods include:

• save_or_update – Performs an upsert operation using a query filter and data dictionary, creating or updating documents atomically to prevent duplicates.
• find_one – Retrieves a single document matching the specified filter.
• find_many – Returns multiple documents with optional limit and skip parameters.
• create_index – Builds indexes on specified fields for optimized queries.

The constructor requires a collection_prefix parameter (e.g., "xhs" for Xiaohongshu, "zhihu" for Zhihu) that prefixes all collection names, ensuring data isolation between platforms in the same database.

Platform-Specific Store Implementations

Each supported platform implements a concrete store class that instantiates MongoDBStoreBase with the appropriate prefix and provides semantic methods for different content types.

For example, in store/xhs/_store_impl.py, the XhsMongoStoreImplement class initializes the base store with collection_prefix="xhs" and exposes methods like store_content, store_comment, and store_creator. These methods internally call save_or_update with specific collection suffixes:

• Contents: collection_suffix="contents" for posts and notes
• Comments: collection_suffix="comments" for user comments
• Creators: collection_suffix="creators" for user profile data

Similarly, store/zhihu/_store_impl.py contains ZhihuMongoStoreImplement using the prefix "zhihu".

Storing Crawled Data: Code Examples

Using the Base Class Directly

For custom storage logic, instantiate MongoDBStoreBase and call the upsert method directly:

from database.mongodb_store_base import MongoDBStoreBase

# Initialize store for Xiaohongshu data

mongo_store = MongoDBStoreBase(collection_prefix="xhs")

# Prepare crawled note data

note_data = {
    "note_id": "abc123",
    "user_id": "u001",
    "title": "My travel vlog",
    "desc": "A short description",
    "liked_count": "150",
    "image_list": ["https://example.com/img1.jpg"],
}

# Upsert the document (creates or updates)

await mongo_store.save_or_update(
    collection_suffix="contents",
    query={"note_id": note_data["note_id"]},
    data=note_data,
)

Using the High-Level Platform Store

For standard platforms, use the provided implementation class to handle boilerplate:

from store.xhs._store_impl import XhsMongoStoreImplement

xhs_store = XhsMongoStoreImplement()

# Store a note - automatically handles collection selection and upsert logic

await xhs_store.store_content(note_data)

# Store a comment

comment_data = {
    "comment_id": "cmt001",
    "note_id": "abc123",
    "text": "Nice post!"
}
await xhs_store.store_comment(comment_data)

Both approaches ensure that repeated crawls update existing records rather than creating duplicates, thanks to the query-based upsert mechanism in save_or_update.

Querying Stored Data

Retrieve data using the base class methods, which return native Python dictionaries:

# Fetch a specific note by ID

note = await mongo_store.find_one("contents", {"note_id": "abc123"})
print(note["title"])  # Output: My travel vlog

# Fetch the latest 100 comments

comments = await mongo_store.find_many("comments", {}, limit=100)

The async methods return Motor cursors that are automatically converted to lists of dictionaries, making integration with the rest of your Python application seamless.

Summary

• Configuration: Define connection parameters in config/db_config.py under mongodb_config; the MongoDBConnection singleton handles async Motor client initialization with optional authentication.
• Storage Layer: MongoDBStoreBase in database/mongodb_store_base.py provides the core CRUD operations including save_or_update for atomic upserts.
• Platform Isolation: Use collection_prefix (e.g., "xhs", "zhihu") to namespace collections, and collection_suffix to separate content types (contents, comments, creators).
• Implementation Pattern: Platform stores like XhsMongoStoreImplement wrap the base class and provide semantic methods for different data models.
• Duplicate Prevention: The upsert logic in save_or_update ensures that re-crawling the same content updates existing documents rather than creating duplicates.

Frequently Asked Questions

How does MediaCrawler handle MongoDB authentication?

MediaCrawler supports authenticated connections through the user and password fields in mongodb_config. When these values are populated in config/db_config.py, the MongoDBConnection class constructs a MongoDB connection URL with credentials. If left empty, it connects to the MongoDB instance without authentication, suitable for local development environments.

What is the difference between MongoDBStoreBase and platform-specific implementations?

MongoDBStoreBase is the generic abstraction layer providing raw CRUD operations against MongoDB collections. Platform-specific implementations like XhsMongoStoreImplement in store/xhs/_store_impl.py instantiate the base class with a specific prefix (e.g., "xhs") and provide convenience methods like store_content that automatically set the correct collection_suffix and normalize data before calling save_or_update.

How does MediaCrawler prevent duplicate entries when re-crawling the same content?

The save_or_update method in MongoDBStoreBase uses MongoDB's upsert functionality. You provide a query dictionary (typically containing unique identifiers like note_id or comment_id) and a data dictionary. If a document matching the query exists, it updates that document; otherwise, it inserts a new one. This ensures that successive crawls synchronize the database state without creating duplicate records.

Can I extend MediaCrawler to store data from new platforms in MongoDB?

Yes. To add MongoDB support for a new platform, create a new file in the store/ directory (e.g., store/new_platform/_store_impl.py), define a class that instantiates MongoDBStoreBase with a unique collection_prefix, and implement methods that call save_or_update with appropriate collection suffixes for your data types (contents, users, etc.). The architecture is designed to be extensible without modifying the core storage logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}