MediaCrawler Data Storage Options Explained: CSV, JSON, SQLite, MongoDB, and Excel

MediaCrawler supports seven distinct storage backends—CSV, JSON, JSONL, SQLite, SQLAlchemy-compatible databases, MongoDB, and Excel—through a unified AbstractStore interface that decouples crawling logic from data persistence.

The MediaCrawler project provides a modular architecture that separates content extraction from data persistence. By implementing the abstract AbstractStore class defined in base/base_crawler.py, developers can switch between flat files, relational databases, and NoSQL stores without modifying core crawler logic. This guide explains each storage option, their implementation details, and how to configure them for platforms like Zhihu, Xiaohongshu (XHS), Weibo, and Tieba.

The AbstractStore Interface

At the heart of MediaCrawler's persistence layer lies the AbstractStore class in base/base_crawler.py (lines 86–100). This abstract base class defines three asynchronous methods that all concrete implementations must provide:

• store_content(content_item)
• store_comment(comment_item)
• store_creator(creator_item)

Each platform-specific store inherits from this interface and handles the translation of Python dictionaries into the target format. The crawler code remains agnostic to the underlying storage mechanism—it simply calls the appropriate method on the injected store instance.

Supported Storage Backends

MediaCrawler provides concrete implementations for multiple persistence strategies, each optimized for different use cases from quick prototyping to production analytics.

CSV Files

The CSV implementation uses AsyncFileWriter (tools/async_file_writer.py) to append rows asynchronously to timestamped files. Platform-specific classes like ZhihuCsvStoreImplement, XhsCsvStoreImplement, and WeiboCsvStoreImplement (located in store/<platform>/_store_impl.py) handle schema normalization before writing to data/<platform>/<platform>_<crawler_type>_<timestamp>.csv.

SQLite and Relational Databases

For structured storage, MediaCrawler offers ZhihuDbStoreImplement, ZhihuSqliteStoreImplement, and similar variants for other platforms. These classes utilize SQLAlchemy async sessions defined in database/db_session.py to perform upsert operations. The SQLite variant inherits from the generic DB implementation, allowing seamless migration from file-based to server-based databases by changing only the connection string.

JSON and JSONL

JSON storage writes single objects per file via AsyncFileWriter.write_single_item_to_json, while JSONL (newline-delimited JSON) appends records continuously using AsyncFileWriter.write_to_jsonl. These formats provide human-readable output suitable for line-oriented processing tools like jq or_log aggregation systems.

MongoDB

The MongoDB implementation wraps a thin helper class MongoDBStoreBase (database/mongodb_store_base.py). Platform-specific implementations like ZhihuMongoStoreImplement perform save-or-update operations on collections prefixed by the platform name (e.g., zhihu_contents). This approach leverages Motor for async I/O while maintaining atomic upserts for duplicate content handling.

Excel Workbooks

ExcelStoreBase (store/excel_store_base.py) operates as a singleton providing rich, formatted Excel exports. Unlike other stores that write incrementally, this implementation accumulates data in memory across separate sheets for Contents, Comments, Creators, and optionally Contacts/Dynamics. Platform-specific wrappers (ZhihuExcelStoreImplement, etc.) return the shared singleton instance, allowing aggregated multi-platform exports. Data flushes to data/<platform>/ only when ExcelStoreBase.flush_all() is called.

How the Storage Abstraction Works

The architecture follows a clean dependency injection pattern that enables three key capabilities:

1. Pluggable Storage: Swap CSV for MongoDB by changing only the store class instantiation—no crawler code changes required.
2. Unified Testing: The repository includes tests/test_store_factory.py, which iterates over all store implementations to verify interface conformance.
3. Centralized Configuration: Store constructors accept platform and crawler_type parameters (e.g., platform="zhihu", crawler_type="search"), automatically determining output paths via config.SAVE_DATA_PATH.

The data flow remains consistent across all backends: the crawler produces a standardized dictionary (containing fields like note_id, title, author, add_ts), and the store implementation handles format-specific serialization.

Practical Implementation Examples

The following code snippets demonstrate identical data persistence across different backends using the Zhihu platform implementations.

Writing to CSV

from store.zhihu._store_impl import ZhihuCsvStoreImplement
import asyncio

async def persist_csv():
    store = ZhihuCsvStoreImplement()
    await store.store_content({
        "note_id": "101",
        "title": "MediaCrawler Guide",
        "author": "developer",
        "add_ts": "2023-11-12T15:00:00Z",
    })
    await store.store_comment({
        "comment_id": "c1",
        "note_id": "101",
        "text": "Helpful tutorial",
    })

if __name__ == "__main__":
    asyncio.run(persist_csv())

Persisting to SQLite

from store.zhihu._store_impl import ZhihuDbStoreImplement
import asyncio

async def persist_sqlite():
    store = ZhihuDbStoreImplement()
    await store.store_content({
        "content_id": "c123",
        "title": "Database Example",
        "author": "analyst",
    })
    await store.store_comment({
        "comment_id": "cm45",
        "content_id": "c123",
        "text": "Structured storage",
    })

if __name__ == "__main__":
    asyncio.run(persist_sqlite())

Storing in MongoDB

from store.zhihu._store_impl import ZhihuMongoStoreImplement
import asyncio

async def persist_mongodb():
    store = ZhihuMongoStoreImplement()
    await store.store_content({
        "note_id": "m001",
        "title": "NoSQL Document",
        "author": "data_engineer",
    })
    await store.store_comment({
        "comment_id": "mc01",
        "note_id": "m001",
        "text": "Document-oriented storage",
    })

if __name__ == "__main__":
    asyncio.run(persist_mongodb())

Generating Excel Reports

from store.zhihu._store_impl import ZhihuExcelStoreImplement
from store.excel_store_base import ExcelStoreBase
import asyncio

async def persist_excel():
    store = ZhihuExcelStoreImplement()  # Returns shared singleton

    await store.store_content({
        "note_id": "e100",
        "title": "Excel Report",
        "author": "manager",
    })
    await store.store_comment({
        "comment_id": "ec10",
        "note_id": "e100",
        "text": "Formatted output",
    })
    # Flush once at program exit or batch completion

    ExcelStoreBase.flush_all()

if __name__ == "__main__":
    asyncio.run(persist_excel())

Summary

MediaCrawler's storage architecture provides enterprise flexibility through these key capabilities:

• Seven Backend Options: Choose from CSV, JSON, JSONL, SQLite, SQLAlchemy databases, MongoDB, or Excel based on your analysis needs.
• Interface Consistency: All implementations extend AbstractStore from base/base_crawler.py, ensuring identical method signatures for store_content, store_comment, and store_creator.
• Async Performance: File operations use AsyncFileWriter while database operations leverage SQLAlchemy async sessions and Motor for non-blocking I/O.
• Singleton Pattern: Excel exports utilize a shared ExcelStoreBase instance to aggregate multi-batch data into formatted workbooks.

Frequently Asked Questions

How do I switch between storage backends without changing crawler code?

Instantiate a different concrete store class while keeping the variable name consistent. Since all stores implement AbstractStore, the calling code remains identical. For example, replace ZhihuCsvStoreImplement() with ZhihuMongoStoreImplement()—the store_content() method signature remains unchanged.

Which storage option is best for large-scale data collection?

MongoDB or SQLAlchemy-compatible databases handle large volumes most efficiently due to indexing and duplicate-checking capabilities. CSV and JSONL work well for smaller datasets or pipeline processing, while Excel suits reporting and manual analysis rather than high-throughput storage.

Does MediaCrawler support concurrent writes to the same storage?

Yes. The AsyncFileWriter utility handles concurrent CSV/JSON/JSONL writes safely, while database implementations use SQLAlchemy's async session management and MongoDB's atomic upserts. The Excel singleton accumulates data in memory and flushes atomically, preventing file corruption during concurrent access.

Where are the output files saved when using file-based storage?

All file-based stores write to the directory specified by config.SAVE_DATA_PATH, typically structured as data/<platform>/<platform>_<crawler_type>_<timestamp>.<ext>. Database and MongoDB stores use connection strings defined in the configuration, allowing remote or local persistence independent of the filesystem.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}