MediaCrawler Data Storage Options: CSV, JSON, MongoDB, Excel, and Relational Databases

The MediaCrawler open-source project supports seven distinct data storage options including CSV, JSON, JSONL, SQLite, MongoDB, Excel, and SQLAlchemy-compatible relational databases, all unified behind a single AbstractStore interface.

MediaCrawler implements a clean separation between crawling logic and data persistence through its store abstraction layer. This architecture allows developers to swap between flat files, document stores, and relational databases without modifying the core crawler code. All storage implementations reside in platform-specific modules under store/<platform>/ and inherit from the AbstractStore base class defined in base/base_crawler.py.

The AbstractStore Interface

The persistence layer centers on AbstractStore (lines 86-100 of base/base_crawler.py), which declares three core async methods:

• store_content(item: dict) – Persists main content items (posts, articles, notes)
• store_comment(item: dict) – Persists comment data
• store_creator(item: dict) – Persists author/creator metadata

Each concrete implementation handles serialization, connection management, and error handling internally. The crawler code receives the store instance via dependency injection and calls these methods uniformly, regardless of whether the backend writes to a CSV file or a MongoDB collection.

Available Storage Backends

CSV Files

CSV storage provides lightweight, human-readable output suitable for data analysis and sharing. Implementations like ZhihuCsvStoreImplement, XhsCsvStoreImplement, and WeiboCsvStoreImplement reside in store/<platform>/_store_impl.py.

These classes utilize AsyncFileWriter (from tools/async_file_writer.py) to append rows asynchronously. Each platform writes to a timestamped file in data/<platform>/ (configured via config.SAVE_DATA_PATH).

JSON and JSONL

For structured data exports, MediaCrawler offers both JSON and JSONL (newline-delimited JSON) formats:

• ZhihuJsonStoreImplement – Writes individual JSON objects per item via AsyncFileWriter.write_single_item_to_json
• ZhihuJsonlStoreImplement – Appends to a line-delimited stream using AsyncFileWriter.write_to_jsonl

JSONL is preferred for large crawls since it supports append operations without parsing entire files, while single JSON files provide complete, valid JSON documents for smaller datasets.

SQLite and Relational Databases

Relational database support leverages SQLAlchemy async sessions defined in database/db_session.py. Concrete implementations include ZhihuDbStoreImplement, ZhihuSqliteStoreImplement, TieBaDbStoreImplement, and WeiboDbStoreImplement.

These stores perform upsert operations to prevent duplicate entries. The SQLite variant inherits from the generic database implementation but targets sqlite+aiosqlite connections, making it suitable for local development without external database infrastructure.

MongoDB

MongoDB integration uses ZhihuMongoStoreImplement and similar platform-specific wrappers that inherit from MongoDBStoreBase (database/mongodb_store_base.py). This thin async wrapper handles connection pooling and performs save-or-update operations on collections prefixed by the platform name (e.g., zhihu_contents).

The implementation uses Motor for async MongoDB operations, ensuring non-blocking I/O during high-volume crawls.

Excel Workbooks

Excel export provides rich, formatted spreadsheets through the singleton class ExcelStoreBase (store/excel_store_base.py). Platform-specific wrappers like ZhihuExcelStoreImplement return a shared singleton instance, ensuring all data aggregates into a single workbook.

The Excel implementation creates separate sheets for Contents, Comments, and Creators, with auto-sized columns and optional sheets for Contacts and Dynamics. Call ExcelStoreBase.flush_all() at program termination to write the .xlsx file to data/<platform>/.

Practical Implementation Examples

Writing to CSV

from store.zhihu._store_impl import ZhihuCsvStoreImplement
import asyncio

async def main():
    store = ZhihuCsvStoreImplement()
    await store.store_content({
        "note_id": "101",
        "title": "Demo article",
        "author": "bob",
        "add_ts": "2023-11-12T15:00:00Z",
    })
    await store.store_comment({
        "comment_id": "c1",
        "note_id": "101",
        "text": "Great post!",
    })

if __name__ == "__main__":
    asyncio.run(main())

Persisting to MongoDB

from store.zhihu._store_impl import ZhihuMongoStoreImplement
import asyncio

async def main():
    store = ZhihuMongoStoreImplement()
    await store.store_content({
        "note_id": "m001",
        "title": "Mongo demo",
        "author": "dave",
    })
    await store.store_comment({
        "comment_id": "mc01",
        "note_id": "m001",
        "text": "Interesting",
    })

if __name__ == "__main__":
    asyncio.run(main())

Generating Excel Reports

from store.zhihu._store_impl import ZhihuExcelStoreImplement
from store.excel_store_base import ExcelStoreBase
import asyncio

async def main():
    store = ZhihuExcelStoreImplement()  # Returns shared singleton

    await store.store_content({
        "note_id": "e100",
        "title": "Excel demo",
        "author": "eve",
    })
    await store.store_comment({
        "comment_id": "ec10",
        "note_id": "e100",
        "text": "Lovely sheet",
    })
    # Flush once at program exit

    ExcelStoreBase.flush_all()

if __name__ == "__main__":
    asyncio.run(main())

Database Storage with SQLite

from store.zhihu._store_impl import ZhihuDbStoreImplement
import asyncio

async def main():
    store = ZhihuDbStoreImplement()
    await store.store_content({
        "content_id": "c123",
        "title": "SQL example",
        "author": "carol",
    })
    await store.store_comment({
        "comment_id": "cm45",
        "content_id": "c123",
        "text": "Nice!",
    })

if __name__ == "__main__":
    asyncio.run(main())

Architecture Benefits

This pluggable design delivers significant advantages for developers:

• Unified Testing – The repository includes tests/test_store_factory.py, which iterates over all store implementations to verify conformance to the AbstractStore contract
• Configuration-Driven – Store constructors accept platform and crawler_type parameters, automatically determining output paths without hardcoded logic
• Zero Core Changes – Switching from CSV to MongoDB requires changing only the imported class; crawler logic remains identical

Summary

MediaCrawler provides comprehensive data storage options through a modular architecture:

• Flat files – CSV, JSON, and JSONL for simple, portable data exports
• Relational databases – SQLAlchemy async support for SQLite and other SQL backends
• NoSQL – MongoDB via Motor for document-oriented storage
• Rich reports – Excel workbooks with formatted sheets and auto-sizing

All implementations inherit from AbstractStore in base/base_crawler.py, ensuring consistent APIs across platforms and simplifying storage backend swaps.

Frequently Asked Questions

How do I switch between storage options in MediaCrawler?

Change the imported store class in your crawler script. For example, replace ZhihuCsvStoreImplement with ZhihuMongoStoreImplement while keeping the store_content() and store_comment() calls identical. The AbstractStore interface ensures all methods remain compatible.

Does MediaCrawler support multiple storage backends simultaneously?

Yes. Instantiate multiple store classes and call their methods within the same event loop. Each store operates independently, allowing you to write to CSV, MongoDB, and Excel concurrently from the same crawled data.

Where are the Excel files saved when using ExcelStoreBase?

Excel files are written to the path defined by config.SAVE_DATA_PATH (typically data/<platform>/). You must call ExcelStoreBase.flush_all() at program termination to persist the buffered workbook data to disk.

What is the difference between ZhihuDbStoreImplement and ZhihuSqliteStoreImplement?

ZhihuDbStoreImplement targets SQLAlchemy async connections for general relational databases, while ZhihuSqliteStoreImplement specializes in SQLite file-based storage. Both use the same SQLAlchemy session management from database/db_session.py, but the SQLite variant configures the connection string for sqlite+aiosqlite automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}