# MediaCrawler Data Storage Options: CSV, JSON, MongoDB, Excel, and Relational Databases > Explore MediaCrawler data storage options including CSV, JSON, MongoDB, Excel, and relational databases. Choose the best format for your project. - Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler) - Tags: architecture - Published: 2026-06-27 --- **The MediaCrawler open-source project supports seven distinct data storage options including CSV, JSON, JSONL, SQLite, MongoDB, Excel, and SQLAlchemy-compatible relational databases, all unified behind a single `AbstractStore` interface.** MediaCrawler implements a clean separation between crawling logic and data persistence through its store abstraction layer. This architecture allows developers to swap between flat files, document stores, and relational databases without modifying the core crawler code. All storage implementations reside in platform-specific modules under `store//` and inherit from the `AbstractStore` base class defined in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py). ## The AbstractStore Interface The persistence layer centers on **`AbstractStore`** (lines 86-100 of [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py)), which declares three core async methods: - `store_content(item: dict)` – Persists main content items (posts, articles, notes) - `store_comment(item: dict)` – Persists comment data - `store_creator(item: dict)` – Persists author/creator metadata Each concrete implementation handles serialization, connection management, and error handling internally. The crawler code receives the store instance via dependency injection and calls these methods uniformly, regardless of whether the backend writes to a CSV file or a MongoDB collection. ## Available Storage Backends ### CSV Files **CSV storage** provides lightweight, human-readable output suitable for data analysis and sharing. Implementations like `ZhihuCsvStoreImplement`, `XhsCsvStoreImplement`, and `WeiboCsvStoreImplement` reside in `store//_store_impl.py`. These classes utilize **`AsyncFileWriter`** (from [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py)) to append rows asynchronously. Each platform writes to a timestamped file in `data//` (configured via `config.SAVE_DATA_PATH`). ### JSON and JSONL For structured data exports, MediaCrawler offers both **JSON** and **JSONL** (newline-delimited JSON) formats: - **`ZhihuJsonStoreImplement`** – Writes individual JSON objects per item via `AsyncFileWriter.write_single_item_to_json` - **`ZhihuJsonlStoreImplement`** – Appends to a line-delimited stream using `AsyncFileWriter.write_to_jsonl` JSONL is preferred for large crawls since it supports append operations without parsing entire files, while single JSON files provide complete, valid JSON documents for smaller datasets. ### SQLite and Relational Databases **Relational database support** leverages SQLAlchemy async sessions defined in [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py). Concrete implementations include `ZhihuDbStoreImplement`, `ZhihuSqliteStoreImplement`, `TieBaDbStoreImplement`, and `WeiboDbStoreImplement`. These stores perform upsert operations to prevent duplicate entries. The SQLite variant inherits from the generic database implementation but targets `sqlite+aiosqlite` connections, making it suitable for local development without external database infrastructure. ### MongoDB **MongoDB integration** uses `ZhihuMongoStoreImplement` and similar platform-specific wrappers that inherit from **`MongoDBStoreBase`** ([`database/mongodb_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/mongodb_store_base.py)). This thin async wrapper handles connection pooling and performs save-or-update operations on collections prefixed by the platform name (e.g., `zhihu_contents`). The implementation uses Motor for async MongoDB operations, ensuring non-blocking I/O during high-volume crawls. ### Excel Workbooks **Excel export** provides rich, formatted spreadsheets through the singleton class **`ExcelStoreBase`** ([`store/excel_store_base.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/excel_store_base.py)). Platform-specific wrappers like `ZhihuExcelStoreImplement` return a shared singleton instance, ensuring all data aggregates into a single workbook. The Excel implementation creates separate sheets for **Contents**, **Comments**, and **Creators**, with auto-sized columns and optional sheets for **Contacts** and **Dynamics**. Call `ExcelStoreBase.flush_all()` at program termination to write the `.xlsx` file to `data//`. ## Practical Implementation Examples ### Writing to CSV ```python from store.zhihu._store_impl import ZhihuCsvStoreImplement import asyncio async def main(): store = ZhihuCsvStoreImplement() await store.store_content({ "note_id": "101", "title": "Demo article", "author": "bob", "add_ts": "2023-11-12T15:00:00Z", }) await store.store_comment({ "comment_id": "c1", "note_id": "101", "text": "Great post!", }) if __name__ == "__main__": asyncio.run(main()) ``` ### Persisting to MongoDB ```python from store.zhihu._store_impl import ZhihuMongoStoreImplement import asyncio async def main(): store = ZhihuMongoStoreImplement() await store.store_content({ "note_id": "m001", "title": "Mongo demo", "author": "dave", }) await store.store_comment({ "comment_id": "mc01", "note_id": "m001", "text": "Interesting", }) if __name__ == "__main__": asyncio.run(main()) ``` ### Generating Excel Reports ```python from store.zhihu._store_impl import ZhihuExcelStoreImplement from store.excel_store_base import ExcelStoreBase import asyncio async def main(): store = ZhihuExcelStoreImplement() # Returns shared singleton await store.store_content({ "note_id": "e100", "title": "Excel demo", "author": "eve", }) await store.store_comment({ "comment_id": "ec10", "note_id": "e100", "text": "Lovely sheet", }) # Flush once at program exit ExcelStoreBase.flush_all() if __name__ == "__main__": asyncio.run(main()) ``` ### Database Storage with SQLite ```python from store.zhihu._store_impl import ZhihuDbStoreImplement import asyncio async def main(): store = ZhihuDbStoreImplement() await store.store_content({ "content_id": "c123", "title": "SQL example", "author": "carol", }) await store.store_comment({ "comment_id": "cm45", "content_id": "c123", "text": "Nice!", }) if __name__ == "__main__": asyncio.run(main()) ``` ## Architecture Benefits This pluggable design delivers significant advantages for developers: - **Unified Testing** – The repository includes [`tests/test_store_factory.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_store_factory.py), which iterates over all store implementations to verify conformance to the `AbstractStore` contract - **Configuration-Driven** – Store constructors accept `platform` and `crawler_type` parameters, automatically determining output paths without hardcoded logic - **Zero Core Changes** – Switching from CSV to MongoDB requires changing only the imported class; crawler logic remains identical ## Summary MediaCrawler provides comprehensive data storage options through a modular architecture: - **Flat files** – CSV, JSON, and JSONL for simple, portable data exports - **Relational databases** – SQLAlchemy async support for SQLite and other SQL backends - **NoSQL** – MongoDB via Motor for document-oriented storage - **Rich reports** – Excel workbooks with formatted sheets and auto-sizing All implementations inherit from `AbstractStore` in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py), ensuring consistent APIs across platforms and simplifying storage backend swaps. ## Frequently Asked Questions ### How do I switch between storage options in MediaCrawler? Change the imported store class in your crawler script. For example, replace `ZhihuCsvStoreImplement` with `ZhihuMongoStoreImplement` while keeping the `store_content()` and `store_comment()` calls identical. The `AbstractStore` interface ensures all methods remain compatible. ### Does MediaCrawler support multiple storage backends simultaneously? Yes. Instantiate multiple store classes and call their methods within the same event loop. Each store operates independently, allowing you to write to CSV, MongoDB, and Excel concurrently from the same crawled data. ### Where are the Excel files saved when using ExcelStoreBase? Excel files are written to the path defined by `config.SAVE_DATA_PATH` (typically `data//`). You must call `ExcelStoreBase.flush_all()` at program termination to persist the buffered workbook data to disk. ### What is the difference between ZhihuDbStoreImplement and ZhihuSqliteStoreImplement? `ZhihuDbStoreImplement` targets SQLAlchemy async connections for general relational databases, while `ZhihuSqliteStoreImplement` specializes in SQLite file-based storage. Both use the same SQLAlchemy session management from [`database/db_session.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db_session.py), but the SQLite variant configures the connection string for `sqlite+aiosqlite` automatically.