How to Store Crawled Data in CSV with MediaCrawler: Complete Configuration Guide

Set SAVE_DATA_OPTION = "csv" in config/base_config.py and run the crawler to automatically export content, comments, and creator data to CSV files using the AsyncFileWriter class.

MediaCrawler is a powerful open-source scraping framework that supports multiple Chinese social media platforms including Zhihu, Xiaohongshu, Weibo, and Tieba. When you need to store crawled data in CSV with MediaCrawler, the framework provides a flexible storage layer that abstracts file creation, header management, and concurrent writes through the AsyncFileWriter implementation.

Understanding the CSV Storage Architecture

MediaCrawler uses a factory-based storage pattern that decouples the crawler logic from output formats. The architecture routes all data through three main components when CSV mode is enabled.

Configuration Layer (base_config.py)

The global configuration in config/base_config.py defines the SAVE_DATA_OPTION parameter that controls output format. Setting this value to "csv" instructs the framework to instantiate CSV-specific store implementations. You can also override this at runtime using the --save-data-option csv CLI flag.

Store Factory Pattern (store/<platform>/__init__.py)

Each platform implements a factory class in its respective store/<platform>/__init__.py file. This factory maps the SAVE_DATA_OPTION string to concrete store classes. When "csv" is selected, the factory returns a platform-specific implementation such as ZhihuCsvStoreImplement (defined in store/zhihu/_store_impl.py) or equivalent classes for other platforms.

AsyncFileWriter Implementation (tools/async_file_writer.py)

The actual file operations are handled by the AsyncFileWriter class in tools/async_file_writer.py. This class manages asynchronous file access with locks, preventing write collisions during concurrent crawling. It handles path generation, directory creation, and CSV formatting through the csv.DictWriter standard library module.

Step-by-Step Configuration

Follow these steps to configure MediaCrawler for CSV output:

1. Set the storage option – Open config/base_config.py and set SAVE_DATA_OPTION = "csv".

2. Configure the output directory – Optionally set SAVE_DATA_PATH to a custom directory (e.g., /tmp/media). If left empty, files default to data/<platform>/csv/.

3. Run the crawler – Execute the main script. The store factory automatically instantiates the appropriate CSV implementation based on your configuration.

4. Locate your files – Find generated CSV files under your configured path using the pattern <platform>/csv/<crawler_type>_contents_YYYY-MM-DD.csv.

How CSV Files Are Generated

When AsyncFileWriter.write_to_csv is invoked (typically at lines 46-55 of tools/async_file_writer.py), the following process occurs:

• Path construction: The _get_file_path method builds the final file path using the platform name, item type (content, comment, or creator), and the crawler type stored in var.py.

• Header writing: The writer creates a csv.DictWriter using the keys of the first dictionary item as column headers. It writes the header only if the file is new or empty.

• Row appending: Subsequent calls append rows to the existing file using writer.writerow(item).

• Concurrency safety: An async lock (async with self.lock) ensures thread-safe writes when multiple coroutines attempt to write simultaneously.

The resulting files use UTF-8 BOM encoding (utf-8-sig) to ensure proper character display in Excel and other tools.

Code Examples

Configure CSV output in config/base_config.py:

# config/base_config.py

SAVE_DATA_OPTION = "csv"          # Enable CSV mode

SAVE_DATA_PATH = "/tmp/media"     # Optional: custom output directory

Run the crawler with CSV storage:

python main.py --platform zhihu --save-data-option csv

Internal factory selection (from store/zhihu/__init__.py):

# store/zhihu/__init__.py

STORE_MAP = {
    "csv": ZhihuCsvStoreImplement,
    "json": ZhihuJsonStoreImplement,
    # other options...

}

# The factory creates the appropriate store

store = ZhihuStoreFactory.create_store(save_option="csv")
await store.store_content(content_item)   # Writes to CSV

await store.store_comment(comment_item)   # Writes to CSV

await store.store_creator(creator_item)   # Writes to CSV

Core writing logic from tools/async_file_writer.py:

async def write_to_csv(self, item: Dict, item_type: str):
    file_path = self._get_file_path('csv', item_type)
    async with self.lock:
        file_exists = os.path.exists(file_path)
        async with aiofiles.open(file_path, 'a', newline='', encoding='utf-8-sig') as f:
            writer = csv.DictWriter(f, fieldnames=item.keys())
            if not file_exists or await f.tell() == 0:
                await writer.writeheader()
            await writer.writerow(item)

Platform-Specific Store Implementations

Each platform implements its own CSV store class following the same pattern:

• Zhihu: ZhihuCsvStoreImplement in store/zhihu/_store_impl.py
• Xiaohongshu: XhsCsvStoreImplement in store/xhs/_store_impl.py
• Weibo: WeiboCsvStoreImplement in store/weibo/_store_impl.py
• Tieba: TiebaCsvStoreImplement in store/tieba/_store_impl.py

Each implementation provides store_content(), store_comment(), and store_creator() methods that forward data to the shared AsyncFileWriter.

Summary

• Set SAVE_DATA_OPTION = "csv" in config/base_config.py to enable CSV export for any supported platform.
• The AsyncFileWriter class in tools/async_file_writer.py handles all file operations with async locking for thread safety.
• Platform-specific factories in store/<platform>/__init__.py route data to the appropriate CSV implementation.
• CSV files are created with UTF-8 BOM encoding and automatically include headers based on dictionary keys.
• Output paths follow the pattern data/<platform>/csv/<crawler_type>_<item_type>_YYYY-MM-DD.csv unless overridden by SAVE_DATA_PATH.

Frequently Asked Questions

How do I change the default CSV output directory?

Set the SAVE_DATA_PATH variable in config/base_config.py to an absolute path. If this variable is empty or unset, MediaCrawler defaults to data/<platform>/csv/ relative to the project root.

Can I export to CSV and JSON simultaneously?

MediaCrawler's SAVE_DATA_OPTION accepts a single value, so you cannot write to multiple formats simultaneously in one process. To get both formats, run the crawler twice with different SAVE_DATA_OPTION values, or implement a custom store class that calls multiple writers.

What encoding does MediaCrawler use for CSV files?

The AsyncFileWriter uses utf-8-sig encoding (UTF-8 with BOM) when opening files. This ensures that Chinese characters and other Unicode text display correctly when opening files in Microsoft Excel or other spreadsheet applications.

How does MediaCrawler handle concurrent writes to CSV files?

The AsyncFileWriter uses an asyncio.Lock (stored as self.lock) to synchronize access to the file handle. When write_to_csv is called, it acquires the lock before opening the file with aiofiles, ensuring that only one coroutine writes at a time and preventing data corruption during high-concurrency crawling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}