# How to Store Crawled Data in CSV with MediaCrawler: Complete Configuration Guide

> Learn how to store crawled data in CSV with MediaCrawler. Configure your settings and export content comments and creator data effortlessly. Get started today.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**Set `SAVE_DATA_OPTION = "csv"` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and run the crawler to automatically export content, comments, and creator data to CSV files using the `AsyncFileWriter` class.**

MediaCrawler is a powerful open-source scraping framework that supports multiple Chinese social media platforms including Zhihu, Xiaohongshu, Weibo, and Tieba. When you need to store crawled data in CSV with MediaCrawler, the framework provides a flexible storage layer that abstracts file creation, header management, and concurrent writes through the `AsyncFileWriter` implementation.

## Understanding the CSV Storage Architecture

MediaCrawler uses a factory-based storage pattern that decouples the crawler logic from output formats. The architecture routes all data through three main components when CSV mode is enabled.

### Configuration Layer ([`base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base_config.py))

The global configuration in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) defines the `SAVE_DATA_OPTION` parameter that controls output format. Setting this value to `"csv"` instructs the framework to instantiate CSV-specific store implementations. You can also override this at runtime using the `--save-data-option csv` CLI flag.

### Store Factory Pattern (`store/<platform>/__init__.py`)

Each platform implements a factory class in its respective `store/<platform>/__init__.py` file. This factory maps the `SAVE_DATA_OPTION` string to concrete store classes. When `"csv"` is selected, the factory returns a platform-specific implementation such as `ZhihuCsvStoreImplement` (defined in [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)) or equivalent classes for other platforms.

### AsyncFileWriter Implementation ([`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py))

The actual file operations are handled by the `AsyncFileWriter` class in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py). This class manages asynchronous file access with locks, preventing write collisions during concurrent crawling. It handles path generation, directory creation, and CSV formatting through the `csv.DictWriter` standard library module.

## Step-by-Step Configuration

Follow these steps to configure MediaCrawler for CSV output:

1. **Set the storage option** – Open [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and set `SAVE_DATA_OPTION = "csv"`.

2. **Configure the output directory** – Optionally set `SAVE_DATA_PATH` to a custom directory (e.g., `/tmp/media`). If left empty, files default to `data/<platform>/csv/`.

3. **Run the crawler** – Execute the main script. The store factory automatically instantiates the appropriate CSV implementation based on your configuration.

4. **Locate your files** – Find generated CSV files under your configured path using the pattern `<platform>/csv/<crawler_type>_contents_YYYY-MM-DD.csv`.

## How CSV Files Are Generated

When `AsyncFileWriter.write_to_csv` is invoked (typically at lines 46-55 of [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py)), the following process occurs:

- **Path construction**: The `_get_file_path` method builds the final file path using the platform name, item type (content, comment, or creator), and the crawler type stored in [`var.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/var.py).

- **Header writing**: The writer creates a `csv.DictWriter` using the keys of the first dictionary item as column headers. It writes the header only if the file is new or empty.

- **Row appending**: Subsequent calls append rows to the existing file using `writer.writerow(item)`.

- **Concurrency safety**: An async lock (`async with self.lock`) ensures thread-safe writes when multiple coroutines attempt to write simultaneously.

The resulting files use **UTF-8 BOM encoding** (`utf-8-sig`) to ensure proper character display in Excel and other tools.

## Code Examples

Configure CSV output in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py):

```python

# config/base_config.py

SAVE_DATA_OPTION = "csv"          # Enable CSV mode

SAVE_DATA_PATH = "/tmp/media"     # Optional: custom output directory

```

Run the crawler with CSV storage:

```bash
python main.py --platform zhihu --save-data-option csv

```

Internal factory selection (from [`store/zhihu/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/__init__.py)):

```python

# store/zhihu/__init__.py

STORE_MAP = {
    "csv": ZhihuCsvStoreImplement,
    "json": ZhihuJsonStoreImplement,
    # other options...

}

# The factory creates the appropriate store

store = ZhihuStoreFactory.create_store(save_option="csv")
await store.store_content(content_item)   # Writes to CSV

await store.store_comment(comment_item)   # Writes to CSV

await store.store_creator(creator_item)   # Writes to CSV

```

Core writing logic from [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py):

```python
async def write_to_csv(self, item: Dict, item_type: str):
    file_path = self._get_file_path('csv', item_type)
    async with self.lock:
        file_exists = os.path.exists(file_path)
        async with aiofiles.open(file_path, 'a', newline='', encoding='utf-8-sig') as f:
            writer = csv.DictWriter(f, fieldnames=item.keys())
            if not file_exists or await f.tell() == 0:
                await writer.writeheader()
            await writer.writerow(item)

```

## Platform-Specific Store Implementations

Each platform implements its own CSV store class following the same pattern:

- **Zhihu**: `ZhihuCsvStoreImplement` in [`store/zhihu/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/_store_impl.py)
- **Xiaohongshu**: `XhsCsvStoreImplement` in [`store/xhs/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/xhs/_store_impl.py)
- **Weibo**: `WeiboCsvStoreImplement` in [`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py)
- **Tieba**: `TiebaCsvStoreImplement` in [`store/tieba/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/tieba/_store_impl.py)

Each implementation provides `store_content()`, `store_comment()`, and `store_creator()` methods that forward data to the shared `AsyncFileWriter`.

## Summary

- Set `SAVE_DATA_OPTION = "csv"` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to enable CSV export for any supported platform.
- The `AsyncFileWriter` class in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) handles all file operations with async locking for thread safety.
- Platform-specific factories in `store/<platform>/__init__.py` route data to the appropriate CSV implementation.
- CSV files are created with UTF-8 BOM encoding and automatically include headers based on dictionary keys.
- Output paths follow the pattern `data/<platform>/csv/<crawler_type>_<item_type>_YYYY-MM-DD.csv` unless overridden by `SAVE_DATA_PATH`.

## Frequently Asked Questions

### How do I change the default CSV output directory?

Set the `SAVE_DATA_PATH` variable in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to an absolute path. If this variable is empty or unset, MediaCrawler defaults to `data/<platform>/csv/` relative to the project root.

### Can I export to CSV and JSON simultaneously?

MediaCrawler's `SAVE_DATA_OPTION` accepts a single value, so you cannot write to multiple formats simultaneously in one process. To get both formats, run the crawler twice with different `SAVE_DATA_OPTION` values, or implement a custom store class that calls multiple writers.

### What encoding does MediaCrawler use for CSV files?

The `AsyncFileWriter` uses `utf-8-sig` encoding (UTF-8 with BOM) when opening files. This ensures that Chinese characters and other Unicode text display correctly when opening files in Microsoft Excel or other spreadsheet applications.

### How does MediaCrawler handle concurrent writes to CSV files?

The `AsyncFileWriter` uses an `asyncio.Lock` (stored as `self.lock`) to synchronize access to the file handle. When `write_to_csv` is called, it acquires the lock before opening the file with `aiofiles`, ensuring that only one coroutine writes at a time and preventing data corruption during high-concurrency crawling.