# How to Store Crawled Data in JSON with MediaCrawler

> Learn how to store crawled data in JSON using MediaCrawler. Simply configure SAVE_DATA_OPTION to json or jsonl and MediaCrawler handles the rest automatically.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**To store crawled data in JSON with MediaCrawler, set the `SAVE_DATA_OPTION` configuration to `"json"` or `"jsonl"` (line-delimited) in your platform-specific config, and the framework automatically writes formatted output to the `data/json/` directory via the `AsyncFileWriter`.**

The NanmiCoder/MediaCrawler repository provides a flexible, pluggable storage architecture that serializes scraped content from platforms like Tieba, Zhihu, Weibo, and Douyin. When you configure the system to store crawled data in JSON with MediaCrawler, the framework delegates persistence to specialized store implementations that handle platform-specific data structures through an asynchronous file writer.

## How the JSON Storage Pipeline Works

The storage flow follows a three-stage pipeline that abstracts platform complexity while delivering consistent JSON output.

### Configuration Layer

The entry point for JSON storage is the `SAVE_DATA_OPTION` flag defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) (or overridden in platform-specific configs like [`config/tieba_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/tieba_config.py)). Valid values are:

- **`json`** – Writes pretty-printed JSON files with indentation
- **`jsonl`** – Writes line-delimited JSON where each item occupies one line

### Factory Pattern Selection

When the crawler starts, the generic `BaseCrawler` class invokes platform-specific store factories. In [`store/tieba/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/tieba/__init__.py), the factory inspects `SAVE_DATA_OPTION` and instantiates the appropriate implementation class—mapping `"json"` to `TieBaJsonStoreImplement` or similar classes for other platforms.

### AsyncFileWriter Execution

Each JSON store implementation acts as a thin wrapper around the `AsyncFileWriter` class located in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py). The store implementation converts crawler items to dictionaries and forwards them to the writer, which handles:

- Directory creation under `data/json/`
- File naming based on item type (e.g., [`contents.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/contents.json), [`comments.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/comments.json))
- Merging new items with existing file content (for standard JSON)
- Appending lines (for JSONL format)

## Enabling JSON Output in Your Config

To activate JSON storage for a specific platform crawler, modify the configuration before initialization:

```python

# config/tieba_config.py or your platform-specific config

from config.tieba_config import TiebaConfig

# Option 1: Pretty-printed JSON (merged array format)

TiebaConfig.SAVE_DATA_OPTION = "json"

# Option 2: Line-delimited JSON (one object per line)

TiebaConfig.SAVE_DATA_OPTION = "jsonl"

```

Once configured, the crawler automatically creates the `data/json/` directory and writes files like [`contents.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/contents.json), [`comments.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/comments.json), and [`creators.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/creators.json) as the crawl progresses.

## Implementation Details and File Structure

The actual persistence logic resides in three key components:

**1. Store Implementation ([`store/tieba/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/tieba/_store_impl.py))**

The concrete store class implements methods like `store_content()` that receive crawler items and delegate to the async writer:

```python

# Excerpt from store/tieba/_store_impl.py

async def store_content(self, content_item: dict):
    """Forward the content dict to the async writer"""
    await self.writer.write_single_item_to_json(
        item_type="contents",
        item=content_item,
    )

```

**2. AsyncFileWriter ([`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py))**

The writer determines the output path and handles format-specific serialization:

- For `json`: Reads existing content, merges the new item into the array, and writes with `indent=4`
- For `jsonl`: Appends `json.dumps(item) + '\n'` to the file without loading existing content

**3. Output Directory Structure**

After crawling completes, the `data/json/` folder contains:

- [`contents.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/contents.json) – Main posts or threads
- [`comments.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/comments.json) – Reply data
- [`creators.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/creators.json) – Author information

## Programmatic Access to JSON Storage

If you need to write custom crawler components that bypass the standard store factories, you can use the `AsyncFileWriter` directly:

```python
import asyncio
from tools.async_file_writer import AsyncFileWriter

async def custom_json_export():
    writer = AsyncFileWriter(base_path="data/json")
    
    item = {
        "id": 123,
        "title": "Example Post",
        "text": "Crawled content here",
        "timestamp": "2024-01-15"
    }
    
    await writer.write_single_item_to_json(
        item_type="custom_contents",
        item=item
    )

asyncio.run(custom_json_export())

```

This produces [`data/json/custom_contents.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/data/json/custom_contents.json) with the serialized item data.

## Summary

- Configure `SAVE_DATA_OPTION` to `"json"` or `"jsonl"` in your platform config to enable JSON storage
- The store factory pattern in `store/*/.__init__.py` automatically selects the correct implementation based on your config
- `AsyncFileWriter` in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) handles all file I/O, supporting both merged JSON arrays and line-delimited JSONL formats
- Output files are organized by item type under `data/json/` with automatic pretty-printing for standard JSON
- All platforms (Tieba, Zhihu, Weibo, Douyin) follow this identical storage pattern

## Frequently Asked Questions

### What is the difference between json and jsonl formats in MediaCrawler?

When `SAVE_DATA_OPTION` is set to `"json"`, MediaCrawler writes a single JSON array per file with pretty-printing (`indent=4`), merging new items into the existing array. When set to `"jsonl"`, the system writes line-delimited JSON where each crawled item is a separate JSON object on its own line, which is more efficient for streaming large datasets and appending without reading existing content.

### Where are the JSON output files located?

According to the `AsyncFileWriter` implementation in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py), all JSON files are saved under the `data/json/` directory relative to the project root. The writer creates this directory automatically if it does not exist. Files are named according to the item type (e.g., [`contents.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/contents.json), [`comments.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/comments.json), [`creators.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/creators.json)).

### Can I use JSON storage with all supported platforms?

Yes. The storage abstraction is consistent across all platforms in the repository. Whether crawling Tieba, Zhihu, Weibo, or Douyin, each platform has a corresponding store factory (e.g., [`store/tieba/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/tieba/__init__.py), [`store/zhihu/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/zhihu/__init__.py)) that supports the same `SAVE_DATA_OPTION` configuration. The pattern of mapping `"json"` to a platform-specific implementation class (like `TieBaJsonStoreImplement`) is replicated across all platform modules.

### How does MediaCrawler handle concurrent writes to JSON files?

The framework uses `AsyncFileWriter` which implements asynchronous file operations. For standard JSON format, it reads the existing file content, merges the new item into the data structure, and writes the complete updated content back. For JSONL format, it simply appends the new line. While the writer handles individual write operations asynchronously, the implementation in [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) manages file I/O serialization to prevent corruption during concurrent access within the same process.