How to Store Crawled Data in JSON with MediaCrawler

To store crawled data in JSON with MediaCrawler, set the SAVE_DATA_OPTION configuration to "json" or "jsonl" (line-delimited) in your platform-specific config, and the framework automatically writes formatted output to the data/json/ directory via the AsyncFileWriter.

The NanmiCoder/MediaCrawler repository provides a flexible, pluggable storage architecture that serializes scraped content from platforms like Tieba, Zhihu, Weibo, and Douyin. When you configure the system to store crawled data in JSON with MediaCrawler, the framework delegates persistence to specialized store implementations that handle platform-specific data structures through an asynchronous file writer.

How the JSON Storage Pipeline Works

The storage flow follows a three-stage pipeline that abstracts platform complexity while delivering consistent JSON output.

Configuration Layer

The entry point for JSON storage is the SAVE_DATA_OPTION flag defined in config/base_config.py (or overridden in platform-specific configs like config/tieba_config.py). Valid values are:

• json – Writes pretty-printed JSON files with indentation
• jsonl – Writes line-delimited JSON where each item occupies one line

Factory Pattern Selection

When the crawler starts, the generic BaseCrawler class invokes platform-specific store factories. In store/tieba/__init__.py, the factory inspects SAVE_DATA_OPTION and instantiates the appropriate implementation class—mapping "json" to TieBaJsonStoreImplement or similar classes for other platforms.

AsyncFileWriter Execution

Each JSON store implementation acts as a thin wrapper around the AsyncFileWriter class located in tools/async_file_writer.py. The store implementation converts crawler items to dictionaries and forwards them to the writer, which handles:

• Directory creation under data/json/
• File naming based on item type (e.g., contents.json, comments.json)
• Merging new items with existing file content (for standard JSON)
• Appending lines (for JSONL format)

Enabling JSON Output in Your Config

To activate JSON storage for a specific platform crawler, modify the configuration before initialization:

# config/tieba_config.py or your platform-specific config

from config.tieba_config import TiebaConfig

# Option 1: Pretty-printed JSON (merged array format)

TiebaConfig.SAVE_DATA_OPTION = "json"

# Option 2: Line-delimited JSON (one object per line)

TiebaConfig.SAVE_DATA_OPTION = "jsonl"

Once configured, the crawler automatically creates the data/json/ directory and writes files like contents.json, comments.json, and creators.json as the crawl progresses.

Implementation Details and File Structure

The actual persistence logic resides in three key components:

1. Store Implementation (store/tieba/_store_impl.py)

The concrete store class implements methods like store_content() that receive crawler items and delegate to the async writer:

# Excerpt from store/tieba/_store_impl.py

async def store_content(self, content_item: dict):
    """Forward the content dict to the async writer"""
    await self.writer.write_single_item_to_json(
        item_type="contents",
        item=content_item,
    )

2. AsyncFileWriter (tools/async_file_writer.py)

The writer determines the output path and handles format-specific serialization:

• For json: Reads existing content, merges the new item into the array, and writes with indent=4
• For jsonl: Appends json.dumps(item) + '\n' to the file without loading existing content

3. Output Directory Structure

After crawling completes, the data/json/ folder contains:

• contents.json – Main posts or threads
• comments.json – Reply data
• creators.json – Author information

Programmatic Access to JSON Storage

If you need to write custom crawler components that bypass the standard store factories, you can use the AsyncFileWriter directly:

import asyncio
from tools.async_file_writer import AsyncFileWriter

async def custom_json_export():
    writer = AsyncFileWriter(base_path="data/json")
    
    item = {
        "id": 123,
        "title": "Example Post",
        "text": "Crawled content here",
        "timestamp": "2024-01-15"
    }
    
    await writer.write_single_item_to_json(
        item_type="custom_contents",
        item=item
    )

asyncio.run(custom_json_export())

This produces data/json/custom_contents.json with the serialized item data.

Summary

• Configure SAVE_DATA_OPTION to "json" or "jsonl" in your platform config to enable JSON storage
• The store factory pattern in store/*/.__init__.py automatically selects the correct implementation based on your config
• AsyncFileWriter in tools/async_file_writer.py handles all file I/O, supporting both merged JSON arrays and line-delimited JSONL formats
• Output files are organized by item type under data/json/ with automatic pretty-printing for standard JSON
• All platforms (Tieba, Zhihu, Weibo, Douyin) follow this identical storage pattern

Frequently Asked Questions

What is the difference between json and jsonl formats in MediaCrawler?

When SAVE_DATA_OPTION is set to "json", MediaCrawler writes a single JSON array per file with pretty-printing (indent=4), merging new items into the existing array. When set to "jsonl", the system writes line-delimited JSON where each crawled item is a separate JSON object on its own line, which is more efficient for streaming large datasets and appending without reading existing content.

Where are the JSON output files located?

According to the AsyncFileWriter implementation in tools/async_file_writer.py, all JSON files are saved under the data/json/ directory relative to the project root. The writer creates this directory automatically if it does not exist. Files are named according to the item type (e.g., contents.json, comments.json, creators.json).

Can I use JSON storage with all supported platforms?

Yes. The storage abstraction is consistent across all platforms in the repository. Whether crawling Tieba, Zhihu, Weibo, or Douyin, each platform has a corresponding store factory (e.g., store/tieba/__init__.py, store/zhihu/__init__.py) that supports the same SAVE_DATA_OPTION configuration. The pattern of mapping "json" to a platform-specific implementation class (like TieBaJsonStoreImplement) is replicated across all platform modules.

How does MediaCrawler handle concurrent writes to JSON files?

The framework uses AsyncFileWriter which implements asynchronous file operations. For standard JSON format, it reads the existing file content, merges the new item into the data structure, and writes the complete updated content back. For JSONL format, it simply appends the new line. While the writer handles individual write operations asynchronously, the implementation in tools/async_file_writer.py manages file I/O serialization to prevent corruption during concurrent access within the same process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}