How to Configure Media Downloads for Images in MediaCrawler

Enable image downloading in MediaCrawler by setting ENABLE_GET_MEIDAS = True in config/base_config.py and optionally customize the storage root with SAVE_DATA_PATH.

MediaCrawler is a multi-platform social media crawler that supports downloading images and videos when properly configured. To save image files locally, you must activate the media mode and understand how platform-specific storage implementations handle the binary data. This guide covers the exact configuration steps and source code paths needed to configure media downloads for images across XiaoHongShu, Weibo, and Douyin platforms.

Enable the Global Media Download Flag

MediaCrawler uses a global configuration flag to control whether media content is persisted to disk.

Locating the Configuration File

Open config/base_config.py and locate the media enable flag at lines 107–108. Set ENABLE_GET_MEIDAS to True to activate image and video downloading:

# config/base_config.py

ENABLE_GET_MEIDAS = True   # enable image / video downloading

When this flag is False, the crawler skips all media storage operations regardless of platform. The flag is read at import time, so you must set it before running the crawler.

Customize Image Storage Locations

By default, MediaCrawler saves images under data/<platform>/images. You can override this root directory using the SAVE_DATA_PATH configuration variable.

Setting a Custom Root Directory

In config/base_config.py, add or modify SAVE_DATA_PATH:

# config/base_config.py

SAVE_DATA_PATH = "/mnt/media_crawler"

With this setting, final image paths resolve to /mnt/media_crawler/<platform>/images. According to the source code in the store implementations (e.g., XiaoHongShuImage at lines 36–40 in store/xhs/xhs_store_media.py), the system checks if config.SAVE_DATA_PATH and appends platform-specific subdirectories to your custom path.

Platform-Specific Image Storage Implementations

Each platform implements its own image storage logic by inheriting from AbstractStoreImage defined in base/base_crawler.py. All implementations use aiofiles for asynchronous file writing and follow the same pattern: create platform-specific sub-folders, build filenames using content IDs, and write binary data.

XiaoHongShu Image Storage

The XiaoHongShu implementation resides in store/xhs/xhs_store_media.py. The XiaoHongShuImage class implements the save_image method, which constructs paths like:

data/xhs/images/<notice_id>/<image_file>.jpg

Weibo Image Storage

Weibo images are handled in store/weibo/weibo_store_media.py by the WeiboImageStoreImplement class. Files are saved directly to:

data/weibo/images/<pic_id>.jpg

Douyin Image Storage

Douyin storage logic is located in store/douyin/douyin_store_media.py within the DouYinImageStoreImplement class. This creates a nested structure:

data/douyin/images/<aweme_id>/<image_file>.png

All three classes extend the abstract base class AbstractStoreImage from base/base_crawler.py, ensuring consistent async handling across platforms.

Running the Crawler and Verifying Downloads

Once ENABLE_GET_MEIDAS is set to True, execute your crawl job normally. No additional command-line flags are required—the configuration is read at runtime.

Expected Directory Structure

After crawling completes, verify the downloads in your configured storage path. You should see this hierarchy:

data/
├── xhs/
│   └── images/
│       └── <notice_id>/
│           └── <image_file>.jpg
├── weibo/
│   └── images/
│       └── <pic_id>.jpg
└── douyin/
    └── images/
        └── <aweme_id>/
            └── <image_file>.png

Log Verification

Each store class prints confirmation messages upon successful writes. Look for entries like:

[XiaoHongShuImageStoreImplement.save_image] save image data/xhs/images/12345/abc.jpg success ...

Example execution code:

from main import run_crawler

# With ENABLE_GET_MEIDAS=True, this will invoke platform-specific store_image methods

run_crawler(platform="xhs", keywords="编程副业,编程兼职")

Summary

• Enable media mode by setting ENABLE_GET_MEIDAS = True in config/base_config.py before running the crawler.
• Customize storage paths using SAVE_DATA_PATH to redirect image files from the default data/ directory to your preferred location.
• Platform implementations handle the actual file I/O: XiaoHongShuImage in store/xhs/xhs_store_media.py, WeiboImageStoreImplement in store/weibo/weibo_store_media.py, and DouYinImageStoreImplement in store/douyin/douyin_store_media.py.
• All stores inherit from AbstractStoreImage in base/base_crawler.py and use aiofiles for asynchronous writes.
• Verification involves checking the data/<platform>/images directory (or your custom path) and reviewing log output for success messages.

Frequently Asked Questions

Why are my images not downloading?

Ensure ENABLE_GET_MEIDAS is set to True in config/base_config.py and that the configuration file is saved before running the crawler. This flag is evaluated at import time, so changes require a restart of the crawler process. Also verify that the target posts actually contain image media and that your network connection can access the CDN URLs.

Can I change the image filename format?

The filename format is hardcoded in each platform's store implementation. For example, XiaoHongShu uses notice_id and pic_id combinations in store/xhs/xhs_store_media.py. To customize filenames, you must modify the save_image method in the respective platform store class, though this requires maintaining a fork of the repository.

Does MediaCrawler support async image downloads?

Yes. All image storage implementations use aiofiles for asynchronous file I/O operations, allowing concurrent downloads without blocking the event loop. The AbstractStoreImage base class in base/base_crawler.py defines the async interface that all platform implementations follow.

Which platforms support image downloading?

According to the source code, XiaoHongShu (store/xhs/xhs_store_media.py), Weibo (store/weibo/weibo_store_media.py), and Douyin (store/douyin/douyin_store_media.py) all implement the AbstractStoreImage interface. Additional platforms may support image storage if they provide similar store implementations following the same inheritance pattern.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}