How to Configure Media Downloads for Videos in MediaCrawler

To configure video downloads in MediaCrawler, set ENABLE_GET_MEIDAS = True in main/config/base_config.py and define your DOWNLOAD_ROOT path so the crawler automatically fetches and stores video files via the async file writer pipeline.

MediaCrawler is an open-source scraping framework that supports multiple Chinese social media platforms. While it defaults to extracting only textual metadata, you can configure it to capture full video files by adjusting two critical settings in the configuration package. This guide walks through the exact steps to activate and customize video downloads using the actual source implementation.

Enable Video Downloads in MediaCrawler

The crawler treats media fetching as an opt-in feature controlled by a central toggle.

Toggle the Media Download Switch

Located in main/config/base_config.py, the ENABLE_GET_MEIDAS boolean acts as the master switch for all binary downloads. When set to False (the default), the crawler stores only JSON metadata and text content. Changing this to True signals the platform handlers to initiate file downloads.

# main/config/base_config.py

ENABLE_GET_MEIDAS = True

Set the Download Directory

By default, files write to a ./download folder relative to the working directory. Override this by setting DOWNLOAD_ROOT to an absolute or relative path. For platform-specific organization, use VIDEO_DOWNLOAD_ROOT to isolate videos from other media types.

# main/config/base_config.py

import os

DOWNLOAD_ROOT = os.path.join(os.getcwd(), "media_downloads")
VIDEO_DOWNLOAD_ROOT = os.path.join(DOWNLOAD_ROOT, "douyin_videos")

How the Video Download Pipeline Works

Understanding the internal flow helps troubleshoot issues and customize behavior. The implementation in media_platform/douyin/core.py (mirrored in other platform modules) follows a four-stage pipeline:

1. Browser initialization – The launcher passes accept_downloads=True to Playwright, permitting the headless browser to handle file streams.
2. Media detection – The get_aweme_media method inspects the post type, routing image notes to note_download_url and videos to video_download_url.
3. Binary fetching – The URL feeds into self.dy_client.get_aweme_media, which returns the raw bytes.
4. Persistent storage – The store.update_dy_aweme_video method delegates to AsyncFileWriter, writing content to the configured root while respecting the DOWNLOAD_ROOT path.

The writer creates subdirectories automatically, structuring files as <DOWNLOAD_ROOT>/douyin_videos/<aweme_id>/video.mp4.

Customizing Download Behavior

Beyond enabling the feature, you can control naming conventions and storage locations.

Filename Patterns and Storage Structure

The default implementation in store/douyin/_store_impl.py uses a static suffix "video.mp4" combined with the unique aweme_id. To modify this pattern, edit the extension_file_name variable in the storage implementation before the AsyncFileWriter call.

# Inside store/douyin/_store_impl.py

extension_file_name = f"{aweme_id}_custom.mp4"

Platform-Specific Overrides

Each platform maintains its own config file (e.g., main/config/dy_config.py for Douyin) that imports from base_config.py. You can override VIDEO_DOWNLOAD_ROOT here to keep platform data separated without modifying the base configuration.

# main/config/dy_config.py

import os
from main.config import base_config

VIDEO_DOWNLOAD_ROOT = os.path.join(base_config.DOWNLOAD_ROOT, "douyin")

Complete Configuration Example

Combine these settings to activate video capture for Douyin content:

# main/config/base_config.py

import os

ENABLE_GET_MEIDAS = True
DOWNLOAD_ROOT = "./media_downloads"
VIDEO_DOWNLOAD_ROOT = os.path.join(DOWNLOAD_ROOT, "douyin_videos")

When executing the crawler via main.py, the configuration propagates automatically:

# main.py

from config import base_config as cfg
from media_platform.douyin.core import DouYinCrawler

async def run():
    crawler = DouYinCrawler()
    await crawler.start()  # Respects cfg.ENABLE_GET_MEIDAS

During execution, media_platform/douyin/core.py processes each item:

video_download_url = douyin_store._extract_video_download_url(aweme_item)
content = await self.dy_client.get_aweme_media(video_download_url)
extension_file_name = "video.mp4"
await douyin_store.update_dy_aweme_video(aweme_id, content, extension_file_name)

The AsyncFileWriter in main/tools/async_file_writer.py commits the bytes to disk at:

media_downloads/douyin_videos/<aweme_id>/video.mp4

Summary

• Enable downloads by setting ENABLE_GET_MEIDAS = True in main/config/base_config.py.
• Define storage paths using DOWNLOAD_ROOT and optional VIDEO_DOWNLOAD_ROOT variables.
• Understand the pipeline spanning from Playwright's accept_downloads through get_aweme_media to AsyncFileWriter.
• Customize filenames by modifying extension_file_name in store/douyin/_store_impl.py.
• Apply platform overrides in dedicated config files like main/config/dy_config.py to keep data organized.

Frequently Asked Questions

Where is the video download setting located in MediaCrawler?

The primary toggle resides in main/config/base_config.py as the boolean constant ENABLE_GET_MEIDAS. Setting this to True activates the download pipeline across all supported platforms, while False restricts the crawler to metadata only.

What file format are videos saved as in MediaCrawler?

By default, the crawler saves videos with the .mp4 extension. The specific filename is constructed in store/douyin/_store_impl.py using the aweme_id and a static "video.mp4" suffix, though you can customize this pattern by editing the extension_file_name parameter.

Can I change the download location for specific platforms only?

Yes. While DOWNLOAD_ROOT in main/config/base_config.py sets the global default, each platform config (such as main/config/dy_config.py for Douyin) can redefine VIDEO_DOWNLOAD_ROOT to isolate that platform's media in a dedicated subdirectory without affecting other crawlers.

Why are my videos not downloading even after enabling the setting?

Verify that ENABLE_GET_MEIDAS is spelled correctly and set to True in the config file actually being imported by your entry script. Additionally, ensure the DOWNLOAD_ROOT directory has write permissions, and check that Playwright's accept_downloads=True parameter is present in the browser launch configuration within media_platform/douyin/core.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}