How to Update MediaCrawler to the Latest Version: A Complete Upgrade Guide

Pull the latest commits from the main branch, synchronize Python dependencies using uv sync, and reinstall Playwright browsers to update MediaCrawler to the latest version while preserving your configuration settings.

MediaCrawler is a modular social media scraping framework maintained by NanmiCoder. Because the project uses a clean separation between platform-specific logic and core infrastructure, updating MediaCrawler to the latest version requires only synchronizing the source code and refreshing dependencies. This guide covers the exact commands and file paths you need to execute a seamless upgrade.

Prerequisites for Updating MediaCrawler

Before starting the upgrade, ensure you have the following tools installed:

• Git – to pull the latest commits from the repository
• uv – the Python package manager used by MediaCrawler for deterministic resolution
• Playwright – browser automation library that requires version-matched binaries

If you do not have uv installed, run the official installer script:

curl -LsSf https://astral.sh/uv/install.sh | sh

Step-by-Step Upgrade Process

Follow these steps in order to ensure all components are correctly synchronized.

1. Fetch the Latest Source Code

Navigate to your local repository and pull the newest changes from the main branch. This updates all Python modules including main.py, base/base_crawler.py, and platform-specific code under media_platform/:

git pull origin main

2. Synchronize Python Dependencies

Run uv sync to reinstall the exact package versions declared in requirements.txt and pyproject.toml. This command ensures compatibility with dependencies like httpx==0.28.1 and playwright==1.45.0 as specified in the project's dependency lock:

uv sync

3. Update Playwright Browser Binaries

Playwright browser binaries are version-specific and must match the Python package version. After updating dependencies, reinstall the browsers to avoid runtime errors:

uv run playwright install

This downloads Chromium, Firefox, and WebKit binaries required by the crawler's automation layer.

4. Refresh Optional Components

If you use the xhshow signature helper for Xiaohongshu crawling, update it separately:

uv run pip install -U xhshow

5. Verify the Installation

Confirm that the entry point loads correctly and all modules are accessible by running the help command:

uv run main.py --help

If the command outputs usage instructions without errors, your update is complete.

Understanding the MediaCrawler Architecture

The project’s modular design makes updates straightforward because components are loosely coupled. Understanding this structure helps you predict which files change during an upgrade:

• main.py – The CLI entry point that parses arguments and dispatches to platform crawlers
• base/base_crawler.py – Abstract base class defining the shared workflow (login → data fetch → storage)
• media_platform/<platform>/ – Directory containing platform-specific implementations (e.g., media_platform/zhihu/core.py)
• config/base_config.py – Global configuration file containing toggles like ENABLE_GET_COMMENTS
• store/ – Data persistence layer with implementations for CSV, JSON, SQLite, MySQL, and MongoDB

Because configuration and data storage are separated from core logic, updating the source code does not overwrite your personal settings unless you manually edit tracked files.

Running MediaCrawler After the Update

Once the update is verified, execute your crawls using the uv run prefix to ensure the correct environment. For example, to crawl Xiaohongshu (XHS) using QR-code login and keyword search:

uv run main.py --platform xhs --lt qrcode --type search

To fetch specific post details by ID:

uv run main.py --platform xhs --lt qrcode --type detail

If you prefer traditional virtual environments, you can substitute uv run with manual activation of your virtualenv, though uv handles dependency resolution more efficiently.

Summary

• Pull the latest source using git pull origin main to receive updates for main.py and all platform modules
• Execute uv sync to align Python packages with the locked versions in requirements.txt
• Run uv run playwright install to match browser binaries with the updated Playwright library
• Update auxiliary tools like xhshow separately if required for signature generation
• Verify the installation with uv run main.py --help before executing production crawls

Frequently Asked Questions

Will updating MediaCrawler overwrite my custom configuration?

No. The project stores global settings in config/base_config.py, which Git tracks separately from your local modifications. As long as you have not committed changes to tracked files, your configuration persists across updates. The modular architecture ensures that platform-specific logic in media_platform/ and storage implementations in store/ update independently of your personal settings.

Do I need to update the Playwright browsers every time?

Yes, whenever requirements.txt specifies a new Playwright version, you must run uv run playwright install. The browser binaries in your system cache must match the Python library version exactly; otherwise, the crawler will fail to launch browser instances when executing login flows in base/base_crawler.py.

Can I use pip instead of uv to update MediaCrawler?

Yes. While the repository recommends uv for its speed and deterministic resolution, you can achieve the same result by running pip install -r requirements.txt followed by playwright install. The main.py entry point and all modules in media_platform/ function identically regardless of which package manager installs the dependencies.

How do I check which version of MediaCrawler I am running?

Since MediaCrawler uses a rolling release model on the main branch, check your current version by viewing the latest commit hash: git log --oneline -1. You can also compare your local base/base_crawler.py or requirements.txt against the repository’s main branch to see if you are behind the latest updates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}