How to Filter Crawled Content by Date or Keywords in MediaCrawler

To filter crawled content by date or keywords in MediaCrawler, modify the comma-separated KEYWORDS string and PUBLISH_TIME_TYPE enum in the configuration files, then run the crawler—the platform-specific clients automatically inject these values into API requests via filter_selected parameters and propagate keywords through ContextVar context variables.

MediaCrawler is an open-source multi-platform crawler supporting Zhihu, Douyin, Weibo, and Tieba. Filtering operates across four distinct layers—Configuration, Crawler Core, Client, and Utility Helpers—allowing you to control search parameters without modifying underlying platform logic. This architecture ensures that keyword lists and date ranges defined in static config files translate directly into API query parameters.

Understanding the Filter Architecture

The filtering system spans four architectural layers that transform static configuration into live API requests. The Configuration layer (config/base_config.py and platform-specific configs) holds global switches like KEYWORDS and PUBLISH_TIME_TYPE. The Crawler Core (media_platform/*/core.py) executes search loops and manages context propagation. The Client layer (media_platform/*/client.py) translates configuration into actual HTTP parameters, while Utility Helpers (tools/utils.py, var.py) handle timestamp conversion and context variable management.

Keyword Filtering Implementation

Keywords are defined as a comma-separated string in config/base_config.py via the KEYWORDS constant. When a crawler initializes—whether ZhihuCrawler, DouYinCrawler, or another platform-specific implementation—the start method iterates over this list:

for keyword in config.KEYWORDS.split(","):
    source_keyword_var.set(keyword)  # Defined in var.py

    utils.logger.info(f"[Crawler] Current search keyword: {keyword}")
    await self.zhihu_client.get_note_by_keyword(keyword=keyword, page=page)

The source_keyword_var is a ContextVar defined in var.py that propagates the active keyword downstream through the async context. When storage layers persist records—such as in store/zhihu/__init__.py—they automatically attach the current keyword value to the source_keyword field, ensuring every database entry retains its originating search term.

Date Filtering by Platform

Date filtering is primarily exposed through the PublishTimeType enum in config/dy_config.py. The Douyin client (and similar platforms) injects these values into API requests via a structured JSON payload. In media_platform/douyin/client.py, the get_note_by_keyword method constructs the filter as follows:

if sort_type.value != SearchSortType.GENERAL.value \
   or publish_time.value != PublishTimeType.UNLIMITED.value:
    query_params["filter_selected"] = json.dumps({
        "sort_type": str(sort_type.value),
        "publish_time": str(publish_time.value)
    })
    query_params["is_filter_search"] = 1

The publish_time value corresponds to integer codes (e.g., 0 for unlimited, 1 for last 24 hours, 2 for last 7 days). Setting is_filter_search to 1 signals the API to apply these constraints. After retrieval, the tools/utils.py helper converts Unix timestamps to readable strings:

def get_time_str_from_unix_time(unixtime: int) -> str:
    return datetime.fromtimestamp(unixtime).strftime("%Y-%m-%d %H:%M:%S")

This formatted string populates the publish_time field in Pydantic models defined in files like model/m_zhihu.py.

Step-by-Step Configuration Guide

Configuring Keywords

Edit config/base_config.py to define your search terms as a comma-separated string:

# config/base_config.py

KEYWORDS = "python,机器学习,AI技术"

Each term will be processed sequentially by the crawler core.

Setting Date Range Filters

For Douyin, modify config/dy_config.py to set the PUBLISH_TIME_TYPE:

# config/dy_config.py

PUBLISH_TIME_TYPE = 2  # 2 = last 7 days; 1 = last 24 hours; 0 = unlimited

Other platforms (Weibo, Tieba) follow similar patterns using their respective config files.

Executing the Crawler

Run the application from the command line:

python main.py

The platform-specific crawler will automatically apply keyword iteration and date filters based on your configuration constants.

Accessing Filtered Data in Storage

Once crawling completes, each record contains metadata tracing back to your filter criteria. The source_keyword field stores the specific keyword from the KEYWORDS list that generated the entry, accessible via the context variable mechanism. For post-processing, use utility functions from tools/utils.py to convert between string dates and timestamps:

from tools import utils

def filter_by_date(notes, start: str, end: str):
    """Filter notes between start and end dates (YYYY-MM-DD)."""
    start_ts = utils.timestamp_from_str(start)
    end_ts = utils.timestamp_from_str(end)
    return [
        n for n in notes
        if start_ts <= utils.timestamp_from_str(n.publish_time) <= end_ts
    ]

To inspect stored keywords:

from store import zhihu as zhihu_store

async def inspect():
    notes = await zhihu_store.query_notes()
    for note in notes:
        print(f"Title: {note.title}, Keyword: {note.source_keyword}")

Summary

• Keyword filtering relies on the KEYWORDS string in config/base_config.py, split by commas and propagated via source_keyword_var (var.py) to attach provenance to every record.
• Date filtering uses the PublishTimeType enum (e.g., config/dy_config.py) and translates to filter_selected JSON payloads with is_filter_search=1 in platform clients like media_platform/douyin/client.py.
• Storage integration automatically persists both the keyword context and formatted publication timestamps via tools/utils.py conversion helpers.
• No code changes required—modify configuration constants and run python main.py to apply filters across all supported platforms.

Frequently Asked Questions

Where do I configure search keywords for crawling?

Configure the comma-separated KEYWORDS string in config/base_config.py. The crawler core automatically splits this string and iterates through each term, setting the active keyword via source_keyword_var.set() in var.py before executing platform-specific searches.

How does MediaCrawler handle date filtering on different platforms?

Each platform client (e.g., media_platform/douyin/client.py) implements date filtering by constructing a filter_selected JSON object containing publish_time and sort_type values, then setting is_filter_search=1 in the query parameters. The specific date range codes are defined in platform-specific config files like config/dy_config.py.

Can I apply custom date filters to already-crawled data?

Yes. Use the get_time_str_from_unix_time function or reverse timestamp conversion utilities in tools/utils.py to compare stored publish_time values against your custom date range. The filter_by_date pattern shown above demonstrates post-processing retrieval without re-crawling.

How is the original keyword attached to each stored record?

The source_keyword_var ContextVar in var.py maintains the current keyword across async execution contexts. When storage implementations (such as store/zhihu/__init__.py) persist records, they read this context variable and populate the source_keyword field, ensuring full traceability from search term to stored result.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}