Logging Mechanisms for Debugging Platform-Specific Issues in MediaCrawler

MediaCrawler uses a centralized Python logging instance configured in tools/utils.py to trace platform-specific execution paths across browser automation, data storage, and async processing modules.

MediaCrawler relies on the standard Python logging library to diagnose issues that vary between operating systems, such as Chrome process management on Windows versus signal handling on Linux. The logging infrastructure is initialized once at import time in tools/utils.py and exported as utils.logger, providing consistent diagnostic output across all platform-specific components.

Centralized Logger Configuration in tools/utils.py

The init_loging_config() function in tools/utils.py establishes a module-level logger named MediaCrawler that handles all diagnostic output. This centralized approach ensures that every component writes to the same stream with identical formatting, making it easier to correlate events across different OS-specific code paths.

# tools/utils.py

def init_loging_config():
    level = logging.INFO
    logging.basicConfig(
        level=level,
        format="%(asctime)s %(name)s %(levelname)s (%(filename)s:%(lineno)d) - %(message)s",
        datefmt='%Y-%m-%d %H:%M:%S'
    )
    _logger = logging.getLogger("MediaCrawler")
    _logger.setLevel(level)

    # Suppress noisy httpx INFO logs

    logging.getLogger("httpx").setLevel(logging.WARNING)

    return _logger

logger = init_loging_config()

The format string includes the timestamp, logger name, severity level, source filename, and line number—critical metadata when tracing platform-specific failures that may occur in different files depending on the host OS. By default, the logger filters httpx library messages to WARNING level, preventing connection pool noise from obscuring MediaCrawler-specific diagnostics.

Platform-Specific Log Usage Across Modules

MediaCrawler injects utils.logger calls throughout components that interact with the underlying OS or external browsers. These logs reveal platform-specific behaviors like Chromium DevTools Protocol (CDP) connection failures or browser process cleanup sequences.

CDP Browser Manager (tools/cdp_browser.py)

The CDP browser manager logs lifecycle events and connection tests that often behave differently across Windows and Unix-like systems.

# Example log statements from tools/cdp_browser.py

utils.logger.info("[CDPBrowserManager] atexit: Cleaning up browser process")
utils.logger.warning("[CDPBrowserManager] CDP connection test failed: {e}")

These messages capture platform-specific signal handling and process termination patterns that vary between Windows (nt) and POSIX systems.

Browser Launcher (tools/browser_launcher.py)

When launching Chromium executables with specific flags or debug ports, the launcher records path resolutions and startup failures.

# Example log statements from tools/browser_launcher.py

utils.logger.info(f"[BrowserLauncher] Launching browser: {browser_path}")
utils.logger.error(f"[BrowserLauncher] Failed to launch browser: {e}")

Path formatting and executable permissions differ between platforms, making these logs essential for diagnosing why Chrome might launch on Linux but fail on Windows.

Async File Writer (tools/async_file_writer.py)

Word-cloud generation and file I/O operations log empty datasets and processing errors.

# Example log statements from tools/async_file_writer.py

utils.logger.info("[AsyncFileWriter.generate_wordcloud_from_comments] No comments data found")
utils.logger.error("[AsyncFileWriter.generate_wordcloud_from_comments] Error generating wordcloud: {e}")

Async event loop behaviors and file system permissions vary by OS, and these logs help isolate whether failures stem from data issues or platform-specific path handling.

Data Store Implementations

Storage backends like the Zhihu MongoDB implementation confirm successful writes for debugging persistence issues.

# Example from store/zhihu/_store_impl.py

utils.logger.info(f"[ZhihuMongoStoreImplement.store_content] Saved note {note_id} to MongoDB")

Database connection pooling and authentication mechanisms often require platform-specific configuration, and these logs verify that the crawler successfully commits data regardless of the host environment.

Adjusting Log Levels for Platform-Specific Debugging

Because the logger is global, you can adjust verbosity at runtime based on the host operating system. This pattern allows you to increase detail for problematic platforms while keeping output concise on stable ones.

# main.py - Platform-specific log configuration

import os
from tools import utils

if os.name == "nt":                     # Windows

    utils.logger.setLevel(logging.DEBUG)
elif os.name == "posix":                # Linux/macOS

    utils.logger.setLevel(logging.INFO)

Setting DEBUG level on Windows captures granular details about Chrome process spawning and CDP port binding, which often require elevated permissions or specific flags compared to Linux. This runtime adjustment requires no changes to the underlying tools/utils.py configuration.

Extending Logging to Persistent Files

By default, logging.basicConfig streams output to stderr, meaning logs appear in the console but are not persisted. To capture platform-specific debugging sessions for later analysis, attach a FileHandler to the existing logger.

# Adding file output for persistent debugging

import logging
from tools import utils

file_handler = logging.FileHandler("media_crawler_platform_debug.log")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
    "%(asctime)s %(name)s %(levelname)s (%(filename)s:%(lineno)d) - %(message)s"
))
utils.logger.addHandler(file_handler)

This extension pattern preserves the original console output while writing duplicate entries to disk, enabling you to review platform-specific browser crashes or connection timeouts after the fact.

Summary

• MediaCrawler centralizes logging in tools/utils.py through a single MediaCrawler logger instance exported as utils.logger.
• The log format includes timestamps, severity levels, and precise line numbers to trace platform-specific code paths across tools/cdp_browser.py, tools/browser_launcher.py, and storage modules.
• httpx library logs are suppressed to WARNING by default to reduce noise from HTTP connection pools.
• You can adjust verbosity per platform by calling utils.logger.setLevel() conditionally based on os.name or other environment detectors.
• Logs stream to stderr by default, but you can add file handlers for persistent debugging records of platform-specific issues.

Frequently Asked Questions

How do I enable DEBUG logging only on Windows?

Import utils from tools and check os.name at runtime. Windows identifies as "nt", so you can conditionally set the level before crawling begins.

import os
from tools import utils

if os.name == "nt":
    utils.logger.setLevel(logging.DEBUG)

Where does MediaCrawler write logs by default?

Logs write to stderr (standard error) through the default logging.basicConfig setup in tools/utils.py. The repository does not ship with a file handler, so console output is the primary destination unless you manually add a FileHandler.

Why am I seeing httpx connection logs mixed with crawler output?

The init_loging_config() function explicitly sets logging.getLogger("httpx").setLevel(logging.WARNING) to suppress INFO-level HTTP client noise. If you see DEBUG-level httpx messages, you may have inadvertently raised the root logger level or removed this suppression.

Can I add JSON formatting for structured logging across platforms?

Yes. Replace the format string in tools/utils.py or add a custom Formatter subclass that overrides format() to output JSON. Because all modules use the single utils.logger instance, formatting changes apply globally to all platform-specific components immediately.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}