Exporting MediaCrawler Data to Excel with Proper Formatting

MediaCrawler provides a singleton-based Excel export layer via ExcelStoreBase that automatically applies professional styling—blue headers, auto-sized columns, and thin borders—to scraped social media data.

The NanmiCoder/MediaCrawler project ships with a dedicated Excel export system that converts raw platform data into clean, analysis-ready workbooks. Understanding how to export MediaCrawler data to Excel with proper formatting ensures your scraped content, comments, and creator metadata maintain readability and professional presentation across supported platforms like Weibo, Bilibili, and Xiaohongshu.

Understanding the Excel Export Architecture

At the heart of MediaCrawler's Excel functionality lies the ExcelStoreBase class defined in store/excel_store_base.py. This implementation follows the Abstract Store pattern established in base/base_crawler.py, where AbstractStore defines the generic contract (store_content, store_comment, etc.) that concrete storage backends must fulfill.

The architecture employs a singleton pattern per platform and crawler type combination. When you call ExcelStoreBase.get_instance(platform="weibo", crawler_type="search"), the class returns a single shared instance for that specific combination, ensuring all data from a crawling session aggregates into one workbook. A class-level threading.Lock protects the singleton registry and file operations, preventing race conditions when multiple crawlers run concurrently.

Configuring Output Paths and Dependencies

Before writing data, ensure your environment meets the requirements. The Excel store relies on openpyxl, which is imported lazily—if missing, the store raises a clear error indicating the dependency is required.

Output location is controlled by config.SAVE_DATA_PATH (defined in config/base_config.py). If this value is empty, files fall back to a local data/ directory. Generated filenames follow the pattern <platform>_<crawler_type>_YYYYMMDD_HHMMSS.xlsx, making each crawl session easily identifiable.

Writing Data to Excel Sheets

The ExcelStoreBase automatically manages four primary worksheets: Contents, Comments, and Creators. For specific platforms like Bilibili, optional Contacts and Dynamics sheets are created on demand.

Storing Content and Comments

Use the store_content and store_comment methods to populate the primary sheets. The first insertion triggers header creation via _write_headers and _apply_header_style.

from store.excel_store_base import ExcelStoreBase

# Obtain singleton for Weibo search crawler

excel_store = ExcelStoreBase.get_instance(platform="weibo", crawler_type="search")

# Store main content

await excel_store.store_content({
    "note_id": "12345",
    "title": "My Weibo Post",
    "likes": 256,
    "created_at": "2025-03-22T14:00:00"
})

# Store associated comment

await excel_store.store_comment({
    "comment_id": "c9876",
    "user": "alice",
    "text": "Nice post!",
    "likes": 12
})

Handling Creator Metadata and Platform-Specific Data

Store author information using store_creator. For Bilibili crawls, use the optional sheet methods:

excel_store = ExcelStoreBase.get_instance(platform="bilibili", crawler_type="detail")

# Store creator information

await excel_store.store_creator({
    "user_id": "u001",
    "nickname": "Alice",
    "followers": 10234
})

# Store contact relationships (fans/following)

await excel_store.store_contact({
    "up_id": "up123",
    "fan_id": "fan456",
    "follow_time": "2024-11-01"
})

# Store dynamic entries (videos)

await excel_store.store_dynamic({
    "dynamic_id": "dyn789",
    "title": "Cool Clip",
    "views": 5870
})

The _write_row method handles data serialization automatically, converting lists and dictionaries to strings and sanitizing None values. Column order is determined by the keys in the first received item, ensuring consistent layout throughout the sheet.

Automatic Formatting and Styling

MediaCrawler applies professional formatting without manual intervention. When headers are first written, _apply_header_style executes:

• Blue fill background color on header cells
• Bold white font for high contrast
• Centered alignment for clean presentation
• Thin borders to define column boundaries

After all rows are appended, _auto_adjust_column_width measures content length per column and sets widths between 10 pixels and 50 pixels. This prevents truncated text while avoiding excessively wide columns that hinder readability.

Finalizing and Persisting Excel Files

Data remains in memory until explicitly flushed. Call flush() on individual instances to remove empty sheets, apply final column width adjustments, and write the .xlsx file to config.SAVE_DATA_PATH.

To ensure all singleton instances persist their data at the end of a crawling session, invoke ExcelStoreBase.flush_all(), which iterates through every active instance and clears the registry:

from store.excel_store_base import ExcelStoreBase

# Called once after all crawlers complete

ExcelStoreBase.flush_all()

This method is typically triggered by the crawler manager in api/services/crawler_manager.py when the scraping job finishes. All actions are logged via tools.utils.logger, providing clear audit trails for production troubleshooting.

Summary

• ExcelStoreBase in store/excel_store_base.py provides a thread-safe, singleton-based Excel export mechanism per platform and crawler type.
• Automatic formatting includes blue headers with white bold text, centered alignment, thin borders, and intelligent column width adjustment (10px–50px).
• Four primary sheets (Contents, Comments, Creators) are always created, with optional Contacts and Dynamics sheets for specific platforms like Bilibili.
• Data persistence requires calling flush() for individual instances or flush_all() at the end of the crawling session to write files to config.SAVE_DATA_PATH.
• Filename convention uses <platform>_<crawler_type>_YYYYMMDD_HHMMSS.xlsx to prevent overwrites and maintain session traceability.

Frequently Asked Questions

What file naming convention does MediaCrawler use for Excel exports?

MediaCrawler generates filenames using the pattern <platform>_<crawler_type>_YYYYMMDD_HHMMSS.xlsx, incorporating the platform identifier, crawler type, and timestamp. This convention prevents file overwrites when running multiple crawling sessions and makes it easy to identify when specific data was collected.

How does MediaCrawler handle concurrent writes to Excel files?

The ExcelStoreBase implementation uses a class-level threading.Lock to protect the singleton dictionary and flushing operations. This ensures that concurrent crawlers accessing the same platform and type combination do not corrupt the workbook or encounter race conditions during file writes.

Can I customize the Excel styling or column widths in MediaCrawler?

The current implementation in store/excel_store_base.py applies fixed styling (blue headers, white bold font, thin borders) and automatic column width adjustment between 10px and 50px through internal methods like _apply_header_style and _auto_adjust_column_width. To customize styling, you would need to subclass ExcelStoreBase and override these protected methods with your own formatting logic using openpyxl capabilities.

Why is my Excel file empty after running the crawler?

If the output file appears empty, you likely missed calling ExcelStoreBase.flush_all() after the crawling session completes. The store keeps data in memory until flushed to optimize performance. Ensure your crawler manager or main script invokes this method, or call flush() directly on your specific store instance before the program exits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}