How Python-100-Days Teaches Web Scraping: A Progressive 5-Day Curriculum

Python-100-Days teaches web scraping through Days 61-65, a structured progression that moves from basic HTTP requests to production-grade Scrapy frameworks, covering concurrency patterns, browser automation, and crawler architecture.

The jackfrued/Python-100-Days repository is one of GitHub’s most comprehensive Chinese-language Python tutorials. Its web scraping module spans five consecutive days (Days 61-65) and employs a layered learning approach that introduces complexity gradually while emphasizing legal ethics and architectural best practices. This guide examines the exact source files, coding patterns, and progressive concepts used to teach these skills.

The Five-Day Web Scraping Track (Days 61-65)

The curriculum structures web scraping education as a three-stage pipeline (download → parse → store) that evolves from simple scripts to enterprise frameworks. Each day corresponds to a specific markdown file in the Day61-65/ directory.

Day 61 – Web Crawler Architecture and Ethics

According to Day61-65/61.网络数据采集概述.md, the module begins with foundational concepts rather than immediate coding. The lesson defines the classic crawler architecture and Python’s suitability for network data collection. It establishes the three-stage pipeline workflow and emphasizes critical legal considerations, including robots.txt compliance and ethical crawling boundaries, before introducing any implementation details.

Day 62 – Acquiring Network Resources with Requests

File Day61-65/62.用Python获取网络资源-1.md introduces static page fetching using the requests library. The lesson demonstrates requests.get() calls, proper status_code handling, and the distinction between resp.text (decoded HTML) and resp.content (raw binary bytes). It stresses the importance of custom User-Agent headers to prevent server blocking.

The curriculum teaches basic extraction using regular expressions before introducing BeautifulSoup or parsing libraries:

import requests
import re

url = 'https://www.sohu.com/'
resp = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

if resp.status_code == 200:
    html = resp.text
    pattern = re.compile(r'<a.*?href="(.*?)".*?title="(.*?)".*?>')
    for href, title in pattern.findall(html):
        print(f'{title}: {href}')

This example from the source material demonstrates fetching a page and extracting anchor tags using re.findall() for immediate practical results.

Day 63 – Concurrent Programming for High-Performance Crawlers

File Day61-65/63.并发编程在爬虫中的应用.md addresses the I/O-bound nature of network requests. The lesson explains why threading and asyncio suit web scraping (network latency dominates execution time), contrasting this with CPU-bound tasks that require multiprocessing.

The curriculum demonstrates ThreadPoolExecutor for parallel page fetching, showing how to maintain polite crawl delays (time.sleep(random.random() * 2 + 1)) while maximizing throughput:

import time, random, requests, re
from concurrent.futures import ThreadPoolExecutor

def fetch_page(page):
    url = f'https://movie.douban.com/top250?start={(page - 1) * 25}'
    resp = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
    if resp.status_code == 200:
        titles = re.findall(r'<span class="title">([^&]*?)</span>', resp.text)
        ranks = re.findall(r'<span class="rating_num".*?>(.*?)</span>', resp.text)
        for t, r in zip(titles, ranks):
            print(t, r)

with ThreadPoolExecutor(max_workers=8) as pool:
    for p in range(1, 11):
        pool.submit(fetch_page, p)
        time.sleep(random.random() * 2 + 1)

This pattern teaches GIL-friendly concurrency for crawling workloads, allowing learners to scale beyond single-threaded performance without introducing process overhead.

Day 64 – Dynamic Content with Selenium

File Day61-65/64.使用Selenium抓取网页动态内容.md tackles JavaScript-rendered content that requests cannot handle. The lesson covers ChromeDriver installation, element location strategies (find_element and find_elements), and synchronization techniques including implicit waits and explicit WebDriverWait conditions.

The curriculum demonstrates headless mode operation and anti-detection evasion techniques. A complete example combines Selenium with concurrent downloads to scrape lazy-loaded image galleries:

import os, time, requests
from concurrent.futures import ThreadPoolExecutor
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

DOWNLOAD_PATH = 'images/'
os.makedirs(DOWNLOAD_PATH, exist_ok=True)

browser = webdriver.Chrome()
browser.get('https://image.so.com/z?ch=beauty')
browser.implicitly_wait(10)

kw = browser.find_element(By.CSS_SELECTOR, 'input[name=q]')
kw.send_keys('风景')
kw.send_keys(Keys.ENTER)

for _ in range(10):
    browser.execute_script('document.documentElement.scrollTop = document.documentElement.scrollHeight')
    time.sleep(1)

imgs = browser.find_elements(By.CSS_SELECTOR, 'div.waterfall img')

def download(url):
    data = requests.get(url).content
    name = os.path.join(DOWNLOAD_PATH, url.split('/')[-1])
    with open(name, 'wb') as f:
        f.write(data)

with ThreadPoolExecutor(max_workers=16) as pool:
    for img in imgs:
        pool.submit(download, img.get_attribute('src'))
browser.quit()

This example teaches browser automation, JavaScript execution context, and hybrid architectures where Selenium extracts URLs while requests handles binary downloads.

Day 65 – Production-Grade Frameworks with Scrapy

File Day61-65/65.爬虫框架Scrapy简介.md transitions from manual scripts to the Scrapy framework. The lesson details Scrapy’s component architecture: engine, scheduler, downloader, spiders, item pipelines, and extensions.

The curriculum provides a minimal spider implementation demonstrating Scrapy’s declarative approach:

import scrapy

class DoubanSpider(scrapy.Spider):
    name = 'douban'
    start_urls = [f'https://movie.douban.com/top250?start={i*25}'
                  for i in range(10)]

    def parse(self, response):
        for sel in response.css('div.item'):
            title = sel.css('span.title::text').get()
            rating = sel.css('span.rating_num::text').get()
            yield {'title': title, 'rating': rating}

Run via scrapy crawl douban -o top250.json, this example teaches the parse() callback mechanism, CSS selector extraction, and Scrapy’s built-in feed exports. The lesson emphasizes how Scrapy abstracts concurrency, request scheduling, and data persistence that Days 62-63 implemented manually.

Summary

• Python-100-Days structures web scraping across five sequential days (61-65) in the Day61-65/ directory, creating a pedagogical progression from fundamentals to frameworks.
• Day 62 establishes HTTP fundamentals using requests with proper header handling, while Day 63 scales these skills using ThreadPoolExecutor for I/O-bound concurrency.
• Day 64 introduces Selenium for JavaScript-rendered content, teaching real browser automation, element waits, and headless execution.
• Day 65 transitions to Scrapy, demonstrating production-grade architecture with built-in pipelines, schedulers, and item processing.
• The curriculum embeds ethical considerations and robots.txt compliance from Day 61, ensuring learners understand legal boundaries alongside technical implementation.

Frequently Asked Questions

What Python libraries does Python-100-Days use for web scraping?

The curriculum teaches requests for HTTP fetching, re (regular expressions) for initial parsing demonstrations, concurrent.futures for threading scalability, selenium for dynamic content automation, and Scrapy for production-grade framework development. This progression allows learners to understand low-level HTTP mechanics before relying on high-level abstractions.

How does the course handle websites that require JavaScript rendering?

Day 64 (64.使用Selenium抓取网页动态内容.md) dedicates an entire lesson to Selenium WebDriver, teaching learners to control Chrome programmatically. The content covers executing JavaScript via browser.execute_script(), handling implicit waits for AJAX content, and scrolling to trigger lazy-loading mechanisms before element extraction.

Does Python-100-Days cover ethical or legal aspects of web scraping?

Yes. Day 61 (61.网络数据采集概述.md) explicitly addresses crawler ethics, the purpose of robots.txt, and legal considerations regarding data collection. The curriculum emphasizes that technical capability does not imply permission, establishing responsible scraping practices before introducing download implementations.

What concurrency approach does the curriculum recommend for high-volume crawling?

The repository teaches a tiered approach: ThreadPoolExecutor for moderate-scale static scraping (Day 63), explaining that threading suits I/O-bound network workloads despite Python’s GIL. For production scale, Day 65 transitions to Scrapy, which uses its own asynchronous Twisted engine to handle thousands of concurrent requests without manual thread management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}