# How Python-100-Days Teaches Web Scraping: A Progressive 5-Day Curriculum

> Learn web scraping with Python-100-Days Days 61-65. Progress from basic requests to Scrapy, mastering concurrency, automation, and crawler design for real-world applications.

- Repository: [骆昊/Python-100-Days](https://github.com/jackfrued/Python-100-Days)
- Tags: tutorial
- Published: 2026-02-24

---

**Python-100-Days teaches web scraping through Days 61-65, a structured progression that moves from basic HTTP requests to production-grade Scrapy frameworks, covering concurrency patterns, browser automation, and crawler architecture.**

The `jackfrued/Python-100-Days` repository is one of GitHub’s most comprehensive Chinese-language Python tutorials. Its web scraping module spans five consecutive days (Days 61-65) and employs a layered learning approach that introduces complexity gradually while emphasizing legal ethics and architectural best practices. This guide examines the exact source files, coding patterns, and progressive concepts used to teach these skills.

## The Five-Day Web Scraping Track (Days 61-65)

The curriculum structures web scraping education as a **three-stage pipeline** (download → parse → store) that evolves from simple scripts to enterprise frameworks. Each day corresponds to a specific markdown file in the `Day61-65/` directory.

### Day 61 – Web Crawler Architecture and Ethics

According to `Day61-65/61.网络数据采集概述.md`, the module begins with foundational concepts rather than immediate coding. The lesson defines the **classic crawler architecture** and Python’s suitability for network data collection. It establishes the three-stage pipeline workflow and emphasizes critical legal considerations, including [`robots.txt`](https://github.com/jackfrued/Python-100-Days/blob/main/robots.txt) compliance and ethical crawling boundaries, before introducing any implementation details.

### Day 62 – Acquiring Network Resources with Requests

File `Day61-65/62.用Python获取网络资源-1.md` introduces static page fetching using the **requests** library. The lesson demonstrates `requests.get()` calls, proper `status_code` handling, and the distinction between `resp.text` (decoded HTML) and `resp.content` (raw binary bytes). It stresses the importance of custom `User-Agent` headers to prevent server blocking.

The curriculum teaches basic extraction using regular expressions before introducing BeautifulSoup or parsing libraries:

```python
import requests
import re

url = 'https://www.sohu.com/'
resp = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

if resp.status_code == 200:
    html = resp.text
    pattern = re.compile(r'<a.*?href="(.*?)".*?title="(.*?)".*?>')
    for href, title in pattern.findall(html):
        print(f'{title}: {href}')

```

This example from the source material demonstrates fetching a page and extracting anchor tags using `re.findall()` for immediate practical results.

### Day 63 – Concurrent Programming for High-Performance Crawlers

File `Day61-65/63.并发编程在爬虫中的应用.md` addresses the I/O-bound nature of network requests. The lesson explains why **threading** and **asyncio** suit web scraping (network latency dominates execution time), contrasting this with CPU-bound tasks that require multiprocessing.

The curriculum demonstrates **ThreadPoolExecutor** for parallel page fetching, showing how to maintain polite crawl delays (`time.sleep(random.random() * 2 + 1)`) while maximizing throughput:

```python
import time, random, requests, re
from concurrent.futures import ThreadPoolExecutor

def fetch_page(page):
    url = f'https://movie.douban.com/top250?start={(page - 1) * 25}'
    resp = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
    if resp.status_code == 200:
        titles = re.findall(r'<span class="title">([^&]*?)</span>', resp.text)
        ranks = re.findall(r'<span class="rating_num".*?>(.*?)</span>', resp.text)
        for t, r in zip(titles, ranks):
            print(t, r)

with ThreadPoolExecutor(max_workers=8) as pool:
    for p in range(1, 11):
        pool.submit(fetch_page, p)
        time.sleep(random.random() * 2 + 1)

```

This pattern teaches GIL-friendly concurrency for crawling workloads, allowing learners to scale beyond single-threaded performance without introducing process overhead.

### Day 64 – Dynamic Content with Selenium

File `Day61-65/64.使用Selenium抓取网页动态内容.md` tackles JavaScript-rendered content that `requests` cannot handle. The lesson covers ChromeDriver installation, element location strategies (`find_element` and `find_elements`), and synchronization techniques including implicit waits and explicit `WebDriverWait` conditions.

The curriculum demonstrates **headless mode** operation and anti-detection evasion techniques. A complete example combines Selenium with concurrent downloads to scrape lazy-loaded image galleries:

```python
import os, time, requests
from concurrent.futures import ThreadPoolExecutor
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

DOWNLOAD_PATH = 'images/'
os.makedirs(DOWNLOAD_PATH, exist_ok=True)

browser = webdriver.Chrome()
browser.get('https://image.so.com/z?ch=beauty')
browser.implicitly_wait(10)

kw = browser.find_element(By.CSS_SELECTOR, 'input[name=q]')
kw.send_keys('风景')
kw.send_keys(Keys.ENTER)

for _ in range(10):
    browser.execute_script('document.documentElement.scrollTop = document.documentElement.scrollHeight')
    time.sleep(1)

imgs = browser.find_elements(By.CSS_SELECTOR, 'div.waterfall img')

def download(url):
    data = requests.get(url).content
    name = os.path.join(DOWNLOAD_PATH, url.split('/')[-1])
    with open(name, 'wb') as f:
        f.write(data)

with ThreadPoolExecutor(max_workers=16) as pool:
    for img in imgs:
        pool.submit(download, img.get_attribute('src'))
browser.quit()

```

This example teaches browser automation, JavaScript execution context, and hybrid architectures where Selenium extracts URLs while `requests` handles binary downloads.

### Day 65 – Production-Grade Frameworks with Scrapy

File `Day61-65/65.爬虫框架Scrapy简介.md` transitions from manual scripts to the **Scrapy** framework. The lesson details Scrapy’s component architecture: **engine**, **scheduler**, **downloader**, **spiders**, **item pipelines**, and **extensions**.

The curriculum provides a minimal spider implementation demonstrating Scrapy’s declarative approach:

```python
import scrapy

class DoubanSpider(scrapy.Spider):
    name = 'douban'
    start_urls = [f'https://movie.douban.com/top250?start={i*25}'
                  for i in range(10)]

    def parse(self, response):
        for sel in response.css('div.item'):
            title = sel.css('span.title::text').get()
            rating = sel.css('span.rating_num::text').get()
            yield {'title': title, 'rating': rating}

```

Run via `scrapy crawl douban -o top250.json`, this example teaches the `parse()` callback mechanism, CSS selector extraction, and Scrapy’s built-in feed exports. The lesson emphasizes how Scrapy abstracts concurrency, request scheduling, and data persistence that Days 62-63 implemented manually.

## Summary

- **Python-100-Days** structures web scraping across five sequential days (61-65) in the `Day61-65/` directory, creating a pedagogical progression from fundamentals to frameworks.
- **Day 62** establishes HTTP fundamentals using `requests` with proper header handling, while **Day 63** scales these skills using `ThreadPoolExecutor` for I/O-bound concurrency.
- **Day 64** introduces **Selenium** for JavaScript-rendered content, teaching real browser automation, element waits, and headless execution.
- **Day 65** transitions to **Scrapy**, demonstrating production-grade architecture with built-in pipelines, schedulers, and item processing.
- The curriculum embeds ethical considerations and [`robots.txt`](https://github.com/jackfrued/Python-100-Days/blob/main/robots.txt) compliance from Day 61, ensuring learners understand legal boundaries alongside technical implementation.

## Frequently Asked Questions

### What Python libraries does Python-100-Days use for web scraping?

The curriculum teaches **requests** for HTTP fetching, **re** (regular expressions) for initial parsing demonstrations, **concurrent.futures** for threading scalability, **selenium** for dynamic content automation, and **Scrapy** for production-grade framework development. This progression allows learners to understand low-level HTTP mechanics before relying on high-level abstractions.

### How does the course handle websites that require JavaScript rendering?

Day 64 (`64.使用Selenium抓取网页动态内容.md`) dedicates an entire lesson to **Selenium WebDriver**, teaching learners to control Chrome programmatically. The content covers executing JavaScript via `browser.execute_script()`, handling implicit waits for AJAX content, and scrolling to trigger lazy-loading mechanisms before element extraction.

### Does Python-100-Days cover ethical or legal aspects of web scraping?

Yes. Day 61 (`61.网络数据采集概述.md`) explicitly addresses crawler ethics, the purpose of [`robots.txt`](https://github.com/jackfrued/Python-100-Days/blob/main/robots.txt), and legal considerations regarding data collection. The curriculum emphasizes that technical capability does not imply permission, establishing responsible scraping practices before introducing download implementations.

### What concurrency approach does the curriculum recommend for high-volume crawling?

The repository teaches a tiered approach: **ThreadPoolExecutor** for moderate-scale static scraping (Day 63), explaining that threading suits I/O-bound network workloads despite Python’s GIL. For production scale, Day 65 transitions to **Scrapy**, which uses its own asynchronous Twisted engine to handle thousands of concurrent requests without manual thread management.