How book-to-skill Handles Concurrent Extractions: A Deep Dive into Parallel Processing

book-to-skill processes multiple documents simultaneously by delegating extraction tasks to worker pools via Python's concurrent.futures module, ensuring isolated execution and thread-safe output writes.

The virgiliojr94/book-to-skill repository implements a robust parallel extraction pipeline designed to convert books and documents into structured skill data efficiently. By leveraging process-based parallelism alongside file-locking mechanisms, the tool maximizes CPU utilization while preventing data corruption during concurrent writes. This architecture ensures that extracting skills from large document collections remains both fast and reliable.

Core Concurrency Architecture

At the heart of book-to-skill's parallel processing lies the ProcessPoolExecutor from Python's standard library, implemented in scripts/extract.py. The system defaults to creating a worker pool sized to match the number of available CPU cores, with an automatic fallback to ThreadPoolExecutor on platforms where process spawning carries high overhead.

When the CLI receives an extraction command, the script constructs a list of independent jobs—one per input file or URL. These jobs are distributed across the pool using executor.map(), which assigns each document to an available worker process. Each worker operates in complete isolation, receiving its own copy of the input data and executing the deterministic parsers located in book_to_skill/parsers/ without accessing shared mutable state.

Thread-Safe Output with File Locking

To prevent race conditions when multiple workers finish simultaneously and attempt to write to the output directory, book-to-skill implements a file-based locking mechanism. The safe_write() function in book_to_skill/utils.py utilizes portalocker to serialize access to the filesystem.

# From book_to_skill/utils.py

import portalocker

def safe_write(path, data):
    with portalocker.Lock(path + ".lock"):
        with open(path, "w", encoding="utf-8") as f:
            f.write(data)

This locking strategy guarantees that even if extraction jobs complete concurrently, only one worker holds the write lock at any given moment. The temporary lock file ensures output integrity without significantly impacting throughput, as the lock duration covers only the final write operation rather than the entire extraction process.

Error Isolation and Graceful Degradation

The architecture implements a fail-fast-per-job strategy that isolates failures to individual documents. If a worker encounters an exception—such as a malformed PDF or corrupted HTML—the ProcessPoolExecutor captures the error in the main process without terminating the entire batch.

Errors are logged on a per-job basis, allowing valid extractions to continue processing while the system records specific failure details. This approach ensures that a single problematic document cannot abort the extraction of an entire dataset, making the pipeline suitable for processing large, heterogeneous collections of source material.

Configuring Concurrent Extractions

Users can control parallelism through the command-line interface by specifying the --workers parameter. The default configuration, defined in book_to_skill/config.py, automatically detects available CPU cores to optimize throughput.

# Run extraction with 8 concurrent workers

python -m book_to_skill extract \
    --input-dir books/ \
    --output-dir skills/ \
    --workers 8

For programmatic usage, the extraction pipeline accepts a workers argument that propagates to the executor initialization:

# Simplified workflow from scripts/extract.py

from concurrent.futures import ProcessPoolExecutor
from book_to_skill.utils import run_extractor, safe_write

def main(args):
    jobs = build_job_list(args.input_dir)  # One job per source file

    
    with ProcessPoolExecutor(max_workers=args.workers) as executor:
        for result in executor.map(run_extractor, jobs):
            # run_extractor returns (path, extracted_data)

            safe_write(*result)  # Atomic write with file locking

Summary

• Process-based parallelism via ProcessPoolExecutor scales worker pools to match CPU core counts for maximum throughput.
• Deterministic isolation ensures each worker operates on independent data copies without shared mutable state conflicts.
• File-locking protection using portalocker in safe_write() prevents race conditions during concurrent filesystem access.
• Graceful error handling isolates failures to individual jobs, allowing batch extractions to continue despite individual document errors.
• Configurable concurrency through CLI flags or programmatic arguments lets users optimize resource utilization for their specific hardware.

Frequently Asked Questions

What concurrency model does book-to-skill use?

book-to-skill utilizes Python's concurrent.futures module, specifically implementing a ProcessPoolExecutor to distribute extraction tasks across multiple CPU cores. This process-based approach circumvents Python's Global Interpreter Lock (GIL) and provides true parallelism for CPU-intensive parsing operations, with an automatic fallback to ThreadPoolExecutor on platforms where process spawning is expensive.

How does book-to-skill prevent file corruption during concurrent writes?

The system prevents corruption through a file-locking mechanism implemented in book_to_skill/utils.py. The safe_write() function wraps file operations with portalocker.Lock(), creating a temporary lock file that serializes write access. This ensures that even when multiple workers complete extraction simultaneously, only one process writes to the target file at any given moment.

What happens if one extraction job fails?

When an individual worker raises an exception—such as encountering a malformed PDF—the ProcessPoolExecutor captures the error in the main process while allowing other jobs to continue executing. This fail-fast-per-job strategy logs the specific error for the problematic document without aborting the entire batch, ensuring robust processing of large document collections containing mixed-quality inputs.

Can I control the number of concurrent workers?

Yes, book-to-skill exposes the --workers CLI flag to configure the size of the process pool, with the default value automatically set to the number of available CPU cores as defined in book_to_skill/config.py. This allows users to balance resource utilization against system load, reducing worker counts on memory-constrained systems or increasing them for high-performance servers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}