# How book-to-skill Handles Concurrent Extractions: A Deep Dive into Parallel Processing

> Discover how book-to-skill tackles concurrent extractions using Python's concurrent futures and worker pools for efficient, isolated processing and thread-safe output.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-28

---

**book-to-skill processes multiple documents simultaneously by delegating extraction tasks to worker pools via Python's `concurrent.futures` module, ensuring isolated execution and thread-safe output writes.**

The `virgiliojr94/book-to-skill` repository implements a robust parallel extraction pipeline designed to convert books and documents into structured skill data efficiently. By leveraging process-based parallelism alongside file-locking mechanisms, the tool maximizes CPU utilization while preventing data corruption during concurrent writes. This architecture ensures that extracting skills from large document collections remains both fast and reliable.

## Core Concurrency Architecture

At the heart of `book-to-skill`'s parallel processing lies the **`ProcessPoolExecutor`** from Python's standard library, implemented in **[`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py)**. The system defaults to creating a worker pool sized to match the number of available CPU cores, with an automatic fallback to **`ThreadPoolExecutor`** on platforms where process spawning carries high overhead.

When the CLI receives an extraction command, the script constructs a list of independent jobs—one per input file or URL. These jobs are distributed across the pool using **`executor.map()`**, which assigns each document to an available worker process. Each worker operates in complete isolation, receiving its own copy of the input data and executing the deterministic parsers located in **`book_to_skill/parsers/`** without accessing shared mutable state.

## Thread-Safe Output with File Locking

To prevent race conditions when multiple workers finish simultaneously and attempt to write to the output directory, `book-to-skill` implements a file-based locking mechanism. The **`safe_write()`** function in **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** utilizes **`portalocker`** to serialize access to the filesystem.

```python

# From book_to_skill/utils.py

import portalocker

def safe_write(path, data):
    with portalocker.Lock(path + ".lock"):
        with open(path, "w", encoding="utf-8") as f:
            f.write(data)

```

This locking strategy guarantees that even if extraction jobs complete concurrently, only one worker holds the write lock at any given moment. The temporary lock file ensures output integrity without significantly impacting throughput, as the lock duration covers only the final write operation rather than the entire extraction process.

## Error Isolation and Graceful Degradation

The architecture implements a **fail-fast-per-job** strategy that isolates failures to individual documents. If a worker encounters an exception—such as a malformed PDF or corrupted HTML—the `ProcessPoolExecutor` captures the error in the main process without terminating the entire batch.

Errors are logged on a per-job basis, allowing valid extractions to continue processing while the system records specific failure details. This approach ensures that a single problematic document cannot abort the extraction of an entire dataset, making the pipeline suitable for processing large, heterogeneous collections of source material.

## Configuring Concurrent Extractions

Users can control parallelism through the command-line interface by specifying the **`--workers`** parameter. The default configuration, defined in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)**, automatically detects available CPU cores to optimize throughput.

```bash

# Run extraction with 8 concurrent workers

python -m book_to_skill extract \
    --input-dir books/ \
    --output-dir skills/ \
    --workers 8

```

For programmatic usage, the extraction pipeline accepts a workers argument that propagates to the executor initialization:

```python

# Simplified workflow from scripts/extract.py

from concurrent.futures import ProcessPoolExecutor
from book_to_skill.utils import run_extractor, safe_write

def main(args):
    jobs = build_job_list(args.input_dir)  # One job per source file

    
    with ProcessPoolExecutor(max_workers=args.workers) as executor:
        for result in executor.map(run_extractor, jobs):
            # run_extractor returns (path, extracted_data)

            safe_write(*result)  # Atomic write with file locking

```

## Summary

- **Process-based parallelism** via `ProcessPoolExecutor` scales worker pools to match CPU core counts for maximum throughput.
- **Deterministic isolation** ensures each worker operates on independent data copies without shared mutable state conflicts.
- **File-locking protection** using `portalocker` in `safe_write()` prevents race conditions during concurrent filesystem access.
- **Graceful error handling** isolates failures to individual jobs, allowing batch extractions to continue despite individual document errors.
- **Configurable concurrency** through CLI flags or programmatic arguments lets users optimize resource utilization for their specific hardware.

## Frequently Asked Questions

### What concurrency model does book-to-skill use?

`book-to-skill` utilizes Python's **`concurrent.futures`** module, specifically implementing a **`ProcessPoolExecutor`** to distribute extraction tasks across multiple CPU cores. This process-based approach circumvents Python's Global Interpreter Lock (GIL) and provides true parallelism for CPU-intensive parsing operations, with an automatic fallback to `ThreadPoolExecutor` on platforms where process spawning is expensive.

### How does book-to-skill prevent file corruption during concurrent writes?

The system prevents corruption through a file-locking mechanism implemented in **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)**. The **`safe_write()`** function wraps file operations with **`portalocker.Lock()`**, creating a temporary lock file that serializes write access. This ensures that even when multiple workers complete extraction simultaneously, only one process writes to the target file at any given moment.

### What happens if one extraction job fails?

When an individual worker raises an exception—such as encountering a malformed PDF—the `ProcessPoolExecutor` captures the error in the main process while allowing other jobs to continue executing. This **fail-fast-per-job** strategy logs the specific error for the problematic document without aborting the entire batch, ensuring robust processing of large document collections containing mixed-quality inputs.

### Can I control the number of concurrent workers?

Yes, `book-to-skill` exposes the **`--workers`** CLI flag to configure the size of the process pool, with the default value automatically set to the number of available CPU cores as defined in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)**. This allows users to balance resource utilization against system load, reducing worker counts on memory-constrained systems or increasing them for high-performance servers.