# How book-to-skill Handles Concurrent Extractions to Prevent Data Loss

> Learn how book-to-skill prevents data loss with concurrent extractions. Discover its unique PID-based directory isolation for safe, simultaneous processing and to avoid overwrites.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-30

---

**`book-to-skill` prevents data loss during concurrent extractions by isolating each run in a unique, process-specific temporary directory derived from the current PID, ensuring simultaneous processes never overwrite each other's intermediate files.**

The `book-to-skill` repository solves a critical race-condition problem that occurs when multiple book extractions run simultaneously. By default, the tool routes each extraction to a distinct workspace based on the operating system process ID, eliminating file collisions that previously destroyed data when runs shared a single fixed temporary path.

## Per-Run Isolation via PID-Based Directories

The core concurrency safety mechanism resides in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)**, specifically within the `default_output_dir()` function. Instead of using a static temporary folder, the function dynamically generates a unique directory name for every process.

### The default_output_dir() Implementation

The implementation combines Python’s `tempfile` module with the current process identifier:

```python
return Path(tempfile.gettempdir()) / f"book_skill_work-{os.getpid()}"

```

This construction guarantees that two simultaneous extractions receive different directories because `os.getpid()` returns a unique integer for every process. Consequently, concurrent runs write to paths like `/tmp/book_skill_work-1111/` and `/tmp/book_skill_work-2222/` rather than competing for a single shared location.

### Legacy Path Compatibility

The new PID-based directory is deliberately positioned as a **sibling** to the legacy fixed path (`$TMPDIR/book_skill_work`). This architectural choice prevents legacy cleanup scripts—which might indiscriminately delete the old folder—from accidentally wiping out active extraction data. By avoiding a parent-child relationship with the historic path, the system ensures that deletion of the legacy directory never cascades into the new per-run workspaces.

## Custom Work Directory Configuration

Users retain full control over extraction locations through the **`BOOK_SKILL_WORKDIR`** environment variable. When this variable is set to a non-empty string, `book-to-skill` bypasses the PID-based default and writes all output to the specified path instead.

The configuration logic explicitly guards against empty-string overrides. If `BOOK_SKILL_WORKDIR` is unset or empty, the system falls back to the safe default, preventing the common pitfall of resolving an empty string as the current working directory.

```bash
export BOOK_SKILL_WORKDIR=/tmp/my_private_workdir
python -m book_to_skill mybook.pdf

```

## Verifying Concurrent Safety in Tests

The test suite in **[`tests/test_per_run_workdir.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_per_run_workdir.py)** validates the concurrency protections through four specific assertions:

- **`test_default_workdir_is_unique_per_process`** – Confirms that different PIDs generate distinct directory paths, preventing collision between simultaneous runs.
- **`test_default_workdir_is_sibling_not_child_of_legacy_path`** – Ensures the new directory structure sits alongside rather than beneath the legacy path, protecting against accidental deletion by legacy cleanup routines.
- **`test_default_workdir_lives_in_tempdir`** – Verifies that default output always resides within the system temporary folder (`tempfile.gettempdir()`).
- **`test_env_override_wins`** and **`test_empty_env_override_does_not_become_cwd`** – Validate that the `BOOK_SKILL_WORKDIR` environment variable correctly overrides defaults while rejecting empty values that would otherwise resolve to the current directory.

## Practical Examples

### Running Concurrent Extractions

To execute multiple extractions simultaneously without data loss, simply invoke the module from separate processes. Each automatically receives its own workspace:

```python
import subprocess
import time

# First extraction instance

p1 = subprocess.Popen(["python", "-m", "book_to_skill", "book_one.pdf"])

# Second overlapping extraction

time.sleep(0.1)
p2 = subprocess.Popen(["python", "-m", "book_to_skill", "book_two.pdf"])

```

Each process writes to isolated directories such as:

```

/tmp/book_skill_work-1234/full_text.txt
/tmp/book_skill_work-5678/full_text.txt

```

### Configuring a Custom Work Directory

For predictable output locations in single-run scenarios, override the default:

```bash
export BOOK_SKILL_WORKDIR=/var/extraction_workspace
python -m book_to_skill document.pdf

```

This writes all files to `/var/extraction_workspace/` rather than generating a PID-suffixed folder.

## Summary

- **`book-to-skill`** assigns every extraction a unique temporary directory based on `os.getpid()` to guarantee isolation during concurrent runs.
- The **`default_output_dir()`** function in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) constructs paths inside `tempfile.gettempdir()` using the format `book_skill_work-{PID}`.
- Directory placement as a **sibling** to the legacy path protects active extractions from legacy cleanup scripts.
- The **`BOOK_SKILL_WORKDIR`** environment variable allows custom path selection while safely handling empty values.
- The test suite in [`tests/test_per_run_workdir.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_per_run_workdir.py) verifies uniqueness, location constraints, and override behavior.

## Frequently Asked Questions

### How does book-to-skill prevent file collisions when extracting multiple books simultaneously?

The tool generates a unique working directory for each process by appending the current PID to the base directory name in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py). Because operating systems guarantee unique process IDs, simultaneous runs write to distinct folders like `/tmp/book_skill_work-1111/` and `/tmp/book_skill_work-2222/`, making overwrites impossible.

### Can I specify a custom directory instead of the default PID-based temporary folder?

Yes. Set the **`BOOK_SKILL_WORKDIR`** environment variable to your desired path before running the extraction. The system will use that location exclusively and skip the PID-based naming. If you unset the variable or leave it empty, the tool reverts to the safe default behavior.

### Why is the new directory a sibling of the legacy path rather than a child directory?

The sibling relationship shields active extractions from legacy cleanup routines that might delete the old fixed path (`$TMPDIR/book_skill_work`). By placing new directories alongside rather than inside the legacy folder, the code ensures that deletion of the historic directory never destroys files from currently running processes.

### What happens if two extractions accidentally use the same BOOK_SKILL_WORKDIR?

If you explicitly set identical paths via `BOOK_SKILL_WORKDIR`, the tool respects your configuration and both runs will share the directory, risking data loss. The PID-based protection only applies to the default automatic directory generation. Always use unique custom paths when running concurrent extractions with manual overrides.