How book-to-skill Handles Concurrent Extractions to Prevent Data Loss

book-to-skill prevents data loss during concurrent extractions by isolating each run in a unique, process-specific temporary directory derived from the current PID, ensuring simultaneous processes never overwrite each other's intermediate files.

The book-to-skill repository solves a critical race-condition problem that occurs when multiple book extractions run simultaneously. By default, the tool routes each extraction to a distinct workspace based on the operating system process ID, eliminating file collisions that previously destroyed data when runs shared a single fixed temporary path.

Per-Run Isolation via PID-Based Directories

The core concurrency safety mechanism resides in book_to_skill/config.py, specifically within the default_output_dir() function. Instead of using a static temporary folder, the function dynamically generates a unique directory name for every process.

The default_output_dir() Implementation

The implementation combines Python’s tempfile module with the current process identifier:

return Path(tempfile.gettempdir()) / f"book_skill_work-{os.getpid()}"

This construction guarantees that two simultaneous extractions receive different directories because os.getpid() returns a unique integer for every process. Consequently, concurrent runs write to paths like /tmp/book_skill_work-1111/ and /tmp/book_skill_work-2222/ rather than competing for a single shared location.

Legacy Path Compatibility

The new PID-based directory is deliberately positioned as a sibling to the legacy fixed path ($TMPDIR/book_skill_work). This architectural choice prevents legacy cleanup scripts—which might indiscriminately delete the old folder—from accidentally wiping out active extraction data. By avoiding a parent-child relationship with the historic path, the system ensures that deletion of the legacy directory never cascades into the new per-run workspaces.

Custom Work Directory Configuration

Users retain full control over extraction locations through the BOOK_SKILL_WORKDIR environment variable. When this variable is set to a non-empty string, book-to-skill bypasses the PID-based default and writes all output to the specified path instead.

The configuration logic explicitly guards against empty-string overrides. If BOOK_SKILL_WORKDIR is unset or empty, the system falls back to the safe default, preventing the common pitfall of resolving an empty string as the current working directory.

export BOOK_SKILL_WORKDIR=/tmp/my_private_workdir
python -m book_to_skill mybook.pdf

Verifying Concurrent Safety in Tests

The test suite in tests/test_per_run_workdir.py validates the concurrency protections through four specific assertions:

• test_default_workdir_is_unique_per_process – Confirms that different PIDs generate distinct directory paths, preventing collision between simultaneous runs.
• test_default_workdir_is_sibling_not_child_of_legacy_path – Ensures the new directory structure sits alongside rather than beneath the legacy path, protecting against accidental deletion by legacy cleanup routines.
• test_default_workdir_lives_in_tempdir – Verifies that default output always resides within the system temporary folder (tempfile.gettempdir()).
• test_env_override_wins and test_empty_env_override_does_not_become_cwd – Validate that the BOOK_SKILL_WORKDIR environment variable correctly overrides defaults while rejecting empty values that would otherwise resolve to the current directory.

Practical Examples

Running Concurrent Extractions

To execute multiple extractions simultaneously without data loss, simply invoke the module from separate processes. Each automatically receives its own workspace:

import subprocess
import time

# First extraction instance

p1 = subprocess.Popen(["python", "-m", "book_to_skill", "book_one.pdf"])

# Second overlapping extraction

time.sleep(0.1)
p2 = subprocess.Popen(["python", "-m", "book_to_skill", "book_two.pdf"])

Each process writes to isolated directories such as:

/tmp/book_skill_work-1234/full_text.txt
/tmp/book_skill_work-5678/full_text.txt

Configuring a Custom Work Directory

For predictable output locations in single-run scenarios, override the default:

export BOOK_SKILL_WORKDIR=/var/extraction_workspace
python -m book_to_skill document.pdf

This writes all files to /var/extraction_workspace/ rather than generating a PID-suffixed folder.

Summary

• book-to-skill assigns every extraction a unique temporary directory based on os.getpid() to guarantee isolation during concurrent runs.
• The default_output_dir() function in book_to_skill/config.py constructs paths inside tempfile.gettempdir() using the format book_skill_work-{PID}.
• Directory placement as a sibling to the legacy path protects active extractions from legacy cleanup scripts.
• The BOOK_SKILL_WORKDIR environment variable allows custom path selection while safely handling empty values.
• The test suite in tests/test_per_run_workdir.py verifies uniqueness, location constraints, and override behavior.

Frequently Asked Questions

How does book-to-skill prevent file collisions when extracting multiple books simultaneously?

The tool generates a unique working directory for each process by appending the current PID to the base directory name in book_to_skill/config.py. Because operating systems guarantee unique process IDs, simultaneous runs write to distinct folders like /tmp/book_skill_work-1111/ and /tmp/book_skill_work-2222/, making overwrites impossible.

Can I specify a custom directory instead of the default PID-based temporary folder?

Yes. Set the BOOK_SKILL_WORKDIR environment variable to your desired path before running the extraction. The system will use that location exclusively and skip the PID-based naming. If you unset the variable or leave it empty, the tool reverts to the safe default behavior.

Why is the new directory a sibling of the legacy path rather than a child directory?

The sibling relationship shields active extractions from legacy cleanup routines that might delete the old fixed path ($TMPDIR/book_skill_work). By placing new directories alongside rather than inside the legacy folder, the code ensures that deletion of the historic directory never destroys files from currently running processes.

What happens if two extractions accidentally use the same BOOK_SKILL_WORKDIR?

If you explicitly set identical paths via BOOK_SKILL_WORKDIR, the tool respects your configuration and both runs will share the directory, risking data loss. The PID-based protection only applies to the default automatic directory generation. Always use unique custom paths when running concurrent extractions with manual overrides.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}