Book-to-Skill Extraction Pipeline Output: The Three Generated Artifacts

The book-to-skill extraction pipeline generates three concrete artifacts: a consolidated plain-text corpus file, a structured metadata JSON file, and a human-readable summary printed to stdout.

The virgiliojr94/book-to-skill repository processes PDF, EPUB, DOCX, HTML, and plain-text documents through a coordinated pipeline spanning scripts/extract.py, book_to_skill/cli.py, and book_to_skill/utils.py. Each execution produces standardized outputs that enable downstream skill generation while preserving raw text for custom processing workflows.

The Three Core Extraction Pipeline Outputs

Combined Plain-Text Corpus

The pipeline extracts and consolidates text from all supplied documents into a single searchable file. According to book_to_skill/utils.py (lines 822-825), the system writes the consolidated_text variable to the path defined by OUTPUT_TEXT in book_to_skill/config.py.

This output contains the extracted text from every input source, separated by clear banner lines that demarcate document boundaries. The default location is book_to_skill/output/full_text.txt, though this is configurable through the central configuration module.

Metadata JSON File

The structured metadata output provides machine-readable documentation of the extraction process. Written to OUTPUT_META (typically book_to_skill/output/metadata.json) after the extraction loop completes, this file is generated in book_to_skill/utils.py (lines 1095-1110).

The JSON structure contains two primary sections:

• Per-source fields: source_file, format, extraction_method, pages, chars, words, estimated_tokens, chapters_detected, and has_toc
• Aggregate statistics: total_sources, file_size_mb, total_pages, and estimated_tokens_human

This metadata enables LLM cost pre-flight calculations and provides provenance tracking for each processed document.

Human-Readable Summary

After metadata serialization, the main() function in book_to_skill/utils.py (lines 1160-1170) prints a concise extraction report to stdout. This summary displays the number of sources processed, total file size, page count, word count, token estimate, detected chapter count (including the detection method), and table-of-contents status.

This immediate feedback allows operators to verify extraction success without inspecting output files directly.

Output Configuration and File Paths

All output paths are centralized in book_to_skill/config.py through the constants OUTPUT_DIR, OUTPUT_TEXT, and OUTPUT_META. The pipeline creates these locations automatically during the initialization phase defined in the utility module.

The default output directory structure follows this pattern:

book_to_skill/output/
├── full_text.txt
└── metadata.json

How to Generate the Pipeline Output

Command-Line Usage

Invoke the extraction pipeline through the CLI entry point to generate all three outputs simultaneously:

# Example: invoking the pipeline from the command line

$ book-to-skill mybook.pdf --mode technical

This command executes the full workflow defined in scripts/extract.py → book_to_skill/cli.py → book_to_skill/utils.py, producing:

• Extracted text written to book_to_skill/output/full_text.txt
• Metadata written to book_to_skill/output/metadata.json
• Console output resembling:

Extraction complete:
   Sources : 1 processed
   Size    : 12.34 MB
   Total Pages: 245
   Words   : 123,456
   Tokens  : ~123K
   Chapters: 12 detected overall (numeric)
   ToC     : yes

Programmatic Access

For integration into larger workflows, import the core extraction helper directly from the utility module:

# Example: programmatic use of the core helper

from pathlib import Path
from book_to_skill.utils import extract_single_file

result = extract_single_file(
    Path("mybook.pdf"), 
    extraction_mode="technical", 
    install_mode="ask"
)

print(result["text"][:200])          # first 200 characters of the extracted text

print(result["chapters_detected"])   # number of chapters found

print(result["estimated_tokens"])    # token estimate used for LLM cost pre‑flight

The extract_single_file() function returns a dictionary containing the raw text and all metadata fields that would otherwise appear in the JSON output file.

Summary

• Combined plain-text corpus: A single consolidated text file containing all extracted content with document separators, written to the path specified by OUTPUT_TEXT.
• Metadata JSON file: Structured data including per-source statistics and aggregate totals, serialized to OUTPUT_META as implemented in utils.py lines 1095-1110.
• Console summary: Human-readable statistics printed to stdout during execution (lines 1160-1170), providing immediate extraction feedback.

Frequently Asked Questions

Where does the book-to-skill extraction pipeline save its output files?

By default, the pipeline writes files to book_to_skill/output/ within the package directory. The specific paths are defined in book_to_skill/config.py through the OUTPUT_TEXT and OUTPUT_META constants, which default to full_text.txt and metadata.json respectively.

What information does the metadata JSON file contain?

The metadata JSON includes per-source details (source_file, format, extraction_method, pages, chars, words, estimated_tokens, chapters_detected, has_toc) and aggregate statistics (total_sources, file_size_mb, total_pages, estimated_tokens_human). This structure is written in book_to_skill/utils.py lines 1095-1110.

Can I customize the output directory for the extraction pipeline?

Yes. While the default output location is defined in book_to_skill/config.py, you can modify the OUTPUT_DIR, OUTPUT_TEXT, and OUTPUT_META constants before running the pipeline. The extraction functions in utils.py reference these configuration variables when writing results.

How do I access extraction results programmatically without writing files?

Import extract_single_file from book_to_skill.utils and call it with a Path object and extraction parameters. This function returns a dictionary containing the extracted text, chapters_detected, estimated_tokens, and other metadata fields directly in memory, bypassing the file-writing logic triggered by the CLI pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}