# Book-to-Skill Extraction Pipeline Output: The Three Generated Artifacts

> Discover the three essential artifacts generated by the book-to-skill extraction pipeline: a text corpus, metadata JSON, and a stdout summary. Learn more about this powerful tool.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: how-to-guide
- Published: 2026-08-28

---

**The book-to-skill extraction pipeline generates three concrete artifacts: a consolidated plain-text corpus file, a structured metadata JSON file, and a human-readable summary printed to stdout.**

The `virgiliojr94/book-to-skill` repository processes PDF, EPUB, DOCX, HTML, and plain-text documents through a coordinated pipeline spanning [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py), and [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py). Each execution produces standardized outputs that enable downstream skill generation while preserving raw text for custom processing workflows.

## The Three Core Extraction Pipeline Outputs

### Combined Plain-Text Corpus

The pipeline extracts and consolidates text from all supplied documents into a single searchable file. According to [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 822-825), the system writes the `consolidated_text` variable to the path defined by `OUTPUT_TEXT` in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py).

This output contains the extracted text from every input source, separated by clear banner lines that demarcate document boundaries. The default location is [`book_to_skill/output/full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/output/full_text.txt), though this is configurable through the central configuration module.

### Metadata JSON File

The structured metadata output provides machine-readable documentation of the extraction process. Written to `OUTPUT_META` (typically [`book_to_skill/output/metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/output/metadata.json)) after the extraction loop completes, this file is generated in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 1095-1110).

The JSON structure contains two primary sections:

- **Per-source fields**: `source_file`, `format`, `extraction_method`, `pages`, `chars`, `words`, `estimated_tokens`, `chapters_detected`, and `has_toc`
- **Aggregate statistics**: `total_sources`, `file_size_mb`, `total_pages`, and `estimated_tokens_human`

This metadata enables LLM cost pre-flight calculations and provides provenance tracking for each processed document.

### Human-Readable Summary

After metadata serialization, the `main()` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 1160-1170) prints a concise extraction report to stdout. This summary displays the number of sources processed, total file size, page count, word count, token estimate, detected chapter count (including the detection method), and table-of-contents status.

This immediate feedback allows operators to verify extraction success without inspecting output files directly.

## Output Configuration and File Paths

All output paths are centralized in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) through the constants `OUTPUT_DIR`, `OUTPUT_TEXT`, and `OUTPUT_META`. The pipeline creates these locations automatically during the initialization phase defined in the utility module.

The default output directory structure follows this pattern:

```

book_to_skill/output/
├── full_text.txt
└── metadata.json

```

## How to Generate the Pipeline Output

### Command-Line Usage

Invoke the extraction pipeline through the CLI entry point to generate all three outputs simultaneously:

```python

# Example: invoking the pipeline from the command line

$ book-to-skill mybook.pdf --mode technical

```

This command executes the full workflow defined in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) → [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) → [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py), producing:
- Extracted text written to [`book_to_skill/output/full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/output/full_text.txt)
- Metadata written to [`book_to_skill/output/metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/output/metadata.json)
- Console output resembling:

```

Extraction complete:
   Sources : 1 processed
   Size    : 12.34 MB
   Total Pages: 245
   Words   : 123,456
   Tokens  : ~123K
   Chapters: 12 detected overall (numeric)
   ToC     : yes

```

### Programmatic Access

For integration into larger workflows, import the core extraction helper directly from the utility module:

```python

# Example: programmatic use of the core helper

from pathlib import Path
from book_to_skill.utils import extract_single_file

result = extract_single_file(
    Path("mybook.pdf"), 
    extraction_mode="technical", 
    install_mode="ask"
)

print(result["text"][:200])          # first 200 characters of the extracted text

print(result["chapters_detected"])   # number of chapters found

print(result["estimated_tokens"])    # token estimate used for LLM cost pre‑flight

```

The `extract_single_file()` function returns a dictionary containing the raw text and all metadata fields that would otherwise appear in the JSON output file.

## Summary

- **Combined plain-text corpus**: A single consolidated text file containing all extracted content with document separators, written to the path specified by `OUTPUT_TEXT`.
- **Metadata JSON file**: Structured data including per-source statistics and aggregate totals, serialized to `OUTPUT_META` as implemented in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) lines 1095-1110.
- **Console summary**: Human-readable statistics printed to stdout during execution (lines 1160-1170), providing immediate extraction feedback.

## Frequently Asked Questions

### Where does the book-to-skill extraction pipeline save its output files?

By default, the pipeline writes files to `book_to_skill/output/` within the package directory. The specific paths are defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) through the `OUTPUT_TEXT` and `OUTPUT_META` constants, which default to [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) respectively.

### What information does the metadata JSON file contain?

The metadata JSON includes per-source details (`source_file`, `format`, `extraction_method`, `pages`, `chars`, `words`, `estimated_tokens`, `chapters_detected`, `has_toc`) and aggregate statistics (`total_sources`, `file_size_mb`, `total_pages`, `estimated_tokens_human`). This structure is written in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) lines 1095-1110.

### Can I customize the output directory for the extraction pipeline?

Yes. While the default output location is defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), you can modify the `OUTPUT_DIR`, `OUTPUT_TEXT`, and `OUTPUT_META` constants before running the pipeline. The extraction functions in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) reference these configuration variables when writing results.

### How do I access extraction results programmatically without writing files?

Import `extract_single_file` from `book_to_skill.utils` and call it with a `Path` object and extraction parameters. This function returns a dictionary containing the extracted `text`, `chapters_detected`, `estimated_tokens`, and other metadata fields directly in memory, bypassing the file-writing logic triggered by the CLI pipeline.