# What Are the Main Components of the book-to-skill Architecture?

> Understand the book-to-skill architecture. Explore its two main components: a Python extractor for text and metadata, and an LLM agent for skill package compilation.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: architecture
- Published: 2026-08-28

---

**The book-to-skill architecture consists of two distinct layers—a deterministic Python-based extractor that parses documents into clean text and metadata, and a spec-driven LLM agent that compiles that output into a loadable skill package.**

The book-to-skill project transforms static documents (PDFs, EPUBs, DOCX, and more) into interactive, queryable skills for AI assistants like Copilot or Claude. Understanding the book-to-skill architecture reveals how the system separates deterministic text processing from generative AI workflows to ensure reliability, security, and cost efficiency.

## Core Components of the book-to-skill Architecture

The repository is explicitly split into two halves that communicate through a well-defined interface: [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json).

### Deterministic Extractor (Python Layer)

The **Deterministic Extractor** is a fully Python-based pipeline that operates without LLM dependencies. Its sole responsibility is ingesting documents and emitting sanitized, structured data.

- **[`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py)** – The entry-point shim that forwards arguments to the CLI.
- **[`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py)** – Implements command-line parsing, multi-source handling, and the main runner orchestration. This module coordinates chapter detection and table-of-contents extraction.
- **`book_to_skill/parsers/`** – Contains one module per supported format (`pdf`, `epub`, `docx`, `html`, `rtf`, `calibre`, `text`). Each parser handles format-specific quirks and extracts raw content.
- **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)** – Declares supported extensions, default output locations, and dependency mappings required by the pipeline.
- **[`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py)** – Probes for optional external tools like `pdftotext` or `ebook-convert` and implements the `--check` report for environment validation.
- **[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)** – Strips invisible Unicode characters and tag-block artifacts to harden the extracted text against downstream parsing errors.

This layer produces two artifacts in a working directory (typically `book_skill_work/`): [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) containing merged, source-marked content, and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) storing page counts, word counts, token estimates, and chapter boundaries.

### Spec-Driven Generator (Agent Layer)

The **Spec-Driven Generator** consumes the extractor's output and executes a sequence of LLM-driven steps defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md).

- **[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)** – The master specification located in the repository root that drives the agent through steps 0-10, including content-type detection, cost estimation, per-chapter summarization, glossary creation, and final skill assembly.
- **`chapters/*.md`** – On-demand chapter files generated by the agent and loaded lazily only when specific queries target that content.
- **[`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md)**, **[`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md)**, **[`cheatsheet.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/cheatsheet.md)** – Supporting artifacts containing extracted terms, reusable techniques, and decision trees compiled from the source material.
- **[`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py)** – Validates a generated skill package against host-specific rules (e.g., Copilot or Claude compatibility).
- **[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** – Performs an advisory scan for prompt-injection vectors or unsafe content in the generated skill.

## How the Extraction Pipeline Works

The deterministic layer follows a strict pipeline from raw document to sanitized output.

### Input Parsing and Sanitization

When you run the extraction command, **[`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py)** dispatches to the appropriate parser in **`book_to_skill/parsers/`** based on file extension. The parser extracts raw text, then passes it through **[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)** to remove zero-width characters, control sequences, and malformed tags that could disrupt LLM tokenization.

```bash

# From the repository root

python -m scripts.extract path/to/book.pdf

```

This command creates [`book_skill_work/full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_skill_work/full_text.txt) and [`book_skill_work/metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_skill_work/metadata.json), with the latter containing structured metadata including detected chapter boundaries and table-of-contents data.

### Metadata Generation

The **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** module handles chapter and ToC detection, calculating token counts using the tokenizer specified in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)**. The metadata JSON includes page ranges, word counts, and content-type hints that inform the generator's cost estimation phase.

## The Generation Workflow

Once extraction completes, the agent layer takes over to compile the skill package.

### SKILL.md Specification

The **[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)** file acts as the control program for the LLM agent. It defines a 10-step workflow: content classification, cost estimation, chunking strategy selection, per-chapter summarization, glossary extraction, pattern identification, and final assembly. According to the source code in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md), this specification implements the "compile-time over runtime" principle by doing heavy processing once rather than at query time.

### On-Demand Chapter Loading

Rather than embedding all content into [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md), the architecture stores individual chapters as separate files in `chapters/*.md`. The main specification file remains small (under typical context window limits), while individual chapter files load lazily when the agent detects a query relevant to that specific content. This design keeps the working memory footprint low and query costs predictable.

```bash

# Assuming extraction already produced the work dir

copilot skill generate \
  --input book_skill_work/full_text.txt \
  --metadata book_skill_work/metadata.json \
  --output ~/.copilot/skills/my-book-skill

```

## Security and Validation Layer

The architecture includes explicit safeguards before deployment.

**[`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py)** checks the generated skill against host-specific constraints:

```bash
python tools/validate_skill.py ~/.copilot/skills/my-book-skill/SKILL.md --lens copilot

```

**[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** provides an additional security pass:

```bash
python tools/scan_generated_skill.py ~/.copilot/skills/my-book-skill

```

These utilities ensure that prompt-injection attempts, unsafe code blocks, or malformed directives do not propagate to end users.

## Design Principles

As documented in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md), three principles shape the book-to-skill architecture:

- **Extract structure, not summaries** – The deterministic layer focuses on frameworks, decision rules, and anti-patterns rather than generating abstractions.
- **Compile-time over runtime** – Heavy lifting occurs during the extraction and generation phases; runtime queries only load necessary chapter slices.
- **Graceful degradation** – Each format parser includes fallback logic in **`book_to_skill/parsers/`**; a single corrupted source does not abort the entire pipeline.

## Summary

- The **Deterministic Extractor** ([`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py), and format-specific parsers) handles document ingestion and sanitization without LLM dependencies.
- The **Spec-Driven Generator** uses [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) to orchestrate LLM steps that create on-demand chapter files (`chapters/*.md`) and reference artifacts ([`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md), [`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md)).
- **Validation tools** ([`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py), [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)) enforce security and host compatibility before deployment.
- The architecture separates extraction from generation to ensure deterministic, reproducible outputs while allowing flexible, cost-effective AI-driven compilation.

## Frequently Asked Questions

### What document formats does the deterministic extractor support?

According to [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), the extractor supports PDF, EPUB, DOCX, HTML, RTF, and plain text files. The system probes for optional external dependencies like `pdftotext` (Poppler) or `ebook-convert` (Calibre) via [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) to handle proprietary formats gracefully.

### How does the architecture prevent prompt injection in generated skills?

The **[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)** module strips invisible Unicode and control characters during extraction, while **[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** performs an advisory scan on the final output for known prompt-injection patterns and unsafe content blocks before the skill is loaded by downstream agents.

### Why are chapters stored as separate files rather than in a single document?

The architecture follows an "on-demand chapters" design principle from [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md). Keeping chapters in `chapters/*.md` allows [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) to remain small and fit within context window limits, while individual chapters load only when queried, reducing token costs and latency during runtime usage.

### Can the extraction pipeline run without installing optional external tools?

Yes. While **[`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py)** checks for optional tools like `pdftotext` or `ebook-convert`, each parser in **`book_to_skill/parsers/`** includes fallback logic. Run `python -m scripts.extract --check` to see which formats your current environment supports without external dependencies.