# How the Generator Component of book-to-skill Works: A Step-by-Step Technical Guide > Explore the generator component of book-to-skill. Learn how this spec-driven agent transforms book text into structured skill folders with summaries and glossaries. A step-by-step technical guide. - Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill) - Tags: deep-dive - Published: 2026-08-27 --- **The generator component of book-to-skill is a spec-driven agent that follows the declarative workflow in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) to convert extracted book text into a structured skill folder containing summaries, glossaries, and reference materials.** The generator component of book-to-skill serves as the intelligent second half of the extraction pipeline, transforming raw text into queryable knowledge bases. While the deterministic Python extractor handles document parsing, the generator acts as an agent that budgets tokens, interacts with users, and synthesizes structured outputs. According to the virgiliojr94/book-to-skill source code, this component strictly adheres to the procedural specification defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) at the repository root. ## Architecture and Data Flow The generator operates downstream from the Python-based extractor, consuming intermediate files produced during the first phase. As illustrated in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md) (lines 12-40), the pipeline follows this sequence: ``` EXTRACTOR (Python) → full_text.txt + metadata.json → GENERATOR (agent follows SKILL.md) → // ``` The generator produces five distinct output types within the skill directory: - **[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)** — Master index containing core frameworks, chapter references, and usage instructions (kept under 4,000 tokens) - **`chapters/*.md`** — On-demand chapter summaries loaded only when queried - **[`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md)** — Alphabetical term definitions - **[`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md)** — Concrete techniques and design patterns - **[`cheatsheet.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/cheatsheet.md)** — Decision rules, tables, and quick-reference guides ## The 10-Step Generator Workflow The generator executes a deterministic sequence defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md), beginning with validation and concluding with optional publication. Each step manages specific token budgets and user interaction points. ### Step 0: Out-of-Scope Validation The generator first validates that at least one document path or glob pattern was supplied via command line arguments. This check, referenced at [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) line 82, prevents execution without input sources. ### Step 1: Input Validation and Content Type Identification The generator ensures supported file types are present and expands directory globs. At [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) line 104, the agent prompts the user to classify the source as either *technical* or *text-heavy*, storing this classification in the `BOOK_TYPE` variable. This determination directly influences token allocation strategies in later steps. ### Step 2: Text Extraction and Cost Estimation The generator calls the extraction shim located at [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) to produce [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 27-31). Before proceeding, the agent reads [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) to calculate token-and-time estimates, presenting these costs to the user for approval ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 78-84). For large books exceeding 50,000 tokens, the generator switches to REPL-style access at [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 15-21, using `grep` and `sed` to probe the corpus without loading the entire file into context. ### Step 3: Structure Analysis The generator analyzes the first 8,000 characters of the extracted text to detect the title, author, chapter headings, and core themes ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 44-50). This structural map drives the subsequent chapter generation process. ### Step 4: Purpose and Depth Determination The agent queries the user to determine the desired *depth* level: `reference` (concise) or `study` (comprehensive). This selection, documented at [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 78-95, establishes token budgets for Steps 7 and 8. ### Step 5: Skill Naming and Directory Creation The generator proposes or accepts a URL-friendly slug for the skill, discovers the host-specific `SKILLS_HOME` directory, and creates the skill root folder. This initialization sequence appears at [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 99-113, including the bash command `mkdir -p "$SKILLS_HOME//chapters"` (lines 33-37). ### Step 6-7: Chapter Summary Generation For each detected chapter, the generator reads the relevant slice of [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and writes a markdown file following the template specified in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) Step 7. These summaries include frameworks, key takeaways, and code examples, with token budgets adapted dynamically based on `BOOK_TYPE` and `DEPTH` settings ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 41-53). ### Step 8: Supporting Files Generation The agent produces auxiliary reference materials with strict length limits: - **Glossary** — Alphabetical definitions - **Patterns** — Concrete implementation techniques - **Cheatsheet** — Decision matrices and quick-reference tables These conform to the constraints outlined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 24-30. ### Step 9: Master File Assembly The generator composes the root [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) file, aggregating core frameworks, chapter indexes, and topic guides while maintaining the body under 4,000 tokens ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 57-64). ### Step 9.5: Security Scanning Before finalizing, the generator executes the advisory security scanner at [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 28-32). This tool detects prompt-injection attempts, invisible Unicode characters, and unauthorized authority tags without exposing offending text in the output. ### Step 10: Cleanup and Reporting The generator deletes the temporary work directory and prints a comprehensive summary of generated files ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 41-48). ### Step 11: Optional Publication If configured, the generator offers to push the completed skill to a GitHub repository using the `gh` CLI, respecting visibility settings and copyright rules ([`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 14-21). ## Practical Usage Examples ### Command-Line Conversion To execute the full generator workflow from the terminal: ```bash # Verify system dependencies for all supported formats book-to-skill --check # Convert a PDF into a structured skill book-to-skill path/to/book.pdf my-skill-slug ``` The CLI entry point is implemented in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py), which parses arguments, initializes the work directory, and delegates to the generator steps defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md). ### Programmatic Invocation To trigger the generator workflow within Python applications: ```python from book_to_skill.cli import main as bts_main # Execute the complete pipeline (extractor + generator) bts_main(["/path/to/book.epub", "my-awesome-skill"]) ``` The `main` function automatically orchestrates the extractor output and generator workflow, executing Steps 0-10 sequentially. ### Security Validation To manually scan a generated skill for policy violations: ```bash python -m tools.scan_generated_skill /home/user/.copilot/skills/my-awesome-skill ``` This invokes the Step 9.5 scanner independently, reporting any detected security issues without displaying sensitive content. ## Key Source Files and Their Roles | Path | Generator Function | |------|-------------------| | [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) | Authoritative specification driving every generation step (0-11) | | [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md) | High-level component diagram showing extractor-to-generator data flow | | [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) | CLI entry point that orchestrates the generator workflow | | [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) | Security scanner implementation for Step 9.5 | | [`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py) | Host-specific validation against Copilot, Amazon Q, or Claude conventions | | [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) | Token budgeting and path resolution utilities | ## Summary - The **generator component of book-to-skill** operates as a spec-driven agent governed entirely by the procedural definitions in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md). - It processes intermediate files ([`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt), [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json)) from the Python extractor through a rigorous 10-step workflow. - **Token budgeting** adapts dynamically based on book type (technical vs. text-heavy) and user-selected depth (reference vs. study). - **REPL-style access** enables processing of books exceeding 50,000 tokens without context overflow. - **Security scanning** at Step 9.5 validates outputs against prompt-injection and Unicode attacks before delivery. - The component produces five standardized output types: master [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md), chapter summaries, glossary, patterns, and cheatsheet. ## Frequently Asked Questions ### What triggers the generator to use REPL-style access instead of full text loading? When the generator detects that [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) exceeds 50,000 tokens based on [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) analysis, it switches to `grep` and `sed` based probing at [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 15-21. This mode allows the agent to query specific sections without exhausting the context window. ### How does the generator determine token budgets for different output sections? The generator applies adaptive budgeting based on two variables set during user interaction: `BOOK_TYPE` (technical vs. text-heavy) and `DEPTH` (reference vs. study). These flags, captured at [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) lines 104 and 78-95 respectively, scale the allocated tokens for chapter summaries in Step 7 and supporting files in Step 8. ### Can the generator workflow be executed without installing the full package? No, the generator requires the complete `book-to-skill` installation because it invokes [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) for text processing and [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) for validation. However, once a skill is generated, the output folder is self-contained and portable across host platforms (Copilot, Amazon Q, Claude). ### What security vulnerabilities does the Step 9.5 scanner detect? The scanner at [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) specifically hunts for three attack vectors: prompt-injection sequences, invisible Unicode characters (zero-width spaces, homoglyphs), and unauthorized authority tags that might trick AI hosts into executing harmful instructions. The scan runs automatically before cleanup but can be invoked manually for existing skill directories.