How the Generator Component of book-to-skill Works: A Step-by-Step Technical Guide

The generator component of book-to-skill is a spec-driven agent that follows the declarative workflow in SKILL.md to convert extracted book text into a structured skill folder containing summaries, glossaries, and reference materials.

The generator component of book-to-skill serves as the intelligent second half of the extraction pipeline, transforming raw text into queryable knowledge bases. While the deterministic Python extractor handles document parsing, the generator acts as an agent that budgets tokens, interacts with users, and synthesizes structured outputs. According to the virgiliojr94/book-to-skill source code, this component strictly adheres to the procedural specification defined in SKILL.md at the repository root.

Architecture and Data Flow

The generator operates downstream from the Python-based extractor, consuming intermediate files produced during the first phase. As illustrated in docs/architecture.md (lines 12-40), the pipeline follows this sequence:

EXTRACTOR (Python) → full_text.txt + metadata.json → GENERATOR (agent follows SKILL.md) → <SKILLS_HOME>/<slug>/

The generator produces five distinct output types within the skill directory:

• SKILL.md — Master index containing core frameworks, chapter references, and usage instructions (kept under 4,000 tokens)
• chapters/*.md — On-demand chapter summaries loaded only when queried
• glossary.md — Alphabetical term definitions
• patterns.md — Concrete techniques and design patterns
• cheatsheet.md — Decision rules, tables, and quick-reference guides

The 10-Step Generator Workflow

The generator executes a deterministic sequence defined in SKILL.md, beginning with validation and concluding with optional publication. Each step manages specific token budgets and user interaction points.

Step 0: Out-of-Scope Validation

The generator first validates that at least one document path or glob pattern was supplied via command line arguments. This check, referenced at SKILL.md line 82, prevents execution without input sources.

Step 1: Input Validation and Content Type Identification

The generator ensures supported file types are present and expands directory globs. At SKILL.md line 104, the agent prompts the user to classify the source as either technical or text-heavy, storing this classification in the BOOK_TYPE variable. This determination directly influences token allocation strategies in later steps.

Step 2: Text Extraction and Cost Estimation

The generator calls the extraction shim located at scripts/extract.py to produce full_text.txt and metadata.json (SKILL.md lines 27-31). Before proceeding, the agent reads metadata.json to calculate token-and-time estimates, presenting these costs to the user for approval (SKILL.md lines 78-84).

For large books exceeding 50,000 tokens, the generator switches to REPL-style access at SKILL.md lines 15-21, using grep and sed to probe the corpus without loading the entire file into context.

Step 3: Structure Analysis

The generator analyzes the first 8,000 characters of the extracted text to detect the title, author, chapter headings, and core themes (SKILL.md lines 44-50). This structural map drives the subsequent chapter generation process.

Step 4: Purpose and Depth Determination

The agent queries the user to determine the desired depth level: reference (concise) or study (comprehensive). This selection, documented at SKILL.md lines 78-95, establishes token budgets for Steps 7 and 8.

Step 5: Skill Naming and Directory Creation

The generator proposes or accepts a URL-friendly slug for the skill, discovers the host-specific SKILLS_HOME directory, and creates the skill root folder. This initialization sequence appears at SKILL.md lines 99-113, including the bash command mkdir -p "$SKILLS_HOME/<skill_name>/chapters" (lines 33-37).

Step 6-7: Chapter Summary Generation

For each detected chapter, the generator reads the relevant slice of full_text.txt and writes a markdown file following the template specified in SKILL.md Step 7. These summaries include frameworks, key takeaways, and code examples, with token budgets adapted dynamically based on BOOK_TYPE and DEPTH settings (SKILL.md lines 41-53).

Step 8: Supporting Files Generation

The agent produces auxiliary reference materials with strict length limits:

• Glossary — Alphabetical definitions
• Patterns — Concrete implementation techniques
• Cheatsheet — Decision matrices and quick-reference tables

These conform to the constraints outlined in SKILL.md lines 24-30.

Step 9: Master File Assembly

The generator composes the root SKILL.md file, aggregating core frameworks, chapter indexes, and topic guides while maintaining the body under 4,000 tokens (SKILL.md lines 57-64).

Step 9.5: Security Scanning

Before finalizing, the generator executes the advisory security scanner at tools/scan_generated_skill.py (SKILL.md lines 28-32). This tool detects prompt-injection attempts, invisible Unicode characters, and unauthorized authority tags without exposing offending text in the output.

Step 10: Cleanup and Reporting

The generator deletes the temporary work directory and prints a comprehensive summary of generated files (SKILL.md lines 41-48).

Step 11: Optional Publication

If configured, the generator offers to push the completed skill to a GitHub repository using the gh CLI, respecting visibility settings and copyright rules (SKILL.md lines 14-21).

Practical Usage Examples

Command-Line Conversion

To execute the full generator workflow from the terminal:

# Verify system dependencies for all supported formats

book-to-skill --check

# Convert a PDF into a structured skill

book-to-skill path/to/book.pdf my-skill-slug

The CLI entry point is implemented in book_to_skill/cli.py, which parses arguments, initializes the work directory, and delegates to the generator steps defined in SKILL.md.

Programmatic Invocation

To trigger the generator workflow within Python applications:

from book_to_skill.cli import main as bts_main

# Execute the complete pipeline (extractor + generator)

bts_main(["/path/to/book.epub", "my-awesome-skill"])

The main function automatically orchestrates the extractor output and generator workflow, executing Steps 0-10 sequentially.

Security Validation

To manually scan a generated skill for policy violations:

python -m tools.scan_generated_skill /home/user/.copilot/skills/my-awesome-skill

This invokes the Step 9.5 scanner independently, reporting any detected security issues without displaying sensitive content.

Key Source Files and Their Roles

Path	Generator Function
SKILL.md	Authoritative specification driving every generation step (0-11)
docs/architecture.md	High-level component diagram showing extractor-to-generator data flow
book_to_skill/cli.py	CLI entry point that orchestrates the generator workflow
tools/scan_generated_skill.py	Security scanner implementation for Step 9.5
tools/validate_skill.py	Host-specific validation against Copilot, Amazon Q, or Claude conventions
book_to_skill/utils.py	Token budgeting and path resolution utilities

Summary

• The generator component of book-to-skill operates as a spec-driven agent governed entirely by the procedural definitions in SKILL.md.
• It processes intermediate files (full_text.txt, metadata.json) from the Python extractor through a rigorous 10-step workflow.
• Token budgeting adapts dynamically based on book type (technical vs. text-heavy) and user-selected depth (reference vs. study).
• REPL-style access enables processing of books exceeding 50,000 tokens without context overflow.
• Security scanning at Step 9.5 validates outputs against prompt-injection and Unicode attacks before delivery.
• The component produces five standardized output types: master SKILL.md, chapter summaries, glossary, patterns, and cheatsheet.

Frequently Asked Questions

What triggers the generator to use REPL-style access instead of full text loading?

When the generator detects that full_text.txt exceeds 50,000 tokens based on metadata.json analysis, it switches to grep and sed based probing at SKILL.md lines 15-21. This mode allows the agent to query specific sections without exhausting the context window.

How does the generator determine token budgets for different output sections?

The generator applies adaptive budgeting based on two variables set during user interaction: BOOK_TYPE (technical vs. text-heavy) and DEPTH (reference vs. study). These flags, captured at SKILL.md lines 104 and 78-95 respectively, scale the allocated tokens for chapter summaries in Step 7 and supporting files in Step 8.

Can the generator workflow be executed without installing the full package?

No, the generator requires the complete book-to-skill installation because it invokes scripts/extract.py for text processing and tools/scan_generated_skill.py for validation. However, once a skill is generated, the output folder is self-contained and portable across host platforms (Copilot, Amazon Q, Claude).

What security vulnerabilities does the Step 9.5 scanner detect?

The scanner at tools/scan_generated_skill.py specifically hunts for three attack vectors: prompt-injection sequences, invisible Unicode characters (zero-width spaces, homoglyphs), and unauthorized authority tags that might trick AI hosts into executing harmful instructions. The scan runs automatically before cleanup but can be invoked manually for existing skill directories.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}