How to Update and Fold‑In New Chapters When Merging Them into an Existing Skill

Merging new chapters into an existing book‑to‑skill skill follows a three‑stage workflow: prepare source files, run the extractor with the --fold-in flag, then validate and commit the updated SKILL.md.

The virgiliojr94/book-to-skill repository provides a deterministic pipeline for converting book content into structured skills. When you need to incorporate additional chapters into an already‑established skill, the fold‑in workflow ensures clean integration without overwriting existing content. This guide walks through the exact process as implemented in the source code.

Understanding the Fold‑In Architecture

The fold‑in mechanism separates content preparation from structural merging. Unlike a full regeneration that rebuilds the entire skill from scratch, fold‑in preserves your existing SKILL.md structure while surgically inserting new chapter blocks.

Two core components handle this:

• scripts/extract.py – The CLI entry point that orchestrates extraction and accepts the --fold-in flag.

• book_to_skill/parsers/*.py – Format‑specific parsers that generate markdown fragments and determine insertion points within the existing hierarchy.

Stage 1: Prepare New Chapter Sources

Before running any commands, place your new chapter files in a dedicated directory. The extractor accepts multiple formats including PDF, EPUB, DOCX, HTML, and RTF.

# Create a staging directory for new content

mkdir -p new_chapter

# Add your source files (example: PDF and EPUB)

curl -L -o new_chapter/chapter7.pdf https://example.com/chapter7.pdf
cp /path/to/chapter8.epub new_chapter/

Optionally, include a metadata.yaml file to specify chapter order, custom tags, or explicit placement instructions. The configuration system in book_to_skill/config.py reads these values to influence parser behavior.

# new_chapter/metadata.yaml

title: "Chapter 7: Advanced Topics"
order: 7
tags: ["advanced", "optimization"]

Stage 2: Execute the Fold‑In Command

The critical operation uses the --fold-in flag, pointing the extractor at both your existing skill directory and the new source files.

python -m book_to_skill extract \
    --fold-in skills/my_skill/ \
    new_chapter/*

Here's what happens under the hood:

1. Format detection – Each parser in book_to_skill/parsers/ identifies file types and extracts raw text.

2. Sanitization – book_to_skill/sanitize.py strips unsafe Unicode (BIDI controls, zero‑width spaces) and normalizes line endings to prevent malformed markdown.

3. Structural insertion – Parsers analyze heading hierarchies in the existing SKILL.md and insert new sections at the appropriate depth, preserving table‑of‑contents integrity as defined in docs/architecture.md.

4. Idempotent output – The merge is deterministic; identical inputs produce identical output, which is essential for CI reproducibility.

Stage 3: Validate and Commit Changes

After extraction, verify structural integrity before committing.

# Validate the updated skill against the architectural contract

python tools/validate_skill.py skills/my_skill/SKILL.md

The validator enforces:

• Proper markdown fence structure
• Heading hierarchy compliance
• Schema adherence as documented in docs/architecture.md

Once validation passes, stage and commit with a descriptive message:

git add skills/my_skill/SKILL.md
git commit -m "Fold-in Chapter 7: Advanced Topics"

The repository's CI pipeline (.github/workflows/ci.yml) automatically runs pytest -q and ruff check . to guard against regressions.

Complete Workflow Example

# 1. Prepare sources

mkdir -p new_chapter
curl -L -o new_chapter/chapter7.pdf https://example.com/chapter7.pdf
curl -L -o new_chapter/chapter8.epub https://example.com/chapter8.epub

# 2. Fold into existing skill

python -m book_to_skill extract \
    --fold-in skills/my_skill/ \
    new_chapter/*

# 3. Validate

python tools/validate_skill.py skills/my_skill/SKILL.md

# 4. Run local quality checks

pytest -q
ruff check .

# 5. Commit

git add skills/my_skill/SKILL.md
git commit -m "Fold-in Chapters 7-8: Advanced Topics and Case Studies"

Key Source Files and Their Roles

File	Purpose	Direct Link
scripts/extract.py	CLI entry point; handles --fold-in flag	extract.py
book_to_skill/parsers/*.py	Format‑specific extraction and insertion logic	parsers/
book_to_skill/sanitize.py	Text normalization and unsafe character removal	sanitize.py
book_to_skill/config.py	Metadata and configuration defaults	config.py
tools/validate_skill.py	Structural validation against architecture spec	validate_skill.py
SKILL.md	Canonical skill definition (updated by fold‑in)	SKILL.md
docs/architecture.md	Skill structure contract (headings, TOC, metadata)	architecture.md
.github/workflows/ci.yml	Automated testing and linting post‑merge	ci.yml

Summary

• Prepare new chapters by placing source files (PDF, EPUB, DOCX, etc.) in a staging directory with optional metadata.yaml.
• Fold‑in using python -m book_to_skill extract --fold-in <skill_dir> <sources> — this preserves existing SKILL.md structure while inserting new content.
• Validate with tools/validate_skill.py to ensure compliance with the architectural contract.
• Commit only after local tests pass (pytest, ruff) to maintain CI pipeline health.

Frequently Asked Questions

What file formats does the fold‑in workflow support?

The extractor supports PDF, EPUB, DOCX, HTML, and RTF through dedicated parsers in book_to_skill/parsers/. Each parser normalizes extracted content to markdown fragments before insertion.

Why use --fold-in instead of regenerating the entire skill?

Fold‑in preserves existing structural edits, metadata annotations, and manual formatting in your SKILL.md. Full regeneration rebuilds from source files only, discarding any hand‑tuned changes you made to the skill output.

How does the validator ensure my skill remains compliant?

tools/validate_skill.py enforces the schema defined in docs/architecture.md, checking heading depth, TOC consistency, markdown fence integrity, and metadata requirements. It catches structural errors before they break downstream consumers.

What happens if the same fold‑in command runs twice?

The process is idempotent — identical inputs produce identical SKILL.md output. This determinism enables reliable CI checks and prevents duplicate content insertion during automated workflows.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}