How book-to-skill Handles Multi-Source Resolution with Glob Patterns and Directory Expansion

book-to-skill resolves multiple input sources through resolve_input_files() in book_to_skill/utils.py, which expands tilde paths, glob patterns, and directories into a deduplicated, deterministically sorted list of supported file paths.

The virgiliojr94/book-to-skill repository provides a CLI tool that converts documents into structured skill extracts. A core challenge is accepting heterogeneous inputs—individual files, glob patterns, and entire directories—while guaranteeing reproducible processing order. This article examines the complete resolution pipeline implemented in the source code.

The Core Resolution Function

All multi-source logic converges in book_to_skill/utils.py → resolve_input_files(). This function transforms raw user arguments into a clean list of Path objects ready for extraction.

The function operates in distinct phases that preserve user intent while enforcing consistency:

# Signature as implemented in book_to_skill/utils.py

def resolve_input_files(input_paths: list[str]) -> list[Path]:
    ...

Phase 1: Tilde Expansion

Before any pattern matching occurs, resolve_input_files() handles shell-quoted home directory references. When users quote glob patterns to prevent premature shell expansion, the tilde character (~) would otherwise be treated as a literal directory name.

The code explicitly expands leading tildes using os.path.expanduser():

• Lines 25-28 in book_to_skill/utils.py check for ~ prefix
• Expansion happens before glob detection or existence checks

# From book_to_skill/utils.py (lines 25-28)

expanded = os.path.expanduser(path) if path.startswith("~") else path

This ensures ~/books/*.epub resolves correctly even when quoted.

Phase 2: Glob Pattern Detection and Expansion

Paths that do not exist as literal files trigger glob handling. The code detects wildcard characters (*, ?, [) and invokes glob.glob() with recursive traversal enabled:

• Lines 36-42 implement the glob expansion
• The recursive=True flag enables ** patterns for nested directory matching

# From book_to_skill/utils.py (lines 36-42)

if any(c in expanded for c in "*?["):
    matches = glob.glob(expanded, recursive=True)

Supported Extension Filtering

Each glob match undergoes validation against SUPPORTED_EXTENSIONS (defined in book_to_skill/config.py). Only files with recognized suffixes proceed:

• Lines 45-47 filter by extension
• Unsupported files are silently discarded during expansion

Phase 3: Deterministic Sorting

To guarantee reproducible builds, expanded glob results are sorted alphabetically with case-insensitive comparison before integration:

• Lines 47-48 apply sorted(matches, key=str.lower)
• This prevents platform-dependent filesystem ordering from affecting output

The deterministic ordering ensures identical inputs produce identical processing sequences across different environments.

Phase 4: Directory Tree Walking

When a path identifies an existing directory, resolve_input_files() recursively traverses the tree using os.walk():

• Lines 51-60 handle directory expansion
• Every file with a supported extension is collected
• Results are again sorted deterministically

# From book_to_skill/utils.py (lines 51-60)

for root, _, files in os.walk(expanded):
    for filename in files:
        if Path(filename).suffix.lower() in SUPPORTED_EXTENSIONS:
            found.append(os.path.join(root, filename))
found.sort(key=str.lower)

This enables bulk processing of entire document libraries without explicit file enumeration.

Phase 5: Deduplication with Order Preservation

The final phase eliminates duplicate paths while respecting the original argument sequence:

• Lines 65-73 implement set-based deduplication
• Explicit files retain their user-specified position
• Expanded sources maintain alphabetical ordering within their expansion group

# From book_to_skill/utils.py (lines 65-73)

seen = set()
result = []
for p in all_paths:
    if p not in seen:
        seen.add(p)
        result.append(p)

This prevents the same file from being processed multiple times when specified through different patterns.

Integration with the CLI Pipeline

The resolved paths flow into book_to_skill/__main__.py → main(), which orchestrates extraction:

• Lines 93-97 invoke resolve_input_files() with parsed CLI arguments
• Each returned Path is passed to extract_single_file()
• Extracted contents merge into a unified corpus

This architectural separation—discovery in utils.py, extraction in __main__.py—enables clean testing and API reuse.

Practical Usage Examples

CLI with Quoted Glob Patterns

# Single recursive glob (quoted prevents shell expansion)

book-to-skill "~/library/**/*.epub"

# Multiple patterns mixed with explicit files

book-to-skill thesis.pdf "drafts/chapter_*.md" references.bib

# Entire directory of supported documents

book-to-skill /mnt/shared/technical_docs/

Programmatic API Access

from book_to_skill.utils import resolve_input_files

# Heterogeneous input list

sources = [
    "~/ebooks/*.pdf",           # glob pattern

    "notes/",                    # directory

    "appendix.md"                # explicit file

]

resolved = resolve_input_files(sources)
print(f"Processing {len(resolved)} documents:")
for path in resolved:
    print(f"  - {path}")

Key Configuration and Extension Points

Component	Location	Purpose
SUPPORTED_EXTENSIONS	book_to_skill/config.py	Central registry of extractable file formats
resolve_input_files()	book_to_skill/utils.py	Complete multi-source resolution implementation
extract_single_file()	book_to_skill/extractor.py	Per-file content extraction (called per resolved path)
Test coverage	tests/test_multi_source_toc.py	Validates resolution with table-of-contents generation

Summary

• Tilde expansion occurs first to handle quoted patterns correctly
• Glob detection uses *?[ character checks with glob.glob(recursive=True)
• Extension filtering applies SUPPORTED_EXTENSIONS from config
• Deterministic sorting guarantees reproducible ordering via case-insensitive sort
• Directory walking recursively collects supported files with os.walk()
• Deduplication preserves user argument order while eliminating repeats

Frequently Asked Questions

How does book-to-skill handle shell-expanded versus quoted glob patterns?

book-to-skill accepts both forms but recommends quoting patterns. When quoted, tilde expansion happens internally via os.path.expanduser(). When unquoted, the shell expands globs before invocation. The resolve_input_files() function processes the resulting paths identically in either case.

What file formats does directory expansion include?

Only extensions listed in SUPPORTED_EXTENSIONS (defined in book_to_skill/config.py) are collected. The current implementation supports common document formats including .pdf, .epub, .txt, and .md, though the exact list is configurable in the source.

Why are glob results sorted alphabetically rather than by modification time?

Deterministic ordering ensures reproducible extraction across runs and environments. Alphabetical sorting by lowercase path (lines 47-48) eliminates dependency on filesystem-specific ordering, which varies between operating systems and storage configurations.

Can the same file appear multiple times in the output list?

No. The deduplication pass (lines 65-73 in book_to_skill/utils.py) tracks seen paths with a set while building the result list, ensuring each file appears exactly once regardless of how many patterns or arguments reference it.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}