Generated-Skill Security Scan: What scan_generated_skill.py Detects and Why

The generated-skill security scanner inspects Markdown files produced by book-to-skill for prompt-injection vectors, hidden Unicode payloads, unauthorized tool invocations, and data exfiltration patterns, reporting advisory findings without modifying source files.

The book-to-skill repository automatically converts documents into LLM-consumable skills. Because these generated Markdown files may later be interpreted by language models as privileged instructions, the generated-skill security scan provides a defensive layer that flags potentially dangerous content before deployment.

Security Patterns Detected by the Scanner

The scanner operates as a read-only advisory tool defined in tools/scan_generated_skill.py. It applies rules from the _CONTENT_RULES tuple (lines 27–88) and the _scan_text helper to surface risks across eight distinct categories.

Invisible Unicode Code Points

Any line containing invisible Unicode characters—such as U+2060 WORD JOINER—triggers an immediate flag. These characters can hide malicious payloads or bypass sanitization logic. The scanner maintains parity with the extractor’s stripping behavior by detecting these code points through logic shared with book_to_skill/sanitize.py and its is_invisible_codepoint utility.

Prompt-Override Phrases

The scanner regexes for phrases like "ignore previous instructions", "disregard system", or "you are now …". These strings represent classic injection attempts that attempt to override system prompts or reassign the assistant’s role, potentially granting user-supplied skills unexpected control over model behavior.

System-Like Prefixes and Tags

Lines starting with system: or developer:, or containing <system>/</system> tags, are flagged as fake system messages. Downstream LLMs may interpret these as privileged input, opening pathways for prompt injection that elevates the skill’s authority beyond intended boundaries.

Chat-Template Delimiters

Tokens such as <|im_start|> or [INST] belong to model-specific chat templates. Their presence in a generated skill may cause the downstream model to treat skill content as part of a system prompt, inadvertently changing execution context. The scanner identifies these under the chat_template_tag rule.

Tool-Call Control Tokens

The scanner detects sequences like <tool_call>, </tool_call>, <|tool_call|>, [TOOL_CALL], [/tool_call], {{tool_call}}, or the literal string "tool_call". These tokens trigger framework tool-calling logic; unintentional inclusion could force unwanted tool execution during skill interpretation.

Exfiltration-Shaped Language

Combinations of "exfiltrate", "curl", "wget", "upload", or HTTPS URLs alongside sensitive-data terms like .env, base64, secret, or api-key suggest data-leakage attempts. The _scan_text function (lines 57–68) flags these patterns to prevent skills from phoning home or extracting credentials.

Front-Matter Authority Expansion

The scanner inspects YAML front-matter for two specific authority-related keys:

• allowed-tools: Presence of this key indicates an explicit expansion of tool authority beyond the default safe set.
• disable-model-invocation set to false or 0: Explicitly enabling model invocation (disable-model-invocation: false) allows the skill to trigger uncontrolled LLM generation, which the scanner warns against to prevent abuse.

File System Safety Checks

Beyond content analysis, _collect_skill_files and _read_skill_files enforce size limits and symlink safety. The scanner rejects extremely large files to prevent resource exhaustion and refuses symbolic links that could enable directory-traversal attacks.

Running the Scanner from the Command Line

Execute the scanner against a generated skill directory to receive a report of advisory findings:

python -m tools.scan_generated_skill ./myskill/

Typical output when issues are detected:

Generated-skill scan found 3 advisory finding(s):
  WARN myskill/chapters/01_introduction.md:12 [prompt.ignore_previous] contains an instruction-override phrase
  WARN myskill/chapters/03_security.md:45 [unicode.invisible] contains invisible Unicode code point(s): U+2060
  WARN myskill/chapters/05_tool_use.md:8 [tool.exfiltration_shape] contains exfiltration-shaped tool or sensitive-data language
Review the generated files before loading, installing, or publishing them.

Successful completion produces:

Generated-skill scan passed: no known injection or authority patterns found in the scanned scope.

Programmatic Integration

Import the scanner directly into Python workflows to gate CI/CD pipelines:

from pathlib import Path
from tools.scan_generated_skill import scan_generated_skill, ScanError

try:
    findings = scan_generated_skill(Path("myskill/"))
    if findings:
        for finding in findings:
            print(f"{finding.path}:{finding.line} [{finding.rule_id}] {finding.message}")
        raise SystemExit("Security findings detected")
    else:
        print("Skill is clean.")
except ScanError as exc:
    print(f"Scan could not be completed: {exc}")
    raise SystemExit(1)

The ScanError exception handles filesystem or permission issues that prevent scan completion, while the findings list contains structured data about each flagged line.

Summary

• The scanner is advisory only: It reports risks in tools/scan_generated_skill.py without modifying generated Markdown.
• Eight categories of detection: Invisible Unicode, prompt overrides, system impersonation, chat delimiters, tool tokens, exfiltration patterns, and front-matter authority keys.
• Defense in depth: Combines with book_to_skill/sanitize.py to ensure invisible characters caught during extraction are also flagged during scanning.
• Safety enforcement: Rejects oversized files and symlinks via _collect_skill_files and _read_skill_files to prevent resource exhaustion and directory traversal.
• CI/CD ready: Provides both CLI and Python API interfaces for automated gating before skill publication.

Frequently Asked Questions

Does the scanner automatically fix the security issues it finds?

No. The scanner operates as a read-only diagnostic tool. It identifies patterns such as invisible Unicode or prompt-injection phrases in _scan_text and reports them via ScanFinding objects, but developers must manually edit the generated Markdown files to remove or mitigate flagged content before loading the skill.

Why does the scanner check for chat-template delimiters like <|im_start|>?

These delimiters are defined in the _CONTENT_RULES under chat_template_tag because they belong to model-specific conversation formats. If a generated skill contains these tokens, a downstream LLM may mistakenly interpret the skill content as part of a system prompt or instruction block, altering the execution context in unpredictable ways.

How does the scanner interact with the sanitize module?

The scanner imports is_invisible_codepoint from book_to_skill/sanitize.py to detect invisible Unicode characters. This ensures consistency: the extractor strips these characters during generation, while the scanner flags their presence to alert developers if sanitization was bypassed or failed.

What should I do if legitimate content triggers the exfiltration rule?

Review the specific finding in the context of your skill’s purpose. The tool.exfiltration_shape rule in _scan_text (lines 57–68) uses keyword combinations like curl + secret or wget + .env to reduce false positives, but technical documentation may legitimately mention these terms. If the usage is intentional and safe, document the exception; otherwise, rephrase to remove the high-risk combination.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}