What Causes Skill Drift Errors in check.py and How to Fix Them

Skill drift errors occur when bundled skill directories in plugins/agent-plugins/ diverge from their canonical sources in plugins/vertical-plugins/, triggering validation failures in check.py when filecmp.dircmp detects content mismatches, missing files, or extraneous files.

In the anthropics/financial-services repository, the check.py validation script ensures that agent plugins contain exact copies of shared skills defined in vertical plugins. When these copies fall out of sync, the script raises skill drift errors to prevent agents from running outdated or inconsistent code.

Understanding Skill Drift Validation in check.py

The validation logic in scripts/check.py builds a mapping of source skills (src_by_name) from plugins/vertical-plugins/<vertical>/skills/ and compares each against its bundled counterpart in plugins/agent-plugins/<slug>/skills/.

The Comparison Mechanism

The script uses Python's filecmp.dircmp class to perform a recursive directory comparison:

src = src_by_name.get(bundled.name)                # source dir

cmp = filecmp.dircmp(src, bundled)                # compare contents

if cmp.diff_files or cmp.left_only or cmp.right_only:
    err(
        f"bundled-skill: {rel(bundled)}: drifted from {rel(src)} "
        "(run scripts/sync-agent-skills.py)"
    )

Three Conditions That Trigger Drift Errors

A skill drift error is raised when any of these conditions are true:

• cmp.diff_files – Files exist in both directories but have differing content.
• cmp.left_only – Files present in the vertical plugin source are missing from the bundled copy.
• cmp.right_only – Files exist in the bundled copy that are not present in the source.

Common Causes of Skill Drift

Skill drift typically occurs when developers edit files directly within an agent plugin's bundled skill directory or when vertical plugin updates are not propagated to agent plugins.

Direct Edits to Bundled Skills

Modifying files under plugins/agent-plugins/*/skills/ creates a mismatch with the canonical source under plugins/vertical-plugins/*/skills/. These bundled directories are derived copies, not sources of truth, and manual edits violate the single-source-of-truth architecture.

Out-of-Sync Workflow Changes

When skills are updated in the vertical plugin but scripts/sync-agent-skills.py is not executed, the agent plugins retain stale versions. Accumulated changes result in drift detection during validation, particularly after pulling upstream changes or switching branches.

How to Resolve Skill Drift Errors

Follow these steps to eliminate drift and restore passing validation status.

1. Run the Synchronization Script

Execute the synchronization utility to overwrite bundled skills with current source versions:

python3 scripts/sync-agent-skills.py

This script deletes outdated bundled skill directories and recopies the current versions from plugins/vertical-plugins/. Typical output appears as:

synced 12 bundled skill dir(s) from vertical-plugins/

2. Verify the Fix

Re-run check.py to confirm drift is eliminated:

python3 scripts/check.py

You should see:

OK — 124 file(s) checked, 0 issues.

If drift persists, the error output specifies exactly which files differ:

FAIL — 1 issue(s) across 124 file(s):
  ✗ bundled-skill: plugins/agent-plugins/valuation-reviewer/skills/client-report: drifted from plugins/vertical-plugins/wealth-management/skills/client-report (run scripts/sync-agent-skills.py)

3. Update Your Development Workflow

Prevent future drift by adhering to these practices:

• Edit sources only – Modify files exclusively under plugins/vertical-plugins/*/skills/, never in agent plugin bundles.
• Sync after changes – Run scripts/sync-agent-skills.py immediately after pulling changes or modifying vertical plugin skills.
• Pre-commit validation – Add check.py to your pre-commit hooks to catch drift before submission.

Key Files and Architecture

Understanding the relationship between these paths clarifies why drift occurs:

Path	Role
scripts/check.py	Validates manifests and detects skill drift via filecmp.dircmp comparison.
scripts/sync-agent-skills.py	Re-copies skills from vertical plugins to agent plugin bundles, eliminating drift.
plugins/vertical-plugins/*/skills/*	Source of truth for all skill implementations.
plugins/agent-plugins/*/skills/*	Derived copies consumed by individual agents; must remain synchronized with sources.

Summary

• Skill drift represents a mismatch between bundled agent skills and their vertical plugin sources, detected by filecmp.dircmp in check.py.
• Three triggers cause errors: differing file contents (diff_files), missing source files (left_only), and extraneous bundled files (right_only).
• Resolution requires running scripts/sync-agent-skills.py to re-copy canonical sources, followed by verification via check.py.
• Prevention involves editing only vertical plugin sources and integrating the sync script into your development workflow.

Frequently Asked Questions

What is the difference between vertical-plugins and agent-plugins in this architecture?

In the anthropics/financial-services repository, plugins/vertical-plugins/ contains canonical skill definitions organized by business vertical (e.g., wealth-management), while plugins/agent-plugins/ contains agent-specific bundles that copy these skills for deployment. Only the vertical plugin versions are authoritative sources; agent plugins contain derived copies that must stay synchronized.

Can I manually fix skill drift by copying files instead of using the sync script?

While manually copying files might temporarily resolve specific differences, scripts/sync-agent-skills.py ensures complete synchronization including deletions of obsolete files and proper directory structures. Manual copying risks leaving orphaned files or incomplete updates, so the sync script is the only supported approach for maintaining repository integrity.

Why does check.py use filecmp.dircmp instead of a simple hash comparison?

The filecmp.dircmp class provides detailed reporting on exactly which files differ, are missing, or are extraneous, enabling precise error messages that identify the specific source and bundled paths. This granularity helps developers understand the exact nature of the drift without manual investigation into which specific files caused the mismatch.

Should I run sync-agent-skills.py before every commit?

You should run scripts/sync-agent-skills.py whenever you modify skills in plugins/vertical-plugins/ or after pulling updates from remote repositories that may include skill changes. Running it before committing ensures your bundled copies match the sources you intend to ship, preventing CI failures from drift errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}