Implementing Agent Compatibility Scans for Repository Validation: A Complete Guide

The Cursor plugins repository provides an agent-compatibility plugin that validates repositories by executing a deterministic CLI scan and orchestrating four specialized behavioral agents to compute a blended Agent Compatibility Score.

The cursor/plugins repository serves as a marketplace for self-contained extensions that enhance the Cursor IDE with specialized tooling. The agent-compatibility plugin offers a systematic approach to implementing agent compatibility scans for repository validation, separating deterministic analysis from behavioral workflow testing to produce actionable health metrics for agent-driven automation.

How the Agent Compatibility Plugin Works

The plugin operates through a clear separation between data collection and behavioral analysis. This architecture ensures that new workflow checks can be added as additional agents without modifying the core scoring logic.

Repository Structure and Manifest

Each plugin in the repository lives in its own top-level directory following a standard layout. The agent-compatibility plugin registers itself through agent-compatibility/.cursor-plugin/plugin.json, which declares plugin metadata, skill locations, and agent definitions. This manifest file connects the plugin to the Cursor marketplace and defines the entry points for the validation workflow.

The Five-Step Workflow

The orchestration logic resides in agent-compatibility/skills/check-agent-compatibility/SKILL.md. According to the source code analysis, steps 14-25 of this markdown file define the complete compatibility pass that coordinates the scan execution. The skill executes the published agent-compatibility CLI to obtain raw deterministic data, then spawns four specialized agents to evaluate distinct workflow dimensions.

The Four Behavioral Agents

Located in agent-compatibility/agents/, these read-only Markdown files are executed by the Cursor runtime to evaluate specific reliability factors:

• startup-review.md – Determines whether an agent can bootstrap the repository without manual intervention.
• validation-review.md – Verifies if an agent can validate small changes without triggering a full repository test loop.
• docs-reliability-review.md – Validates whether documentation accurately reflects the actual setup path.
• compatibility-scan-review.md – Wraps and interprets the raw CLI output from the deterministic scan.

These agents never modify the repository; they only emit scores and prioritized problem lists.

Calculating the Agent Compatibility Score

The plugin computes a final score using a weighted blend of deterministic and behavioral metrics. The formula implemented in the workflow is:

round((deterministic * 0.7) + (workflow * 0.3))

The deterministic component represents the objective compatibility score returned by the CLI tool. The workflow component represents the average of the three behavioral agent scores (excluding the compatibility-scan-review wrapper). This weighting prioritizes concrete technical compatibility while ensuring real-world agent usability factors into the final metric.

Running Agent Compatibility Scans

You can execute scans via command line, programmatic JSON output, or directly within Cursor as a skill.

Command Line Execution

Run the full compatibility pass using npx:

npx -y agent-compatibility@latest .

This prints a compact dashboard showing the deterministic score and detected issues.

Programmatic JSON Output

For CI pipelines or automated consumption, request structured output:

npx -y agent-compatibility@latest --json .

The JSON response contains the deterministic score, raw issues array, and per-component scores that downstream agents consume.

Cursor Skill Invocation

Inside a Cursor session, invoke the skill directly:

/check-agent-compatibility

This command launches the four sub-agents, aggregates the results using the weighted formula, and returns a synthesized markdown report:

## Agent Compatibility Score: 78/100

Top fixes
- Missing README section that explains local setup
- No scoped test command for quick validation
- Startup script requires manual secret injection

Custom Configuration

Override default behavior by creating agent-compatibility.config.json in your repository root and passing it to the CLI:

npx -y agent-compatibility@latest . --config ./agent-compatibility.config.json

This configuration file can adjust weightings or specify ignored paths for generated files.

Architecture and Key Implementation Files

The plugin’s extensible design relies on specific file paths that separate concerns between manifest declaration, workflow orchestration, and agent definitions:

• agent-compatibility/.cursor-plugin/plugin.json – Registers the plugin with the marketplace and declares skill and agent locations.
• agent-compatibility/skills/check-agent-compatibility/SKILL.md – Contains the main orchestrator defining the five-step compatibility workflow (steps 14-25).
• agent-compatibility/agents/validation-review.md – Implements the validation loop scoring logic and emits plain-text reports.
• agent-compatibility/agents/startup-review.md, docs-reliability-review.md, compatibility-scan-review.md – Provide the behavioral analysis agents.
• agent-compatibility/README.md – Documents the high-level plugin purpose and public API.
• README.md (repository root) – Lists the plugin among the marketplace catalog.

Summary

• The agent-compatibility plugin validates repositories by combining deterministic CLI scans with four specialized behavioral agents.
• The Agent Compatibility Score blends 70% deterministic analysis with 30% workflow behavioral averages using the formula round((deterministic * 0.7) + (workflow * 0.3)).
• Four agents—startup-review, validation-review, docs-reliability-review, and compatibility-scan-review—evaluate distinct automation friction points.
• Scans can be triggered via CLI, JSON output, or the /check-agent-compatibility skill inside Cursor.
• Configuration files and skill definitions reside in specific paths under agent-compatibility/.cursor-plugin/ and agent-compatibility/skills/.

Frequently Asked Questions

How is the Agent Compatibility Score calculated?

The score uses a weighted formula: round((deterministic * 0.7) + (workflow * 0.3)). The deterministic component comes from the CLI scan, while the workflow component averages the scores from three behavioral agents. This weighting prioritizes technical compatibility while ensuring the repository supports practical agent workflows.

What do the four specialized agents evaluate?

Each agent in agent-compatibility/agents/ targets a specific automation bottleneck. startup-review checks bootstrapping capability, validation-review verifies granular change testing, docs-reliability-review confirms documentation accuracy, and compatibility-scan-review processes raw CLI output. Together they identify friction points like missing setup documentation or lack of scoped test commands.

Can I run the scan outside of the Cursor IDE?

Yes. The plugin uses the published agent-compatibility CLI which runs independently via npx -y agent-compatibility@latest .. You can output JSON for programmatic consumption using the --json flag, making the tool suitable for CI/CD pipelines and pre-commit hooks.

How do I customize which files the scan ignores?

Create an agent-compatibility.config.json file in your repository root and pass it to the CLI using --config ./agent-compatibility.config.json. This configuration file allows you to specify ignored paths, such as generated files or directories that should not factor into the compatibility calculation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}