How to Test Codex Plugins Locally Before Publishing: A Complete Guide

You can test Codex plugins locally by installing the plugin into ~/.agents/plugins, then running the plugin-eval CLI to lint manifests, validate skills, and benchmark live executions before submitting to the marketplace.

The openai/plugins repository provides a built-in evaluation harness that mirrors how Codex discovers plugins in production. Testing locally ensures your manifest, skills, and runtime commands are functional before publication. This workflow uses the plugin-eval tool located in plugins/plugin-eval/ to perform static analysis and live benchmarking.

Setting Up Local Installation

Codex loads plugins from a local marketplace directory, allowing you to test changes without pushing to a remote registry.

Create the Marketplace Directory Structure

Codex scans $HOME/.agents/plugins to discover available plugins. Create this directory and symlink your plugin source into it:

mkdir -p ~/.agents/plugins
ln -sfn "$(pwd)/plugins/<plugin-name>" "$HOME/.agents/plugins/<plugin-name>"

For example, to test the render plugin:

ln -sfn "$(pwd)/plugins/render" "$HOME/.agents/plugins/render"

Register the Plugin in the Marketplace

Create or update your local marketplace.json so Codex can discover the plugin. This file resides in ~/.agents/plugins/ and follows the schema documented in plugins/plugin-eval/README.md:

{
  "name": "local",
  "interface": { "displayName": "Local Plugins" },
  "plugins": [
    {
      "name": "<plugin-name>",
      "source": { "source": "local", "path": "./plugins/<plugin-name>" },
      "policy": {
        "installation": "AVAILABLE",
        "authentication": "ON_INSTALL"
      },
      "category": "Developer Tools"
    }
  ]
}

After updating the marketplace entry, restart Codex (or the Codex CLI) to reload the plugin registry.

Validating Plugin Structure

Before executing any commands, verify that your plugin meets the required schema and safety constraints.

Verify the Manifest

Each plugin must contain a valid .codex-plugin/plugin.json file. Run the lint command from the repository root:

node ./plugins/plugin-eval/scripts/plugin-eval.js analyze ./plugins/<plugin-name> --format markdown

This checks for required fields (name, description, version, manifestVersion) and validates that the JSON schema is well-formed. If the manifest is malformed, plugin-eval reports the exact line and field requiring correction.

Run Static Analysis

The analyze command performs comprehensive validation beyond basic JSON syntax:

• Skill front-matter: Validates presence of retrieval.aliases, intents, and pathPatterns in SKILL.md files
• Asset consistency: Confirms all referenced files exist and flags stray binaries
• Safety constraints: Detects hard-coded secrets and verifies proper permission declarations

The tool explicitly protects sensitive values by never printing secret environment variables like OPENAI_API_KEY in its output, as noted in the plugin-eval documentation.

Testing Plugin Execution

Static analysis ensures structural correctness, but you must also verify runtime behavior.

Perform Chat-First Sanity Checks

The start command simulates a natural language request and displays the exact CLI sequence that Codex will execute:

node ./plugins/plugin-eval/scripts/plugin-eval.js start ./plugins/<plugin-name> \
  --request "Evaluate this plugin." \
  --format markdown

This outputs the routed skill, the subsequent analyze command, and any additional steps like init-benchmark. Review this output to confirm that Codex correctly maps user intents to your plugin's capabilities.

Benchmark Live Executions

For plugins that interact with external services (such as Vercel deployments), generate a realistic usage profile:

node ./plugins/plugin-eval/scripts/plugin-eval.js init-benchmark ./plugins/<plugin-name>
node ./plugins/plugin-eval/scripts/plugin-eval.js benchmark ./plugins/<plugin-name> --format markdown

The benchmark writes a JSONL usage log to .plugin-eval/runs/<timestamp>/usage.jsonl. You can then generate a measurement plan to analyze resource utilization:

node ./plugins/plugin-eval/scripts/plugin-eval.js measurement-plan ./plugins/<plugin-name> \
  --observed-usage .plugin-eval/runs/<timestamp>/usage.jsonl \
  --format markdown

The full benchmark harness specification is documented in plugins/plugin-eval/references/benchmark-harness.md.

Automating the Testing Workflow

Combine these steps into a reusable script to iterate quickly during development.

Full Local Testing Script

Save this as test-plugin.sh in your repository root:

#!/usr/bin/env bash
set -euo pipefail

PLUGIN=render   # change to your plugin name

ROOT=$(pwd)

# Link plugin into marketplace

mkdir -p ~/.agents/plugins
ln -sfn "$ROOT/plugins/$PLUGIN" "$HOME/.agents/plugins/$PLUGIN"

# Restart Codex (if CLI is installed)

codex restart   # or quit/reopen the Codex UI

# Static analysis

node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  analyze "./plugins/$PLUGIN" --format markdown

# Chat-first sanity check

node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  start "./plugins/$PLUGIN" \
  --request "Evaluate this plugin." \
  --format markdown

# Optional benchmark

node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  init-benchmark "./plugins/$PLUGIN"
node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  benchmark "./plugins/$PLUGIN" --format markdown

echo "✅ All tests passed for $PLUGIN"

Make it executable and run:

chmod +x test-plugin.sh
./test-plugin.sh

CI-Friendly Direct Execution

If you prefer not to modify your local marketplace (useful for CI pipelines), invoke the CLI directly against any plugin path:

node ./plugins/plugin-eval/scripts/plugin-eval.js analyze ./plugins/<plugin-name> --format markdown

This works without marketplace registration, though it skips the full Codex discovery simulation that requires the symlink setup.

Summary

• Install locally by symlinking your plugin to ~/.agents/plugins/ and updating marketplace.json so Codex can discover it
• Validate structure using plugin-eval analyze to check plugin.json schema and skill front-matter in plugins/<plugin-name>/skills/*.md
• Test execution with plugin-eval start to simulate user requests and verify command routing
• Benchmark performance using plugin-eval init-benchmark and plugin-eval benchmark to generate usage profiles and measurement plans
• Iterate until plugin-eval reports zero errors and all diagnostics pass

Frequently Asked Questions

Where does Codex look for local plugins?

Codex scans the $HOME/.agents/plugins directory to discover locally installed plugins. You must create this directory and symlink your plugin source code into it, then register the plugin in a marketplace.json file within that directory.

What does the plugin-eval CLI check during static analysis?

According to the plugins/plugin-eval/README.md, the analyze command validates the plugin.json manifest schema, verifies skill front-matter includes retrieval.aliases and intents, checks that all referenced assets exist, and enforces safety constraints by detecting hard-coded secrets and validating permission declarations.

Can I test plugins without installing them into the marketplace?

Yes. You can run node ./plugins/plugin-eval/scripts/plugin-eval.js analyze <path> directly against any plugin directory. This bypasses the marketplace registration requirement and works well for CI environments, though it skips the full Codex discovery simulation that requires the symlink setup in ~/.agents/plugins.

How do I fix validation errors reported by plugin-eval?

Common failures include missing SKILL.md front-matter fields, incorrect paths in .codex-plugin/plugin.json, or unchecked environment variables. The tool reports specific line numbers and field names. Fix the reported issues in your source files and re-run the relevant plugin-eval command until the output shows no errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}