# How to Test Codex Plugins Locally Before Publishing: A Complete Guide

> Test Codex plugins locally before publishing. Install plugins, then use plugin-eval CLI to lint, validate, and benchmark live executions for smooth marketplace submission.

- Repository: [OpenAI/plugins](https://github.com/openai/plugins)
- Tags: how-to-guide
- Published: 2026-06-12

---

**You can test Codex plugins locally by installing the plugin into `~/.agents/plugins`, then running the `plugin-eval` CLI to lint manifests, validate skills, and benchmark live executions before submitting to the marketplace.**

The openai/plugins repository provides a built-in evaluation harness that mirrors how Codex discovers plugins in production. Testing locally ensures your **manifest**, **skills**, and **runtime commands** are functional before publication. This workflow uses the `plugin-eval` tool located in `plugins/plugin-eval/` to perform static analysis and live benchmarking.

## Setting Up Local Installation

Codex loads plugins from a local marketplace directory, allowing you to test changes without pushing to a remote registry.

### Create the Marketplace Directory Structure

Codex scans `$HOME/.agents/plugins` to discover available plugins. Create this directory and symlink your plugin source into it:

```bash
mkdir -p ~/.agents/plugins
ln -sfn "$(pwd)/plugins/<plugin-name>" "$HOME/.agents/plugins/<plugin-name>"

```

For example, to test the `render` plugin:

```bash
ln -sfn "$(pwd)/plugins/render" "$HOME/.agents/plugins/render"

```

### Register the Plugin in the Marketplace

Create or update your local [`marketplace.json`](https://github.com/openai/plugins/blob/main/marketplace.json) so Codex can discover the plugin. This file resides in `~/.agents/plugins/` and follows the schema documented in [`plugins/plugin-eval/README.md`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/README.md):

```json
{
  "name": "local",
  "interface": { "displayName": "Local Plugins" },
  "plugins": [
    {
      "name": "<plugin-name>",
      "source": { "source": "local", "path": "./plugins/<plugin-name>" },
      "policy": {
        "installation": "AVAILABLE",
        "authentication": "ON_INSTALL"
      },
      "category": "Developer Tools"
    }
  ]
}

```

After updating the marketplace entry, restart Codex (or the Codex CLI) to reload the plugin registry.

## Validating Plugin Structure

Before executing any commands, verify that your plugin meets the required schema and safety constraints.

### Verify the Manifest

Each plugin must contain a valid [`.codex-plugin/plugin.json`](https://github.com/openai/plugins/blob/main/.codex-plugin/plugin.json) file. Run the lint command from the repository root:

```bash
node ./plugins/plugin-eval/scripts/plugin-eval.js analyze ./plugins/<plugin-name> --format markdown

```

This checks for required fields (`name`, `description`, `version`, `manifestVersion`) and validates that the JSON schema is well-formed. If the manifest is malformed, `plugin-eval` reports the exact line and field requiring correction.

### Run Static Analysis

The `analyze` command performs comprehensive validation beyond basic JSON syntax:

- **Skill front-matter**: Validates presence of `retrieval.aliases`, `intents`, and `pathPatterns` in [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) files
- **Asset consistency**: Confirms all referenced files exist and flags stray binaries
- **Safety constraints**: Detects hard-coded secrets and verifies proper permission declarations

The tool explicitly protects sensitive values by never printing secret environment variables like `OPENAI_API_KEY` in its output, as noted in the `plugin-eval` documentation.

## Testing Plugin Execution

Static analysis ensures structural correctness, but you must also verify runtime behavior.

### Perform Chat-First Sanity Checks

The `start` command simulates a natural language request and displays the exact CLI sequence that Codex will execute:

```bash
node ./plugins/plugin-eval/scripts/plugin-eval.js start ./plugins/<plugin-name> \
  --request "Evaluate this plugin." \
  --format markdown

```

This outputs the routed skill, the subsequent `analyze` command, and any additional steps like `init-benchmark`. Review this output to confirm that Codex correctly maps user intents to your plugin's capabilities.

### Benchmark Live Executions

For plugins that interact with external services (such as Vercel deployments), generate a realistic usage profile:

```bash
node ./plugins/plugin-eval/scripts/plugin-eval.js init-benchmark ./plugins/<plugin-name>
node ./plugins/plugin-eval/scripts/plugin-eval.js benchmark ./plugins/<plugin-name> --format markdown

```

The benchmark writes a JSONL usage log to `.plugin-eval/runs/<timestamp>/usage.jsonl`. You can then generate a measurement plan to analyze resource utilization:

```bash
node ./plugins/plugin-eval/scripts/plugin-eval.js measurement-plan ./plugins/<plugin-name> \
  --observed-usage .plugin-eval/runs/<timestamp>/usage.jsonl \
  --format markdown

```

The full benchmark harness specification is documented in [`plugins/plugin-eval/references/benchmark-harness.md`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/references/benchmark-harness.md).

## Automating the Testing Workflow

Combine these steps into a reusable script to iterate quickly during development.

### Full Local Testing Script

Save this as [`test-plugin.sh`](https://github.com/openai/plugins/blob/main/test-plugin.sh) in your repository root:

```bash
#!/usr/bin/env bash
set -euo pipefail

PLUGIN=render   # change to your plugin name

ROOT=$(pwd)

# Link plugin into marketplace

mkdir -p ~/.agents/plugins
ln -sfn "$ROOT/plugins/$PLUGIN" "$HOME/.agents/plugins/$PLUGIN"

# Restart Codex (if CLI is installed)

codex restart   # or quit/reopen the Codex UI

# Static analysis

node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  analyze "./plugins/$PLUGIN" --format markdown

# Chat-first sanity check

node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  start "./plugins/$PLUGIN" \
  --request "Evaluate this plugin." \
  --format markdown

# Optional benchmark

node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  init-benchmark "./plugins/$PLUGIN"
node "$ROOT/plugins/plugin-eval/scripts/plugin-eval.js" \
  benchmark "./plugins/$PLUGIN" --format markdown

echo "✅ All tests passed for $PLUGIN"

```

Make it executable and run:

```bash
chmod +x test-plugin.sh
./test-plugin.sh

```

### CI-Friendly Direct Execution

If you prefer not to modify your local marketplace (useful for CI pipelines), invoke the CLI directly against any plugin path:

```bash
node ./plugins/plugin-eval/scripts/plugin-eval.js analyze ./plugins/<plugin-name> --format markdown

```

This works without marketplace registration, though it skips the full Codex discovery simulation that requires the symlink setup.

## Summary

- **Install locally** by symlinking your plugin to `~/.agents/plugins/` and updating [`marketplace.json`](https://github.com/openai/plugins/blob/main/marketplace.json) so Codex can discover it
- **Validate structure** using `plugin-eval analyze` to check [`plugin.json`](https://github.com/openai/plugins/blob/main/plugin.json) schema and skill front-matter in `plugins/<plugin-name>/skills/*.md`
- **Test execution** with `plugin-eval start` to simulate user requests and verify command routing
- **Benchmark performance** using `plugin-eval init-benchmark` and `plugin-eval benchmark` to generate usage profiles and measurement plans
- **Iterate** until `plugin-eval` reports zero errors and all diagnostics pass

## Frequently Asked Questions

### Where does Codex look for local plugins?

Codex scans the `$HOME/.agents/plugins` directory to discover locally installed plugins. You must create this directory and symlink your plugin source code into it, then register the plugin in a [`marketplace.json`](https://github.com/openai/plugins/blob/main/marketplace.json) file within that directory.

### What does the `plugin-eval` CLI check during static analysis?

According to the [`plugins/plugin-eval/README.md`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/README.md), the `analyze` command validates the [`plugin.json`](https://github.com/openai/plugins/blob/main/plugin.json) manifest schema, verifies skill front-matter includes `retrieval.aliases` and `intents`, checks that all referenced assets exist, and enforces safety constraints by detecting hard-coded secrets and validating permission declarations.

### Can I test plugins without installing them into the marketplace?

Yes. You can run `node ./plugins/plugin-eval/scripts/plugin-eval.js analyze <path>` directly against any plugin directory. This bypasses the marketplace registration requirement and works well for CI environments, though it skips the full Codex discovery simulation that requires the symlink setup in `~/.agents/plugins`.

### How do I fix validation errors reported by `plugin-eval`?

Common failures include missing [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) front-matter fields, incorrect paths in [`.codex-plugin/plugin.json`](https://github.com/openai/plugins/blob/main/.codex-plugin/plugin.json), or unchecked environment variables. The tool reports specific line numbers and field names. Fix the reported issues in your source files and re-run the relevant `plugin-eval` command until the output shows no errors.