# Testing and Debugging Plugin Configurations for OpenAI Projects: A Complete Guide

> Learn to test and debug plugin configurations for OpenAI projects with the plugin-eval CLI tool. Validate manifests, debug policy mismatches, and verify asset paths locally.

- Repository: [OpenAI/plugins](https://github.com/openai/plugins)
- Tags: how-to-guide
- Published: 2026-06-27

---

**The OpenAI Plugins repository provides the `plugin-eval` CLI tool that analyzes, reports, and benchmarks plugin configurations locally, enabling developers to validate manifests, debug policy mismatches, and verify asset paths before publishing to the Codex marketplace.**

The **openai/plugins** repository contains a curated collection of **Codex** plugin examples that convert third-party services into conversational agents. When testing and debugging plugin configurations for OpenAI projects, developers navigate a standardized directory structure and leverage specialized evaluation tools to ensure compatibility with the Codex runtime. Understanding the relationship between manifest files, skill definitions, and policy configurations is essential for creating robust plugins that pass validation.

## Plugin Architecture and Directory Structure

Every plugin in the repository follows a consistent layout under `plugins/<plugin-name>/`. This structure separates metadata, business logic, and visual assets into distinct components that the Codex runtime consumes.

### The Manifest File (plugin.json)

The **manifest** at `plugins/<plugin-name>/.codex-plugin/plugin.json` drives the Codex runtime. This JSON document declares the plugin's public interface (`interface` object), including fields like `displayName`, `shortDescription`, `longDescription`, and UI cues such as `composerIcon` and `logo`. It also references the plugin's skills via the `skills` field and declares required authentication through `policy.installation` and `policy.authentication` objects.

### Skills and Agent Policies

Each skill resides in `plugins/<plugin-name>/skills/` as a self-contained workflow. The [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) file contains YAML front-matter describing the skill's name, description, and configuration. The `agents/` subfolder contains policy files like [`openai.yaml`](https://github.com/openai/plugins/blob/main/openai.yaml) that specify whether the skill permits `allow_implicit_invocation` or requires explicit prompts. The system parses front-matter using the `parseFrontmatter` utility from [`src/lib/frontmatter.js`](https://github.com/openai/plugins/blob/main/src/lib/frontmatter.js) and validates policy through the `parseCodexJsonStream` pipeline.

### Marketplace Integration

The central catalog at [`.agents/plugins/marketplace.json`](https://github.com/openai/plugins/blob/main/.agents/plugins/marketplace.json) aggregates all publishable plugins, defining categories, installation policies, and source paths. This file powers the Codex UI that lets users discover and install plugins, making it the authoritative registry for plugin distribution.

## Testing Configurations with the Plugin-Eval CLI

The **plugin-eval** command-line tool, located at `plugins/plugin-eval/`, provides a local evaluation engine that simulates Codex runtime behavior. The entry point at [`plugins/plugin-eval/scripts/plugin-eval.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/scripts/plugin-eval.js) exposes commands for analysis, reporting, comparison, and benchmarking.

### Analyzing Skills and Computing Token Budgets

Use the `analyze` command to parse skill front-matter, compute token-budget estimates, and run static analysis such as cyclomatic complexity checks. The core logic resides in [`plugins/plugin-eval/src/core/analyze.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/src/core/analyze.js).

```bash
plugin-eval analyze plugins/zoom/skills/video-sdk \
  --output result.json \
  --brief-out brief.json \
  --observed-usage plugins/zoom/skills/video-sdk/observed-usage/responses.jsonl

```

This command validates that [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) files contain parsable front-matter and that referenced assets exist relative to the plugin root.

### Generating Reports and Benchmarks

Convert analysis results into human-readable markdown using the `report` command, which leverages renderers in [`plugins/plugin-eval/src/renderers/index.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/src/renderers/index.js).

```bash
plugin-eval report result.json --format markdown --output report.md

```

For performance testing, the `init-benchmark` and `benchmark` commands simulate Codex CLI workflows:

```bash
plugin-eval init-benchmark plugins/zoom/skills/video-sdk \
  --format markdown --output benchmark.md

plugin-eval benchmark plugins/zoom/skills/video-sdk \
  --format markdown --output run.md

```

### Comparing Configurations

Identify regressions between plugin versions using the `compare` command, which takes two analysis JSON files and outputs differences:

```bash
plugin-eval compare old-result.json new-result.json \
  --format markdown --output diff.md

```

### Explaining Token Budgets

Validate resource limits before deployment:

```bash
plugin-eval explain-budget plugins/zoom/skills/video-sdk

```

## Debugging Common Configuration Issues

The `plugin-eval` test suite at [`plugins/plugin-eval/tests/plugin-eval.test.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/tests/plugin-eval.test.js) demonstrates end-to-end validation workflows. When debugging, focus on three common failure categories:

### Missing Asset References

If the manifest references a `composerIcon` or `logo` that does not exist in `plugins/<plugin-name>/assets/`, the evaluator flags the path error during analysis. Always verify asset paths relative to the plugin root directory.

### Policy Mismatches

When a skill's [`openai.yaml`](https://github.com/openai/plugins/blob/main/openai.yaml) disallows `allow_implicit_invocation` but the manifest expects automatic triggers, the evaluator reports `explicit-only` constraints. Align the agent policy files in `plugins/<plugin-name>/skills/*/agents/openai.yaml` with your manifest's invocation expectations.

### Front-Matter Parsing Errors

Malformed YAML in [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) produces `frontmatter-invalid` checks. While the parser tolerates folded (`>`) and literal (`|`) YAML block scalars, syntax errors or improper indentation cause validation failures. Run the `analyze` command to isolate specific line errors in your front-matter.

## Summary

- The **openai/plugins** repository structures plugins under `plugins/<plugin-name>/` with mandatory manifests at [`.codex-plugin/plugin.json`](https://github.com/openai/plugins/blob/main/.codex-plugin/plugin.json), skills in `skills/`, and optional assets.
- The **plugin-eval** CLI provides commands for `analyze`, `report`, `compare`, `explain-budget`, and `benchmark` to validate configurations locally before marketplace submission.
- Configuration errors commonly involve missing assets referenced in the manifest, mismatched implicit invocation policies between manifests and [`agents/openai.yaml`](https://github.com/openai/plugins/blob/main/agents/openai.yaml) files, and invalid YAML front-matter in [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) files.
- The evaluation engine processes skills through [`plugins/plugin-eval/src/core/analyze.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/src/core/analyze.js) and renders output via [`plugins/plugin-eval/src/renderers/index.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/src/renderers/index.js), with the CLI entry point at [`plugins/plugin-eval/scripts/plugin-eval.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/scripts/plugin-eval.js).

## Frequently Asked Questions

### How do I validate a plugin manifest before submitting to the marketplace?

Run the `plugin-eval analyze` command against your plugin directory and inspect the JSON output for validation errors. The tool checks manifest completeness against the schema used by [`.agents/plugins/marketplace.json`](https://github.com/openai/plugins/blob/main/.agents/plugins/marketplace.json), verifies that `composerIcon` and `logo` assets exist, and ensures all referenced skills contain valid [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) front-matter.

### What causes "frontmatter-invalid" errors in SKILL.md files?

This error occurs when the YAML front-matter at the top of [`SKILL.md`](https://github.com/openai/plugins/blob/main/SKILL.md) contains syntax errors or unsupported formatting. While the `parseFrontmatter` utility from [`src/lib/frontmatter.js`](https://github.com/openai/plugins/blob/main/src/lib/frontmatter.js) supports folded (`>`) and literal (`|`) block scalars, it requires valid YAML delimiters and consistent indentation. Run the analyze command with the `--output` flag to identify specific line errors.

### Can I test implicit invocation behavior locally?

Yes. The `plugin-eval` tool reads the [`openai.yaml`](https://github.com/openai/plugins/blob/main/openai.yaml) policy file in each skill's `agents/` directory to determine invocation rules. If `allow_implicit_invocation` is set to false but your manifest assumes automatic triggers, the comparison report will flag this mismatch before you deploy to the Codex runtime.

### Where is the plugin-eval CLI entry point defined?

The command-line interface is implemented in [`plugins/plugin-eval/scripts/plugin-eval.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/scripts/plugin-eval.js), which exposes the `analyze`, `report`, `compare`, `explain-budget`, and `benchmark` commands. The core analysis logic resides in [`plugins/plugin-eval/src/core/analyze.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/src/core/analyze.js), while output formatting is handled by [`plugins/plugin-eval/src/renderers/index.js`](https://github.com/openai/plugins/blob/main/plugins/plugin-eval/src/renderers/index.js).