# Implementing Reinforcement Fine-Tuning with Model Graders: The OpenAI Cookbook Approach

> Learn to implement reinforcement fine-tuning with model graders using the OpenAI Cookbook. Build a continuous improvement loop for better AI model outputs.

- Repository: [OpenAI/openai-cookbook](https://github.com/openai/openai-cookbook)
- Tags: how-to-guide
- Published: 2026-03-02

---

**Reinforcement fine-tuning with model graders creates a continuous improvement loop where specialized evaluators score model outputs, aggregate those scores into reward signals, and iteratively refine prompts before fine-tuning the model on optimized completions.**

Implementing reinforcement fine-tuning with model graders enables automated quality assessment and iterative model improvement without manual intervention. The openai/openai-cookbook repository demonstrates this pattern through production-ready notebooks that combine OpenAI Eval, custom grader definitions, and the fine-tuning API into a cohesive optimization pipeline.

## Core Components of Model-Based Grading

### Grader Definitions and Testing Criteria

In `examples/partners/self_evolving_agents/autonomous_agent_retraining.ipynb`, **model graders** are implemented as specialized OpenAI Eval tests where the *testing criteria* is itself a prompt that instructs a judge model—typically `gpt-4o` or `gpt-4-turbo`—to produce numeric scores or binary pass/fail judgments. The cookbook defines four primary grader archetypes:

- **`chemical_name_grader`** – Validates whether chemical names appear correctly in generated content
- **`word_length_deviation_grader`** – Enforces conciseness by checking output length against constraints
- **`cosine_similarity`** – Measures semantic similarity between generated and reference outputs
- **`llm_as_judge`** – A generic evaluator that returns confidence scores along with explanatory reasoning

Each grader outputs a structured payload containing the score, pass status, and optional reasoning text that feeds into the feedback loop.

### Eval Creation and Execution

The notebook demonstrates bundling multiple graders into a single **Eval** object using `client.create_eval` with a list of grader specifications. When the optimization loop calls `client.run_eval`, the system executes all graders against the candidate output and returns a JSON array of results:

```json
[
  {"grader_name":"chemical_name_grader-<id>","score":0.5,"passed":false},
  {"grader_name":"word_length_deviation_grader-<id>","score":0.8,"passed":true},
  {"grader_name":"cosine_similarity-<id>","score":0.91,"passed":true},
  {"grader_name":"llm_as_judge-<id>","score":0.8,"passed":true}
]

```

This multi-dimensional scoring enables granular assessment of output quality across distinct dimensions.

## Building the Reinforcement Loop

### Score Aggregation and Pass Logic

The cookbook provides helper functions in `autonomous_agent_retraining.ipynb` to transform raw grader outputs into actionable metrics. The **`calculate_grader_score`** function computes average scores across individual graders, while **`calculate_total_grader_score`** aggregates these into an overall performance metric. The **`is_lenient_pass`** function implements a pragmatic early-stopping condition that accepts candidates meeting a threshold (e.g., ≥ 75 % of graders passing or average score ≥ 0.85), preventing over-optimization on perfect scores.

### Feedback Synthesis with `collect_grader_feedback`

The function **`collect_grader_feedback`** extracts reasoning fields from each grader result and merges them into a textual prompt. This synthesized feedback feeds into a *meta-prompt* that instructs the target model how to improve, effectively converting scalar scores into actionable textual guidance for the next iteration.

### The Optimization Workflow

Section 3 of the notebook implements the core reinforcement loop through functions like **`apply_best_candidate_if_needed`**. The workflow caches eval results for `(section, summary)` pairs to minimize redundant API calls, then repeatedly:

1. Generates candidate responses using the current system prompt
2. Executes the eval suite via `client.run_eval`
3. Checks `is_lenient_pass` to determine if quality thresholds are met
4. If thresholds fail, uses `collect_grader_feedback` to rewrite the prompt via `client.chat.completions.create`
5. Stores the best-scoring prompt version for downstream fine-tuning

This caching mechanism significantly reduces API usage during iterative refinement.

## From Prompt Optimization to Model Fine-Tuning

Once the optimization loop identifies a high-performing prompt version, the cookbook demonstrates constructing a fine-tuning dataset that pairs original inputs with the *optimized* outputs generated using the refined prompt. The OpenAI Fine-tuning API is then invoked via `client.fine_tuning.jobs.create` to produce a **reinforcement-fine-tuned** model that internalizes the grader-derived preferences.

After fine-tuning completes, the same grader suite runs against the new model to verify that average scores improved and that the lenient pass criteria are satisfied more consistently than the base model.

## Domain-Specific Implementations

The pattern extends beyond autonomous agents. In `examples/partners/eval_driven_system_design/receipt_inspection.ipynb`, the cookbook demonstrates building a **model grader** for receipt item extraction tasks, showing how to plug domain-specific evaluators into the same reinforcement pipeline. The [`registry.yaml`](https://github.com/openai/openai-cookbook/blob/main/registry.yaml) file catalogs these notebooks under the "partners" tag, enabling discovery of reinforcement fine-tuning examples across different domains.

For advanced use cases, the notebook references the **GEPAAdapter** (around line 1660) to integrate third-party reinforcement-learning libraries while preserving the OpenAI Eval grading interface.

## Summary

- **Model graders act as differentiable reward signals** by outputting scalar scores that aggregate into loss-like metrics for optimization
- **Iterative prompt rewriting bridges evaluation and training** by converting grader reasoning into actionable system prompt improvements before committing to expensive fine-tuning jobs
- **Lenient pass criteria provide pragmatic convergence** by accepting high-quality candidates that meet aggregate thresholds rather than requiring perfect scores from every grader
- **Caching eval results** via `(section, summary)` memoization reduces API costs during the reinforcement loop
- **The final fine-tuning stage** freezes the optimized behavior into model weights using `client.fine_tuning.jobs.create` with datasets constructed from the best prompt version

## Frequently Asked Questions

### What is a model grader in the context of reinforcement fine-tuning?

A model grader is an OpenAI Eval test that uses a language model—such as `gpt-4o`—as a judge to evaluate output quality according to specific criteria. Unlike static heuristics, model graders can assess nuanced qualities like factual correctness, semantic similarity, and adherence to complex constraints, returning structured scores that serve as reward signals for the reinforcement loop.

### How does the lenient pass criteria determine when to stop iterating?

The `is_lenient_pass` function implements an aggregate threshold—typically requiring 75 % of graders to pass or an average score above 0.85—rather than demanding perfect scores from every evaluator. This approach prevents over-optimization, reduces API costs by enabling early stopping, and accepts high-quality candidates that excel across the majority of evaluation dimensions even if individual graders flag minor issues.

### Can I implement custom graders for specialized domains like code generation?

Yes. The cookbook demonstrates this extensibility in both `autonomous_agent_retraining.ipynb` and `receipt_inspection.ipynb`. You define custom graders by specifying new entries in the `example_graders` list with domain-specific testing criteria, such as regex patterns for code syntax validation, static analysis integration, or LLM-as-judge prompts trained to evaluate algorithmic correctness.

### What distinguishes prompt rewriting from the final fine-tuning stage?

**Prompt rewriting** occurs during the reinforcement loop as a rapid, low-cost iteration mechanism that modifies the system prompt based on `collect_grader_feedback` output to improve immediate responses. **Fine-tuning** occurs after the loop converges on an optimal prompt, using that prompt to generate a static training dataset that is uploaded via `openai.File.create` and processed through `client.fine_tuning.jobs.create` to permanently embed the grader-derived preferences into the model's weights.