# How to Evaluate the Success of Knowledge Editing in LLMs: The EasyEdit Metrics Framework

> Learn how to evaluate knowledge editing in LLMs with the EasyEdit metrics framework. Discover reliability, generality, locality, and portability measures for successful fact updates without compromising model capabilities.

- Repository: [Tongxin Yuan/dive-into-llms](https://github.com/Lordog/dive-into-llms)
- Tags: best-practices
- Published: 2026-04-16

---

**Knowledge editing success is measured through four complementary metrics—reliability, generality, locality, and portability—implemented in the EasyEdit toolkit to verify that specific facts are updated without degrading general model capabilities.**

The `Lordog/dive-into-llms` repository provides a comprehensive tutorial on model editing using the **EasyEdit** framework, which includes a unified evaluation system for assessing edit quality. When you modify a language model's internal representation of a specific fact, you must verify that the change took effect properly, generalizes to related contexts, avoids side-effects on neighboring knowledge, and remains robust across model variants. According to the source documentation in [`documents/chapter3/README.md`](https://github.com/Lordog/dive-into-llms/blob/main/documents/chapter3/README.md), EasyEdit automates this assessment through a built-in Evaluate module that returns standardized metrics.


## The Four Core Metrics for Evaluating Knowledge Editing

EasyEdit evaluates knowledge editing through four dimensions that together determine whether an edit is successful. These metrics are automatically computed when you call `editor.edit()` and returned in a `metrics` dictionary.

### Reliability: Did the Edit Take Effect?

**Reliability** measures whether the model correctly outputs the new fact when presented with the target prompt. This metric checks accuracy on the specific edited entry—if you changed "Lionel Messi plays football" to "basketball," reliability verifies the model now answers "basketball." A high reliability score confirms the editing mechanism successfully modified the model's parameters for that specific fact.

### Generality: Does the Knowledge Generalize?

**Generality** tests whether the edited fact propagates to semantically related prompts involving the same entity in different contexts. For example, if you edit Messi's sport, generality checks whether the model correctly answers "What sport does Messi play professionally?" or "Messi's favorite game is..." This ensures the edit is not overfitted to the exact prompt template used during editing.

### Locality: Are Side-Effects Contained?

**Locality** evaluates whether the edit remains **local** to the target fact without corrupting unrelated knowledge about similar entities. This metric uses **neighborhood** prompts—syntactically similar but semantically unrelated statements (e.g., "Larry Bird is a professional..." when editing Messi's sport). High locality scores indicate the model avoided **forgetting** or **distorting** adjacent facts, which is critical for maintaining model stability after multiple edits.

### Portability: Is the Edit Robust Across Model Variants?

**Portability** assesses whether the edited knowledge persists when the model is fine-tuned, distilled, or transferred to different architectures. This measures the **durability** of the edit across model modifications, ensuring that the updated fact remains accessible even as the model evolves or is deployed in resource-constrained environments using smaller variants.


## Implementing Knowledge Editing Evaluation with EasyEdit

According to the [`documents/chapter3/README.md`](https://github.com/Lordog/dive-into-llms/blob/main/documents/chapter3/README.md) in `Lordog/dive-into-llms`, the EasyEdit framework returns evaluation metrics automatically through the `editor.edit()` method. When you execute an edit, the framework computes the four metrics based on the provided prompts and optional evaluation datasets.

The `editor.edit()` function accepts parameters including `prompts`, `ground_truth`, `target_new`, `subject`, and optional `locality_inputs`. After execution, it returns a tuple containing `metrics`, `edited_model`, and additional outputs. The `metrics` dictionary contains the reliability, generality, locality, and portability scores computed by the Evaluate module.

```python

# Install EasyEdit as documented in chapter3

# pip install git+https://github.com/zjunlp/EasyEdit.git

from easyeditor import BaseEditor
from easyeditor import ROMEHyperParams

# Prepare edit specification

prompts = ["Question:What sport does Lionel Messi play? Answer:"]
ground_truth = ["football"]           # Original fact to be replaced

target_new = ["basketball"]           # Desired new fact

subject = ["Lionel Messi"]            # Entity being edited

# Configure editing hyperparameters

hparams = ROMEHyperParams.from_hparams("./hparams/ROME/gpt2-xl.yaml")
editor = BaseEditor.from_hparams(hparams)

# Execute edit and retrieve evaluation metrics

metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    ground_truth=ground_truth,
    target_new=target_new,
    subject=subject,
    keep_original_weight=False
)

# Access evaluation results

print(f"Reliability: {metrics['reliability']}")
print(f"Generality: {metrics['generality']}")
print(f"Locality: {metrics['locality']}")
print(f"Portability: {metrics.get('portability', 'N/A')}")

```

The `metrics` dictionary returned by `editor.edit()` contains numeric scores (typically between 0 and 1) for each dimension, providing a quantitative success report for the knowledge editing operation.


## Constructing Locality and Generality Test Sets

To properly **evaluate locality**, you must provide `locality_inputs` when calling `editor.edit()`. As shown in [`documents/chapter3/README.md`](https://github.com/Lordog/dive-into-llms/blob/main/documents/chapter3/README.md) (lines 80-88), these inputs consist of neighborhood prompts that are syntactically similar to the target but semantically unrelated to the edit.

The `locality_inputs` dictionary expects a "neighborhood" key containing `prompt` and `ground_truth` lists. These prompts test whether the model incorrectly changes its behavior on unrelated facts after editing. For example, when editing Messi's sport, you would test whether the model still correctly identifies Larry Bird's sport or Joseph Fischhof's profession.

```python

# Constructing locality evaluation inputs

locality_inputs = {
    "neighborhood": {
        "prompt": [
            "Joseph Fischhof, the",
            "Larry Bird is a professional",
            "In Forssa, they understand"
        ],
        "ground_truth": ["piano", "basketball", "Finnish"]
    }
}

# Pass locality_inputs to editor.edit() to enable locality metric calculation

metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    ground_truth=ground_truth,
    target_new=target_new,
    subject=subject,
    locality_inputs=locality_inputs,  # Enables locality evaluation

    keep_original_weight=False
)

```

For **generality** and **portability**, you can provide additional test sets following the same pattern, though the specific parameter names may vary based on the EasyEdit configuration. The Evaluate module automatically computes scores based on whatever evaluation data you provide during the `edit()` call.


## Summary

Evaluating knowledge editing success in large language models requires a multi-dimensional approach that balances effectiveness against stability. The key takeaways from the `Lordog/dive-into-llms` EasyEdit implementation include:

- **Four metrics define success**: Reliability confirms the edit works on the target prompt; generality ensures it spreads to related contexts; locality prevents damage to neighboring knowledge; and portability verifies robustness across model variants.
- **Automated evaluation via `editor.edit()`**: The EasyEdit framework automatically computes these metrics when you invoke the edit method, returning them in a `metrics` dictionary.
- **Locality requires explicit test data**: To measure locality, you must provide `locality_inputs` containing neighborhood prompts that are syntactically similar but semantically unrelated to your edit.
- **Quantitative assessment**: Each metric returns a numeric score (typically 0-1), allowing you to determine whether an edit meets your reliability and safety thresholds before deployment.


## Frequently Asked Questions

### What is knowledge editing in LLMs?

Knowledge editing is a technique that modifies specific factual associations within a pre-trained language model's parameters without retraining the entire model. Unlike fine-tuning, which updates the model broadly, knowledge editing targets discrete facts—such as changing "Paris is the capital of France" to "Lyon is the capital of France"—while attempting to preserve the model's general knowledge and capabilities.

### How does EasyEdit calculate the reliability metric?

According to the [`documents/chapter3/README.md`](https://github.com/Lordog/dive-into-llms/blob/main/documents/chapter3/README.md) in `Lordog/dive-into-llms`, EasyEdit calculates reliability by measuring the accuracy of the edited model on the exact target prompt provided during the `editor.edit()` call. The framework compares the model's output against the `target_new` value specified in the edit request. The reliability score reflects whether the model correctly produces the new fact when queried with the original prompt template.

### What is the difference between locality and generality in knowledge editing evaluation?

Locality and generality measure different aspects of edit propagation. **Generality** tests whether the edited fact generalizes to semantically related prompts involving the same entity in different contexts—ensuring the model understands the new fact broadly. **Locality**, conversely, tests whether the edit remains confined to the target fact by checking accuracy on syntactically similar but semantically unrelated neighborhood prompts—ensuring the model hasn't corrupted adjacent knowledge. High generality indicates successful knowledge transfer; high locality indicates minimal side-effects.

### Can I evaluate knowledge editing on custom datasets using EasyEdit?

Yes, EasyEdit supports custom evaluation datasets through the `locality_inputs` parameter and additional test configurations passed to `editor.edit()`. You can construct your own neighborhood prompts for locality testing or create paraphrased prompts for generality testing. The framework's Evaluate module processes any provided groundtruth-answer pairs to compute the four standard metrics, allowing you to benchmark editing methods against domain-specific or proprietary datasets beyond the built-in benchmarks.