What Does a High KL Divergence Score Mean in Heretic? Interpretation Guide

A high KL divergence score in Heretic indicates that the ablated model has drifted significantly from the original model's probability distribution, signaling potential degradation of core language capabilities and factual knowledge.

In the Heretic framework for model unlearning, KL divergence serves as a critical guardrail during optimization. This metric quantifies how much the probability distribution over the first generated token of the ablated model differs from that of the base model when both process the same set of "good" (non-refusal-inducing) prompts.

How Heretic Calculates KL Divergence

The calculation occurs in src/heretic/evaluator.py within the get_score method. The evaluator computes per-prompt log-probabilities using model.get_logprobs_batched for both the current trial and the untouched base model. These tensors feed into torch.nn.functional.kl_div with specific parameters:

• reduction="batchmean" – averages across the batch dimension
• log_target=True – indicates the target is already log-probabilities

The resulting scalar is stored as trial.user_attrs["kl_divergence"] in src/heretic/main.py (lines 542-556) and subsequently displayed in the CLI UI and markdown reports.

Interpreting KL Divergence Values in Heretic

The magnitude of the KL divergence directly correlates with model fidelity. Heretic implements specific thresholds to guide interpretation:

KL Divergence	Interpretation
≈ 0	The ablated model behaves almost identically to the base model on good prompts, indicating minimal alteration of knowledge or language ability.
0 – 0.01	The default target range (settings.kl_divergence_target). The model successfully avoids refusing harmful prompts while preserving original capabilities.
> 0.01	The optimizer begins penalizing the trial. The model distribution is drifting away from the original, potentially compromising performance.
> 1	Critical threshold flagged in the UI (src/heretic/main.py, lines 662-668). Significant damage to core capabilities—fluency, factual recall, and coherence. The model may no longer resemble the original.

KL Divergence in Multi-Objective Optimization

Heretic employs KL divergence as a constraint within a multi-objective optimization framework managed by Optuna. Two configuration parameters in src/heretic/config.py (lines 163-176) govern its behavior:

• kl_divergence_target (default: 0.01) – The desired maximum divergence
• kl_divergence_scale (default: 1.0) – Scaling factor for the penalty term

When the measured KL falls below the target, the optimizer prioritizes reducing refusals. When KL exceeds the target, the divergence penalty dominates the objective function, preventing runaway model damage.

Accessing and Configuring KL Divergence

Retrieving KL Scores from Completed Trials

import optuna

# Assume `study` is an optuna Study that has completed trials

for trial in study.trials:
    if trial.state == optuna.trial.TrialState.COMPLETE:
        kl = trial.user_attrs["kl_divergence"]
        refusals = trial.user_attrs["refusals"]
        print(f"Trial {trial.user_attrs['index']}: KL={kl:.4f}, refusals={refusals}")

The user_attrs dictionary is populated in src/heretic/main.py after each evaluation.

Adjusting KL Targets in Configuration

# config.toml – user-provided overrides

kl_divergence_target = 0.02   # relax the target slightly

kl_divergence_scale = 2.0     # make KL penalties less aggressive

These fields are defined in src/heretic/config.py and directly influence the optimization objective.

Summary

• KL divergence in Heretic measures distribution shift between the ablated and base models on good prompts, calculated in src/heretic/evaluator.py using torch.nn.functional.kl_div.
• High scores (>0.01) trigger optimization penalties; scores >1 indicate severe model degradation and trigger UI warnings in src/heretic/main.py.
• The default target of 0.01 balances refusal reduction with capability preservation, configurable via kl_divergence_target and kl_divergence_scale in src/heretic/config.py.
• Access runtime KL values through trial.user_attrs["kl_divergence"] after evaluation.

Frequently Asked Questions

What is the default KL divergence target in Heretic?

The default target is 0.01, defined in src/heretic/config.py as kl_divergence_target. This value represents the desired maximum divergence where the model should maintain original capabilities while avoiding refusals. The optimizer treats values below this threshold as acceptable and begins penalizing trials that exceed it.

How does Heretic calculate KL divergence between models?

Heretic calculates KL divergence in src/heretic/evaluator.py by first obtaining log-probabilities for the next token from both the ablated trial model and the base model using model.get_logprobs_batched. These tensors are passed to torch.nn.functional.kl_div with reduction="batchmean" and log_target=True, producing a single scalar value that quantifies the distribution difference.

What should I do if my KL divergence exceeds 1.0?

If KL divergence exceeds 1.0, Heretic displays a warning in the UI (implemented in src/heretic/main.py, lines 662-668) indicating significant model damage. You should consider stopping the optimization and either adjusting your ablation parameters or increasing the kl_divergence_scale in your configuration to apply stronger penalties earlier, preventing further degradation of core language capabilities.

Can I adjust how strongly KL divergence affects the optimization score?

Yes, you can adjust the impact of KL divergence through the kl_divergence_scale parameter in src/heretic/config.py (default 1.0). Increasing this value makes the penalty term more aggressive when divergence exceeds the target, while decreasing it allows more flexibility at the risk of greater model drift. You can also modify kl_divergence_target (default 0.01) to set a different acceptable threshold for your specific use case.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}