# How to Interpret Validation Loss Curves During Needle Fine-Tuning: A Complete Guide

> Learn to interpret validation loss curves for Needle fine-tuning. Spot overfitting and improve model performance by understanding these crucial training indicators.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: tutorial
- Published: 2026-08-16

---

**Validation loss curves indicate whether your Needle model is learning transferable patterns or over-fitting to the training data—when validation loss rises while training loss falls, stop training immediately.**

Needle, the lightweight JAX-based fine-tuning framework by Cactus Compute, reports two loss values at every epoch during `needle finetune`. Understanding the relationship between these curves is essential for training efficient, generalizable models. This guide walks through exactly how Needle computes validation loss, what different curve patterns mean, and how to act on them based on the `cactus-compute/needle` source code.

## What Validation Loss Represents in Needle

When you run `needle finetune`, the training loop maintains a **hold-out validation set** (default 10% of your data) that is never used for gradient updates. At the end of each epoch, Needle computes the average loss on this set and prints it alongside training loss.

The metric definitions are straightforward:

- **Training loss** — Average loss on batches used for weight updates; should decrease steadily
- **Validation loss** — Average loss on held-out data; computed without parameter updates

Validation loss appears in console output like this:

```

epoch 3  loss 0.4321  val 0.4512

```

This line is emitted by [[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py#L381-L385), specifically at line 385:

```python
emit(f"  {'epoch':<9} {epoch + 1}/{args.epochs}  loss {last:.4f}  val {val:.4f}")

```

## How Needle Computes Validation Loss

The validation loss calculation uses JIT-compiled evaluation without gradients. In [`finetune.py`](https://github.com/cactus-compute/needle/blob/main/finetune.py) lines 366 and 381-384:

```python
eval_step = jax.jit(loss_fn)                               # line 366

...
val = np.mean([float(eval_step(lora,
                               jnp.asarray(val_seqs[i:i + batch]),
                               jnp.asarray(val_masks[i:i + batch])))
               for i in range(0, n_val, batch)])          # lines 381-384

```

Key implementation details:

- `loss_fn` returns standard cross-entropy loss for each batch
- `eval_step` runs without gradient computation—parameters remain frozen
- The mean across all validation batches produces the final `val` number

This design ensures validation loss reflects true generalization capability, not training-set memorization.

## Reading the Validation Loss Curve: Four Critical Patterns

### Pattern 1: Both Losses Decreasing

When training loss and validation loss both trend downward, your model is learning generalizable patterns. Continue training unless computational budget forces early stopping.

### Pattern 2: Training Loss ↓ While Validation Loss ↑

This divergence signals **over-fitting**. The model memorizes training examples rather than learning underlying structure. According to [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md):

> "When it rises while the training loss keeps falling, the run is overfitting: stop there, or add data."

Recommended actions:

- **Early stopping** — Halt at the epoch with lowest validation loss
- **Increase data diversity** — Add examples or apply data augmentation
- **Adjust validation split** — Use `--val-split` to change hold-out proportion

### Pattern 3: Validation Loss Plateaus

A flat validation curve while training loss improves suggests diminishing returns. The model may still benefit from continued training, or you may be near convergence. Monitor for several epochs before deciding.

### Pattern 4: Validation Loss Much Higher Than Training Loss From Start

A large initial gap indicates insufficient model capacity relative to problem complexity, or potential data preprocessing issues. Consider:

- Reducing model complexity (opposite of typical intuition—may indicate too *simple* a model can't capture patterns)
- Verifying train/validation data distributions match
- Checking for label leakage or preprocessing errors

## Controlling the Validation Split

Needle exposes validation set size through the `--val-split` CLI argument, defined in [[`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)](https://github.com/cactus-compute/needle/blob/main/needle/cli.py#L130-L131):

```python
p.add_argument("--val-split", type=float, default=0.1,
               help="Fraction of examples held out for validation (0 disables)")

```

Usage examples:

```bash

# Default 10% validation split

needle finetune --batch-size 16 --epochs 5 data/train.jsonl

# Larger 20% validation set for more stable estimates

needle finetune --batch-size 16 --epochs 5 --val-split 0.2 data/train.jsonl

```

Setting `--val-split 0` disables validation entirely—not recommended for production workflows.

## Practical Monitoring Workflow

### Step 1: Capture Structured Output

Redirect training logs for programmatic analysis:

```bash
needle finetune --batch-size 16 --epochs 10 --val-split 0.15 data/train.jsonl 2>&1 | tee training.log

```

### Step 2: Parse Loss Values

Extract epoch-level metrics for plotting:

```bash
grep "epoch" training.log | awk '{print $2,$6,$8}'

# Output format: epoch_number training_loss validation_loss

# 1 0.4321 0.4512

# 2 0.3987 0.4205

# 3 0.3870 0.4301

```

### Step 3: Identify the inflection point

A typical problematic run shows validation loss minimum around epoch 2, then increase:

```

epoch     1/5  loss 0.4321  val 0.4512
epoch     2/5  loss 0.3987  val 0.4205   <-- lowest validation loss
epoch     3/5  loss 0.3870  val 0.4301   <-- rising; over-fitting begins

```

Stop at epoch 2 and use the corresponding checkpoint.

## Troubleshooting Flat Validation Curves

If validation loss never changes:

| Checkpoint | Action |
|------------|--------|
| Verify `--val-split` > 0 | Ensure validation isn't disabled |
| Check `n_val > 0` in logs | Confirm hold-out set contains examples |
| Inspect data shuffling | Validation examples should be representative |
| Review `val_seqs`/`val_masks` | These arrays feed the eval loop in [`finetune.py`](https://github.com/cactus-compute/needle/blob/main/finetune.py) |

The training script reports hold-out size at startup: `holdout   200 examples for validation`. If this line is missing or shows 0, your validation split configuration needs adjustment.

## Summary

- **Validation loss** in Needle measures generalization on a held-out set, computed without gradients via `jax.jit(loss_fn)`
- **Rising validation loss** while training loss falls is the critical over-fitting signal—implement early stopping immediately
- **Control validation set size** with `--val-split` (default 0.1, defined in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) line 130)
- **Monitor programmatically** by capturing and parsing epoch lines from `needle finetune` output
- **Key source files**: [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) (computation), [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) (configuration), [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) (interpretation guidance)

## Frequently Asked Questions

### What does it mean if validation loss is lower than training loss?

This occurs occasionally early in training when regularization (dropout, weight decay) is applied during training but not validation. As training progresses, training loss should typically drop below validation loss. Persistent inversion may indicate data leakage between splits.

### How do I implement automatic early stopping in Needle?

The current Needle release (`cactus-compute/needle`) does not include built-in early stopping with patience. You must manually monitor validation loss curves and interrupt training. Parse the log output as shown above, identify the epoch with minimum validation loss, and load the corresponding checkpoint.

### Why does my validation loss fluctuate between epochs?

Small validation sets (default 10%) produce noisy estimates. Increase `--val-split` to 0.15–0.20 for stabler curves, or aggregate multiple training runs. The fluctuation diminishes as dataset size grows.

### Can I disable validation to speed up training?

Yes—set `--val-split 0`—but this removes your primary over-fitting detection mechanism. Only disable validation when you have separate evaluation protocols or extremely limited compute budgets. The parameter is defined in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) line 130 with help text explicitly warning that 0 disables validation.