# Quiz Debiasing Techniques in AI Engineering: How `debias_quizzes.py` Eliminates Positional Bias

> Discover how debias_quizzes.py eliminates positional bias in AI quizzes. Learn about deterministic, content-seeded permutation for fair answer distribution and semantic integrity.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-09-04

---

**The [`debias_quizzes.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debias_quizzes.py) script removes systematic bias in multiple-choice quiz answer positions through deterministic, content-seeded permutation that shuffles options while preserving semantic anchors and correct answer alignment.**

The `rohitg00/ai-engineering-from-scratch` repository implements sophisticated **quiz debiasing techniques** to ensure fair assessment distribution across its learning phases. This Python script processes JSON quiz files throughout the repository, eliminating patterns where correct answers cluster in specific positions—such as always appearing as option "C" or "D"—that could give learners unfair predictive advantages.

## Detecting Positional Anchors to Preserve Semantic Meaning

Before shuffling, the script identifies questions containing **positional anchors**—language that binds meaning to specific option locations. In [`scripts/debias_quizzes.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/debias_quizzes.py), the regular expression `ANCHOR` (lines 29-34) scans for phrases like "all of the above" or "both A and B" that would become nonsensical if moved.

The helper function `has_positional_anchor` (lines 43-44) flags these questions for exclusion. This safety mechanism ensures that semantic relationships between options remain intact, preventing the debiasing process from corrupting question logic while still processing safe questions.

## Content-Seeded Randomness for Idempotent Shuffling

For questions without anchors, the script derives a deterministic seed from the **content itself** rather than using random entropy. The `seed_for` function (lines 38-40) concatenates the file path with the question text, generating a SHA-256 hash truncated to 16 hexadecimal characters and cast to an integer.

This approach guarantees **idempotent behavior**—running the script multiple times produces identical shuffle orders for the same question. The deterministic seeding ensures reproducibility across different environments while maintaining statistical randomness across the broader question set.

## Deterministic Permutation Algorithm

The core shuffling logic in [`debias_quizzes.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debias_quizzes.py) executes a controlled permutation through three distinct steps. First, options undergo **canonical sorting** (`base = sorted(options, key=str)` on line 62) to eliminate any existing positional bias from the source material. Next, a `random.Random` instance uses the content-derived seed to shuffle a list of indices (`perm` on line 63). Finally, these shuffled indices reorder the base list to produce `new_options` (line 64).

This method ensures that the final arrangement depends solely on the question content, not arbitrary randomness or original file ordering, creating a reproducible yet unbiased distribution.

## Correct Index Realignment

After permutation, the script must update the correct answer pointer to maintain data integrity. The original correct value is preserved, then its new position is located within the shuffled array:

```python
new_correct = new_options.index(correct_val)  # Line 65

```

The updated options array and corrected index are written back to the question object (lines 70-71). This maintains semantic correctness while randomizing presentation order, ensuring the right answer stays correct regardless of its new position.

## Preserving Original File Formatting

When writing modifications, the `serialize` function (lines 75-104) respects the original JSON structure. The script detects whether the `"options"` array was stored as inline or multiline JSON, preserving whitespace and indentation styles exactly.

This formatting awareness minimizes git diffs, making reviews easier and maintaining human-readable quiz files. The debiasing process becomes invisible to content creators while fixing underlying statistical bias in the answer distribution.

## Usage and Reporting Modes

The script supports two operational modes controlled via command-line arguments.

To rewrite quizzes in place with debiased positions:

```bash
python3 scripts/debias_quizzes.py

```

To analyze current bias distribution without modifying files:

```bash
python3 scripts/debias_quizzes.py --check

```

The `main` function (lines 17-70) aggregates statistics on correct-answer distribution across all processed files, reporting how many questions were changed and the uniformity of the new distribution.

Programmatic usage allows integration into larger assessment pipelines:

```python
from scripts.debias_quizzes import debias_question

q = {
    "question": "What is the capital of France?",
    "options": ["Paris", "Berlin", "Madrid", "Rome"],
    "correct": 0,
}

changed = debias_question("phases/01-example/quiz.json", q)
print("Options shuffled?" , changed)
print(q["options"], q["correct"])

```

## Summary

- **Positional anchor detection** identifies questions with ordering-dependent language using the `ANCHOR` regex (lines 29-34) and `has_positional_anchor` helper (lines 43-44), excluding them from shuffling.
- **Content-derived seeding** via `seed_for` creates deterministic, reproducible randomness using SHA-256 hashes of file paths and question text.
- **Canonical sorting and indexed permutation** eliminate existing bias while maintaining content-consistent shuffling through the seeded `random.Random` implementation.
- **Index realignment** preserves correct answer semantics by locating the shuffled position of the original correct value and updating the reference.
- **Formatting preservation** through the `serialize` function (lines 75-104) ensures minimal diffs and human-readable JSON output.
- **Dual execution modes** support both destructive debiasing and read-only analysis via the `--check` flag.

## Frequently Asked Questions

### What are quiz debiasing techniques?

**Quiz debiasing techniques** are algorithmic methods used to eliminate systematic patterns in multiple-choice questions where correct answers cluster in specific positions (such as always being option "C" or "D"). In `rohitg00/ai-engineering-from-scratch`, these techniques include content-seeded randomization, positional anchor detection, and deterministic permutation to ensure fair, unpredictable answer distributions across assessment banks.

### How does [`debias_quizzes.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debias_quizzes.py) ensure the same shuffle order every time?

The script uses **content-seeded randomness** through the `seed_for` function (lines 38-40), which generates a SHA-256 hash from the combination of the file path and question text. This hash seeds Python's `random.Random` generator, guaranteeing that identical questions always receive identical shuffle orders (idempotent behavior) while maintaining randomness across different questions.

### Why are some questions excluded from shuffling?

Questions containing **positional anchors**—phrases like "all of the above" or "both A and B"—are excluded because their meaning depends on specific option positions. The `ANCHOR` regex (lines 29-34) and `has_positional_anchor` helper (lines 43-44) detect these cases to prevent breaking question semantics during the debiasing process.

### Can I run the script without modifying my quiz files?

Yes, the script provides a **check-only mode** activated by the `--check` flag. This mode analyzes the current distribution of correct answers across all quiz files matched by the `QUIZ_GLOB` pattern and reports statistics without writing any changes to disk, allowing you to assess bias levels before applying fixes.