Quiz Debiasing Techniques in AI Engineering: How `debias_quizzes.py` Eliminates Positional Bias

The debias_quizzes.py script removes systematic bias in multiple-choice quiz answer positions through deterministic, content-seeded permutation that shuffles options while preserving semantic anchors and correct answer alignment.

The rohitg00/ai-engineering-from-scratch repository implements sophisticated quiz debiasing techniques to ensure fair assessment distribution across its learning phases. This Python script processes JSON quiz files throughout the repository, eliminating patterns where correct answers cluster in specific positions—such as always appearing as option "C" or "D"—that could give learners unfair predictive advantages.

Detecting Positional Anchors to Preserve Semantic Meaning

Before shuffling, the script identifies questions containing positional anchors—language that binds meaning to specific option locations. In scripts/debias_quizzes.py, the regular expression ANCHOR (lines 29-34) scans for phrases like "all of the above" or "both A and B" that would become nonsensical if moved.

The helper function has_positional_anchor (lines 43-44) flags these questions for exclusion. This safety mechanism ensures that semantic relationships between options remain intact, preventing the debiasing process from corrupting question logic while still processing safe questions.

Content-Seeded Randomness for Idempotent Shuffling

For questions without anchors, the script derives a deterministic seed from the content itself rather than using random entropy. The seed_for function (lines 38-40) concatenates the file path with the question text, generating a SHA-256 hash truncated to 16 hexadecimal characters and cast to an integer.

This approach guarantees idempotent behavior—running the script multiple times produces identical shuffle orders for the same question. The deterministic seeding ensures reproducibility across different environments while maintaining statistical randomness across the broader question set.

Deterministic Permutation Algorithm

The core shuffling logic in debias_quizzes.py executes a controlled permutation through three distinct steps. First, options undergo canonical sorting (base = sorted(options, key=str) on line 62) to eliminate any existing positional bias from the source material. Next, a random.Random instance uses the content-derived seed to shuffle a list of indices (perm on line 63). Finally, these shuffled indices reorder the base list to produce new_options (line 64).

This method ensures that the final arrangement depends solely on the question content, not arbitrary randomness or original file ordering, creating a reproducible yet unbiased distribution.

Correct Index Realignment

After permutation, the script must update the correct answer pointer to maintain data integrity. The original correct value is preserved, then its new position is located within the shuffled array:

new_correct = new_options.index(correct_val)  # Line 65

The updated options array and corrected index are written back to the question object (lines 70-71). This maintains semantic correctness while randomizing presentation order, ensuring the right answer stays correct regardless of its new position.

Preserving Original File Formatting

When writing modifications, the serialize function (lines 75-104) respects the original JSON structure. The script detects whether the "options" array was stored as inline or multiline JSON, preserving whitespace and indentation styles exactly.

This formatting awareness minimizes git diffs, making reviews easier and maintaining human-readable quiz files. The debiasing process becomes invisible to content creators while fixing underlying statistical bias in the answer distribution.

Usage and Reporting Modes

The script supports two operational modes controlled via command-line arguments.

To rewrite quizzes in place with debiased positions:

python3 scripts/debias_quizzes.py

To analyze current bias distribution without modifying files:

python3 scripts/debias_quizzes.py --check

The main function (lines 17-70) aggregates statistics on correct-answer distribution across all processed files, reporting how many questions were changed and the uniformity of the new distribution.

Programmatic usage allows integration into larger assessment pipelines:

from scripts.debias_quizzes import debias_question

q = {
    "question": "What is the capital of France?",
    "options": ["Paris", "Berlin", "Madrid", "Rome"],
    "correct": 0,
}

changed = debias_question("phases/01-example/quiz.json", q)
print("Options shuffled?" , changed)
print(q["options"], q["correct"])

Summary

• Positional anchor detection identifies questions with ordering-dependent language using the ANCHOR regex (lines 29-34) and has_positional_anchor helper (lines 43-44), excluding them from shuffling.
• Content-derived seeding via seed_for creates deterministic, reproducible randomness using SHA-256 hashes of file paths and question text.
• Canonical sorting and indexed permutation eliminate existing bias while maintaining content-consistent shuffling through the seeded random.Random implementation.
• Index realignment preserves correct answer semantics by locating the shuffled position of the original correct value and updating the reference.
• Formatting preservation through the serialize function (lines 75-104) ensures minimal diffs and human-readable JSON output.
• Dual execution modes support both destructive debiasing and read-only analysis via the --check flag.

Frequently Asked Questions

What are quiz debiasing techniques?

Quiz debiasing techniques are algorithmic methods used to eliminate systematic patterns in multiple-choice questions where correct answers cluster in specific positions (such as always being option "C" or "D"). In rohitg00/ai-engineering-from-scratch, these techniques include content-seeded randomization, positional anchor detection, and deterministic permutation to ensure fair, unpredictable answer distributions across assessment banks.

How does debias_quizzes.py ensure the same shuffle order every time?

The script uses content-seeded randomness through the seed_for function (lines 38-40), which generates a SHA-256 hash from the combination of the file path and question text. This hash seeds Python's random.Random generator, guaranteeing that identical questions always receive identical shuffle orders (idempotent behavior) while maintaining randomness across different questions.

Why are some questions excluded from shuffling?

Questions containing positional anchors—phrases like "all of the above" or "both A and B"—are excluded because their meaning depends on specific option positions. The ANCHOR regex (lines 29-34) and has_positional_anchor helper (lines 43-44) detect these cases to prevent breaking question semantics during the debiasing process.

Can I run the script without modifying my quiz files?

Yes, the script provides a check-only mode activated by the --check flag. This mode analyzes the current distribution of correct answers across all quiz files matched by the QUIZ_GLOB pattern and reports statistics without writing any changes to disk, allowing you to assess bias levels before applying fixes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}