Debiasing Strategies in `debias_quizzes.py` and `debias_certification_questions.py`: Eliminating Answer-Position Bias

The debias_quizzes.py and debias_certification_questions.py scripts in the rohitg00/ai-engineering-from-scratch repository implement deterministic, content-seeded shuffling and file-wide cycle balancing to eliminate systematic answer-position bias in multiple-choice assessments.

These debiasing strategies ensure that learners cannot exploit positional patterns (such as correct answers clustering in option "C") while maintaining semantic integrity for questions that reference specific answer letters. Both scripts guarantee idempotent, reproducible transformations suitable for automated CI pipelines.

Per-Question Randomization in debias_quizzes.py

The scripts/debias_quizzes.py module processes individual quiz files to apply deterministic, question-level randomization. It ensures that identical questions always produce identical shuffles across rebuilds, preventing answer-key drift while distributing correct answers uniformly across positions A through D.

Deterministic Seeding with Content Identity

To guarantee reproducibility, the script generates a unique seed for each question by hashing a composite string of the file path and question text.

# scripts/debias_quizzes.py#L38-L41

def seed_for(path: Path, question_text: str) -> int:
    raw = f"{path}\x00{question_text}".encode()
    return int(hashlib.sha256(raw).hexdigest()[:16], 16)

This content-derived seed ensures that moving a question between files or editing its text generates a new shuffle, while identical content always shuffles to the same order.

Positional-Anchor Detection

Before shuffling, the script checks for positional anchors—phrases like "both A and B" or "all of the above" that would break if options were reordered. The ANCHOR regular expression (scripts/debias_quizzes.py#L29-L35) identifies these dependencies and skips the question to preserve semantic validity.

Duplicate-Option Guards and Canonical Ordering

The debias_question() function (scripts/debias_quizzes.py#L57-L58) validates that no two options contain identical text. Duplicate options would make permutation mapping ambiguous, so these questions are excluded from processing.

For valid questions, the script establishes a canonical base order by sorting options alphabetically:

# scripts/debias_quizzes.py#L61

base = sorted(options, key=str)

This step ensures that the starting arrangement is deterministic regardless of how the options were originally ordered in the JSON file.

Content-Seeded Permutation and Correct-Index Rewrite

Using the content hash as a seed, the script creates a random.Random instance and shuffles the canonical base list:

# scripts/debias_quizzes.py#L63

random.Random(seed_for(path, question_text)).shuffle(perm)

After shuffling, the script locates the new position of the original correct answer and updates the correct field accordingly:

# scripts/debias_quizzes.py#L65

new_correct = new_options.index(correct_val)

The function returns a boolean indicating whether changes occurred, enabling idempotent behavior—re-running the script produces no further modifications if the file is already debiased (scripts/debias_quizzes.py#L67-L72).

File-Wide Balancing in debias_certification_questions.py

While the quiz script randomizes per-question, scripts/debias_certification_questions.py implements file-wide distribution balancing to ensure that correct answers follow a uniform cyclic pattern across an entire certification lesson or assessment.

Target Cycle Generation and Deterministic Offset

The target_cycles() function (scripts/debias_certification_questions.py#L61-L68) pre-computes admissible position cycles based on option count and number of correct answers. For single-answer questions with four options, the cycles are (0,), (1,), (2,), and (3,).

The script selects a deterministic cycle offset per file using a hash of the relative path and a descriptor string:

# scripts/debias_certification_questions.py#L103-L104

offset = seed_for(f"{rel_path}\x00answer-cycle:{optionCount}:{correctCount}") % len(cycles)

This ensures that different certification files use different starting positions in the cycle, preventing systematic bias at the curriculum level.

Group-Wise Indexing for Distribution Balance

The script tracks how many questions of each shape (defined by optionCount × correctCount) have been processed using group_seen[group] counters (scripts/debias_certification_questions.py#L50-L53). As it iterates through questions, it advances through the pre-computed cycle, assigning the next position in the sequence to each new question.

This group-wise indexing guarantees that across the entire file, each possible correct position appears roughly the same number of times, neutralizing positional guessing strategies.

Stable Ordering and Safety Guards

Before applying the cycle, the script invokes stable_order() (scripts/debias_certification_questions.py#L71-L74) to sort correct and incorrect option strings deterministically (using JSON serialization) and then shuffle them with a question-specific seed. This prevents option text ordering from influencing the final arrangement.

Like the quiz script, it employs a POSITIONAL_ANCHOR regex (scripts/debias_certification_questions.py#L31-L35) to skip questions containing phrases like "both A and B", and it guards against duplicate options in rewrite_question() (scripts/debias_certification_questions.py#L85-L88).

After constructing the new options list according to the selected cycle, the script writes either a single integer or list of integers back to the correct field (scripts/debias_certification_questions.py#L14-L19).

Shared Architectural Principles

Both debiasing scripts follow consistent design patterns that ensure reliability in production environments:

• Deterministic seeding – SHA-256 hashes of file paths and content tokens generate reproducible pseudo-random numbers
• Idempotence – Re-running the scripts yields no changes if the files are already debiased, preventing unnecessary CI rebuilds
• Safety guards – Automatic detection of positional anchors and duplicate options prevents semantic corruption
• CLI validation – Both support a --check flag for dry-run reporting without file modification

Running the Debiasing Scripts

Execute these commands from the repository root to apply the debiasing strategies:

# Process all lesson quizzes with per-question randomization

python3 scripts/debias_quizzes.py

# Verify quiz distributions without modifying files (CI-friendly)

python3 scripts/debias_quizzes.py --check

# Balance certification question positions across entire files

python3 scripts/debias_certification_questions.py

# Check certification balance without writing changes

python3 scripts/debias_certification_questions.py --check

These scripts consume data from phases/*/*/quiz.json (quizzes) and certifications/claude/lessons/*/quiz.json or certifications/claude/assessments/*/*.json (certification content).

Summary

• debias_quizzes.py applies deterministic, content-seeded shuffling to individual questions, using SHA-256 hashing and canonical sorting to ensure reproducible randomization while preserving questions containing positional anchors.
• debias_certification_questions.py implements file-wide cycle balancing, tracking question groups by shape and distributing correct answers uniformly across possible positions using pre-computed target cycles.
• Both scripts skip questions where reordering would break semantics (positional anchors) or where duplicate options would create ambiguous mappings.
• The --check flag enables CI integration for validation without modification.
• Idempotent design ensures that repeated executions remain stable and do not introduce drift.

Frequently Asked Questions

What is answer-position bias and why does it matter?

Answer-position bias occurs when correct answers in multiple-choice questions cluster in specific positions (typically "C" or "B"), allowing students to exploit pattern recognition rather than knowledge. In educational datasets, this bias can artificially inflate assessment scores and reduce the validity of learning evaluations. The debiasing strategies in these scripts eliminate these patterns while maintaining content integrity.

How do the scripts handle questions with "all of the above" options?

Both scripts use regular expression anchors (ANCHOR in debias_quizzes.py and POSITIONAL_ANCHOR in debias_certification_questions.py) to detect phrases like "both A and B", "none of the above", or "all of the above". When detected, these questions are excluded from shuffling because reordering the options would change the semantic meaning of the question text.

Are the debiasing changes reversible or deterministic?

The changes are deterministic but not reversible. Because the shuffle seed is derived from a SHA-256 hash of the file path and question content, the same question always produces the same option order. However, the original order is not preserved, so you cannot "unshuffle" a question without maintaining separate backup files. The deterministic nature ensures that rebuilds produce consistent, stable output.

Can these scripts be integrated into CI/CD pipelines?

Yes. Both scripts support a --check flag that performs a dry run and reports deviations without modifying files, making them ideal for pre-commit hooks or CI validation steps. The idempotent design ensures that running the scripts multiple times produces no side effects after the initial debiasing, allowing safe integration into automated build processes for the ai-engineering-from-scratch curriculum.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}