Why Does the Adversarial Safety Tier Show a Lower Prompt Score in Ponytail?

The adversarial safety tier intentionally lowers the prompt score when generated code fails runtime safety checks, prioritizing defensive programming and security robustness over superficial task completion metrics.

The DietrichGebert/ponytail repository implements a rigorous evaluation framework that distinguishes between functional correctness and runtime safety. When reviewing benchmark results, developers often observe that models producing seemingly correct code receive lower overall scores than expected. This discrepancy stems from how the adversarial safety tier penalizes unsafe behavior during execution against hostile inputs.

Understanding the Two Evaluation Dimensions

Prompt Score: Measuring Task Completion

The standard prompt score evaluates a model's output based solely on static task fulfillment. According to benchmarks/agentic/README.md, this metric measures code completeness, syntactic correctness, and brevity without executing the generated code against adversarial inputs. A model receives high marks for producing code that superficially solves the problem, regardless of whether it contains security vulnerabilities or runtime instability when facing malicious data.

Adversarial Safety Tier: Runtime Validation

In contrast, the adversarial safety tier executes generated code against crafted adversarial inputs designed to trigger edge cases and security flaws. As documented in the repository's benchmark suite, this tier verifies that code does not crash, leak sensitive data, or behave incorrectly when facing type confusion attacks or injection attempts. The evaluation logic immediately deducts points when code fails validation, throws unhandled exceptions, or demonstrates unsafe type coercion during execution.

Why Safety Constraints Reduce the Prompt Score

The adversarial safety tier produces lower prompt scores because it applies stricter, multi-dimensional criteria than standard benchmarks. When a model generates defensive code with comprehensive input validation, error handling, and type checking, it often sacrifices the concise implementations that maximize prompt scores.

According to the results documented in benchmarks/results/2026-06-18-agentic.md, the benchmark explicitly favors safety over brevity. A "ponytail (safe)" variant achieves 100% safety survival against adversarial inputs but displays a lower numerical prompt score compared to "YAGNI + one-liners" approaches that achieve higher prompt scores yet only 95% safety ratings due to missing validation logic.

Code Examples: Safe vs. Unsafe Implementations

The following JavaScript examples from the Ponytail test suite illustrate how validation affects safety tier scoring.

Safe implementation that passes adversarial probes:

// Example of a safe function – passes the adversarial tier
export function add(a, b) {
  // Validate inputs to prevent NaN or injection attacks
  if (typeof a !== 'number' || typeof b !== 'number') {
    throw new TypeError('Both arguments must be numbers');
  }
  return a + b;
}

Unsafe implementation that fails safety checks:

// Example that would fail the safety tier
export function add(a, b) {
  // No validation – unsafe if a or b is malicious
  return a + b;
}

The first version maintains a 100% safety score by explicitly validating arguments before operations, while the second version risks NaN propagation or type coercion attacks, triggering deductions in the adversarial safety tier despite producing superficially correct arithmetic results.

How the Benchmark Evaluates Safety

The benchmarks/agentic/README.md file details the evaluation process that creates this scoring differential. The pipeline first generates code using the target model, then subjects that code to a battery of adversarial probes designed to trigger edge cases, type confusion, and injection vectors.

Unlike static analysis, this runtime verification simulates production conditions where hostile inputs are common. The README.md in the repository root clarifies that this approach intentionally separates "code that works" from "code that survives," ensuring that only robust, production-ready implementations achieve top-tier safety scores.

Summary

• The adversarial safety tier executes code against malicious inputs, while the prompt score measures only static task completion and brevity.
• Safety tier deductions occur when code crashes, leaks data, or mishandles adversarial probes, regardless of functional correctness on standard inputs.
• Defensive programming patterns—input validation, type checking, and error handling—reduce prompt scores but increase safety ratings in the Ponytail benchmark.
• The benchmarks/results/2026-06-18-agentic.md file documents concrete comparisons showing safe variants with lower prompt scores but perfect safety survival rates.

Frequently Asked Questions

What is the adversarial safety tier in Ponytail?

The adversarial safety tier is a runtime evaluation mechanism in the DietrichGebert/ponytail benchmark that executes generated code against crafted hostile inputs. It verifies that implementations remain stable and secure when facing malformed data, type confusion attacks, and injection attempts that would compromise production systems.

Why does defensive coding lower the prompt score?

Defensive coding introduces validation logic, error handling, and type checking that increases code verbosity and line count. Since the prompt score emphasizes brevity and direct task completion metrics, these safety-oriented additions reduce superficial efficiency scores while significantly increasing actual runtime reliability against adversarial inputs.

How does Ponytail's benchmark detect unsafe code?

According to benchmarks/agentic/README.md, the benchmark uses automated adversarial probes—systematic tests with malicious inputs designed to trigger vulnerabilities. If code throws unhandled exceptions, performs unsafe type coercion, or leaks data during execution, the safety tier records failures and adjusts the final score downward.

Where can I view specific benchmark comparisons between safe and unsafe variants?

Detailed scoring comparisons appear in benchmarks/results/2026-06-18-agentic.md, which documents side-by-side results showing how "ponytail (safe)" variants achieve perfect 100% safety scores with lower prompt scores compared to unsafe alternatives that maximize task completion metrics while failing security probes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}