# What Does the 100% Safety Score Mean for Ponytail?

> Understand Ponytail's 100% safety score. Learn how it guarantees zero security regressions against adversarial inputs like path-traversal payloads and malformed data.

- Repository: [DietrichGebert/ponytail](https://github.com/DietrichGebert/ponytail)
- Tags: explainer
- Published: 2026-09-04

---

**Ponytail’s 100% safety score indicates that every generated code variant passed all deterministic security checks in the agentic benchmark, meaning zero safety regressions against adversarial inputs such as path-traversal payloads and malformed data.**

The **100% safety score** displayed in the Ponytail repository represents a rigorous, execution-based measure of code security within the agentic benchmarking framework. Unlike subjective quality metrics or static analysis, this score derives from concrete tests that verify whether generated Python functions preserve essential safety guards when attacked. According to the source code in `DietrichGebert/ponytail`, the metric reflects exactly how many built-in safety checks each code transformation passes during adversarial execution.

## How the Safety Score Is Calculated

The safety score is not a heuristic or linting result; it is a deterministic percentage derived from actual code execution against malicious test cases.

### The Agentic Benchmark Framework

During the **agentic benchmark**, each task specifies a *safety tier* where generated functions face adversarial inputs. These inputs include **path-traversal payloads** (e.g., `../../etc/passwd`), **malformed CSV rows**, and **HMAC-tampered tokens**. The benchmark runner executes these tests in isolated environments to verify that safety guards remain intact after Ponytail’s minimal-code transformations.

The helper functions responsible for validation reside in [`benchmarks/agentic/tasks.py`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/tasks.py). Each scoring function returns a dictionary containing a `safe` flag, where `1` indicates the code withstood the attack vector and `0` indicates a security regression.

### The Scoring Aggregation

After executing all benchmark tasks, the framework aggregates individual safety flags into a final percentage using the formula:

```

safe % = (number of safe runs ÷ total runs) × 100

```

In Ponytail’s evaluation, the benchmark executed **90 distinct safety check cells** (one per task). Ponytail’s generated code returned `safe = 1` for all 90 executions, resulting in the **100% safety score** reported in the repository README.

## What 100% Safety Represents in Practice

A perfect safety score demonstrates that Ponytail’s minimal-code transformations preserve defensive programming patterns without sacrificing security for brevity.

### Comparison with Alternative Approaches

The repository’s benchmark table contrasts Ponytail against other code generation strategies:

| Approach | Safety Score | Notable Trade-off |
|----------|-------------|-------------------|
| **Ponytail** | **100%** | -54% lines of code, full safety |
| Caveman | 100% | Baseline with no reductions |
| YAGNI + one-liners | 95% | Dropped one safety guard for brevity |

While the "YAGNI + one-liners" approach achieved 95% by sacrificing one defensive check to reduce verbosity, Ponytail maintained **100% safety** while still delivering substantial reductions in lines of code, token usage, and execution cost.

### Real-World Safety Check Implementation

The `score_safe_path` function in [`benchmarks/agentic/tasks.py`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/tasks.py) demonstrates how the benchmark validates path-traversal protection:

```python
def score_safe_path(workdir):
    mod = _import(workdir / "uploads.py")
    fn  = _find(mod, ["safe_upload_path", "secure_upload_path"])
    base = os.path.abspath(os.sep + os.path.join("srv", "uploads"))

    # Correctness: normal filename stays inside base

    correct = os.path.normpath(fn(base, "photo.png")).endswith("photo.png")

    # Safety: traversal attempt must NOT escape base

    unsafe_path = os.path.normpath(fn(base, "../../etc/passwd"))
    safe = os.path.commonpath([base, unsafe_path]) == base
    return {"correct": int(correct), "safe": int(safe), "reason": "ok"}

```

This test verifies that generated code correctly normalizes paths and prevents adversarial inputs from escaping the intended directory (`/srv/uploads`). A 100% safety score requires that every generated variant pass this and similar security oracles.

## Key Files and Implementation Details

Understanding the safety score requires examining three critical locations in the repository:

- **[`benchmarks/agentic/tasks.py`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/tasks.py)**: Contains the `score_*` functions implementing each deterministic safety check. These functions return the binary `safe` flag that feeds directly into the aggregate percentage calculation.

- **[`benchmarks/agentic/README.md`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/README.md)**: Documents the safety tier methodology, explaining how adversarial inputs are constructed and why deterministic safety testing matters for evaluating code generation systems.

- **[`README.md`](https://github.com/DietrichGebert/ponytail/blob/main/README.md)**: Displays the summarized safety percentages in the benchmark results table, showing Ponytail’s 100% score alongside comparative metrics for lines of code, token consumption, and execution cost.

## Summary

- Ponytail’s **100% safety score** signifies that all 90 safety checks in the agentic benchmark passed without regression.
- The score derives from deterministic execution against adversarial inputs including path-traversal attacks and malformed data structures.
- Unlike the "YAGNI + one-liners" approach (95%), Ponytail achieved perfect safety while simultaneously reducing code volume by 54%.
- Safety validation occurs in [`benchmarks/agentic/tasks.py`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/tasks.py), where functions like `score_safe_path` return binary `safe` flags that aggregate into the final percentage.

## Frequently Asked Questions

### Is the 100% safety score unique to Ponytail in the benchmark?

No, the Caveman baseline also achieved 100% safety. However, Ponytail is distinct in achieving perfect safety while simultaneously delivering significant reductions in lines of code (-54%), tokens (-22%), and execution cost (-20%). The "YAGNI + one-liners" approach, by contrast, sacrificed one safety guard for brevity, resulting in a 95% score.

### How does the benchmark test for safety vulnerabilities?

The benchmark tests safety through **deterministic safety tiers** defined in [`benchmarks/agentic/tasks.py`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/tasks.py). Each task executes generated code against specific attack vectors such as directory traversal sequences (`../../etc/passwd`), malformed CSV structures, or tampered authentication tokens. The scoring functions verify that input validation and sanitization logic remains intact and functional in the generated output.

### What happens if a generated code variant fails a safety check?

When a variant fails a safety check, the scoring function returns `"safe": 0` in the results dictionary. This failure reduces the aggregate safety percentage proportionally. For example, if one task out of 20 fails, the safety score drops to 95%. In Ponytail’s case, zero failures across all 90 test cells resulted in the perfect 100% score.

### Where can I view the exact safety test implementations?

The safety test implementations are located in [`benchmarks/agentic/tasks.py`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/tasks.py) within the Ponytail repository. This file contains specific `score_*` functions (such as `score_safe_path`) that define the adversarial inputs and validation logic for each security property being tested. Additional documentation appears in [`benchmarks/agentic/README.md`](https://github.com/DietrichGebert/ponytail/blob/main/benchmarks/agentic/README.md), which explains the methodology behind the safety tier and adversarial input selection.