# Extracting Evaluation Tables from READMEs to YAML Format in Hugging Face Skills

> Convert evaluation tables from Hugging Face model READMEs to YAML format. The hugging-face-evaluation skill uses token matching and auto detection for standardized model-index YAML.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: how-to-guide
- Published: 2026-03-08

---

**The hugging-face-evaluation skill converts markdown evaluation tables from model READMEs into standardized `model-index` YAML blocks using exact token matching and automatic table format detection.**

This guide explains how the `huggingface/skills` repository implements a robust pipeline to parse evaluation tables—commonly found in model cards—and transform them into structured YAML metadata. The solution handles multiple table orientations (benchmarks-as-rows, benchmarks-as-columns, and transposed layouts) while ensuring precise model name matching to avoid data misattribution.

## Pipeline Architecture

The extraction workflow operates through three distinct layers implemented in [`skills/hugging-face-evaluation/scripts/evaluation_manager.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-evaluation/scripts/evaluation_manager.py).

### CLI Orchestration Layer

The `main()` function serves as the entry point, handling argument parsing via **argparse** and loading environment variables through **python-dotenv**. It supports four primary commands: `inspect-tables`, `extract-readme`, `get-prs`, and the internal YAML generation logic. By default, all extraction commands run in dry-run mode, printing YAML to stdout without modifying the remote repository.

### Table Discovery and Parsing Layer

The system downloads README content using `huggingface_hub.ModelCard.load()`, then processes the markdown through **markdown-it-py** (a GitHub-flavored markdown parser). The `extract_tables_with_parser()` function isolates table structures while ignoring code fences and inline markup. Subsequent processing by `detect_table_format()` classifies each table into one of three categories:

- **`rows`** – Benchmarks occupy rows, with each model represented as a column (most common format)
- **`columns`** – Benchmarks occupy columns, with each model represented as a row
- **`transposed`** – Models are rows, benchmarks are columns (requires row-based extraction)

This classification relies on header keyword analysis, numeric density calculations, and column count heuristics.

### Model-Specific Extraction and YAML Generation Layer

Once a table format is identified, the pipeline executes `extract_evaluations_from_readme()`, which normalizes model names by lower-casing, stripping punctuation, and collapsing hyphens/underscores into spaces. The `find_main_model_column()` or `find_main_model_row()` functions perform **exact token set matching** against these normalized names to locate the correct data slice.

The `extract_metrics_from_table()` function then extracts numeric benchmark values, constructs metric dictionaries, and wraps them in the canonical `model-index` structure. Finally, **PyYAML** serializes the data via `yaml.dump()`.

## Key Design Choices

### Exact Token Matching for Model Identification

Rather than using partial string matching or fuzzy logic, the implementation requires the complete tokenized model name to match the column or row header exactly. This prevents accidental extraction of scores from similarly named models in multi-model comparison tables. The normalization process removes markdown markup, converts to lowercase, and treats `-` and `_` as equivalent whitespace before comparison.

### Automatic Format Detection

The `detect_table_format()` heuristic eliminates manual configuration for most README styles. It analyzes header content for benchmark keywords (e.g., "MMLU", "HumanEval"), calculates the ratio of numeric cells per column, and evaluates column naming patterns to determine whether the table uses a row-major or column-major layout for benchmarks.

### Safety-First Workflow

The CLI defaults to read-only operations. Users must explicitly add `--apply` to push changes directly to a repository they own, or `--create-pr` to open a pull request against third-party models. The `get-prs` command checks for existing open PRs before extraction to prevent duplicate submissions.

### Environment Variable Management

Sensitive credentials like `HF_TOKEN` and `AA_API_KEY` load automatically from a `.env` file when present, keeping secrets out of shell history and CI logs.

## Command-Line Usage

### Inspecting README Tables

First, identify which tables contain evaluation data and locate your model's column index:

```bash
uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model"

```

This outputs table numbers, detected formats, and the indices of columns matching the normalized model name.

### Extracting and Previewing YAML

Generate the `model-index` block for review without modifying the repository:

```bash
uv run scripts/evaluation_manager.py extract-readme \
    --repo-id "username/model" \
    --table 1 \
    --model-column-index 3

```

### Applying Changes

Push directly to your own repository:

```bash
uv run scripts/evaluation_manager.py extract-readme \
    --repo-id "username/model" \
    --table 1 \
    --model-column-index 3 \
    --apply

```

Or create a pull request for community models:

```bash
uv run scripts/evaluation_manager.py extract-readme \
    --repo-id "username/model" \
    --table 1 \
    --model-column-index 3 \
    --create-pr

```

The system automatically merges new metrics with existing `model-index` entries rather than overwriting them.

## Programmatic Python API

Import the extraction functions directly for custom workflows:

```python
from evaluation_manager import (
    extract_tables_with_parser,
    detect_table_format,
    extract_metrics_from_table,
)
import yaml

# Load README content

readme_md = """## Evaluation Results

| Model | MMLU | HumanEval |
|-------|------|-----------|
| my-model | 0.85 | 0.92 |
| other-model | 0.80 | 0.88 |
"""

# Extract all tables

tables = extract_tables_with_parser(readme_md)

# Process the first evaluation table

for tbl in tables:
    format_info = detect_table_format(tbl, repo_id="org/my-model")
    if format_info["format"] != "unknown":
        header, rows = tbl["headers"], tbl["rows"]
        metrics = extract_metrics_from_table(
            header,
            rows,
            model_name="my-model",
            model_column_index=None,  # Auto-detect using token matching

        )
        
        # Generate YAML output

        model_index = {
            "model-index": [{
                "name": "my-model",
                "results": [{
                    "task": {"type": "text-generation"},
                    "dataset": {"name": "Benchmarks", "type": "benchmark"},
                    "metrics": metrics,
                    "source": {
                        "name": "Model README",
                        "url": "https://huggingface.co/org/my-model"
                    }
                }]
            }]
        }
        print(yaml.dump(model_index))
        break

```

The script supports PEP 723 inline dependencies, allowing direct execution via `uv run` without separate package installation.

## Summary

- The **huggingface/skills** repository provides a complete CLI tool in [`evaluation_manager.py`](https://github.com/huggingface/skills/blob/main/evaluation_manager.py) for converting README evaluation tables to YAML.
- **Exact token matching** ensures scores are extracted only from the correct model column or row, preventing attribution errors.
- **Automatic format detection** handles benchmarks-as-rows, benchmarks-as-columns, and transposed table layouts without manual configuration.
- The pipeline uses **markdown-it-py** for robust parsing and **PyYAML** for standard-compliant YAML generation.
- Safety features include dry-run defaults, duplicate PR prevention via `get-prs`, and `.env` file support for credentials.

## Frequently Asked Questions

### How does the tool handle different table formats in READMEs?

The `detect_table_format()` function analyzes table headers and cell content to classify structures into three categories: `rows` (benchmarks as rows, models as columns), `columns` (benchmarks as columns, models as rows), or `transposed` (models as rows, benchmarks as columns). It examines keyword presence, numeric density, and column naming patterns to determine the correct extraction strategy automatically.

### What prevents the extractor from grabbing scores from the wrong model?

The implementation uses exact token set matching after normalizing model names. It lower-cases the model identifier, removes punctuation, and treats hyphens and underscores as spaces, then compares the complete token set against table headers. Only columns or rows with identical normalized token sets are selected for extraction, eliminating partial match errors in multi-model comparison tables.

### Can I run this without pushing changes to Hugging Face?

Yes. By default, all extraction commands operate in preview mode. The `extract-readme` command prints the generated YAML to stdout without network modifications unless you explicitly add `--apply` for direct pushes or `--create-pr` for pull requests. You can also use the Python API to process README strings locally without any repository interaction.

### Which dependencies are required to run the extraction pipeline?

The script requires `huggingface_hub` for model card retrieval, `markdown-it-py` for parsing markdown tables, `python-dotenv` for environment variable management, and `PyYAML` for YAML serialization. These dependencies are declared in the PEP 723 header of [`evaluation_manager.py`](https://github.com/huggingface/skills/blob/main/evaluation_manager.py), enabling direct execution with `uv run` without manual installation.