Extracting Evaluation Tables from READMEs to YAML Format in Hugging Face Skills

The hugging-face-evaluation skill converts markdown evaluation tables from model READMEs into standardized model-index YAML blocks using exact token matching and automatic table format detection.

This guide explains how the huggingface/skills repository implements a robust pipeline to parse evaluation tables—commonly found in model cards—and transform them into structured YAML metadata. The solution handles multiple table orientations (benchmarks-as-rows, benchmarks-as-columns, and transposed layouts) while ensuring precise model name matching to avoid data misattribution.

Pipeline Architecture

The extraction workflow operates through three distinct layers implemented in skills/hugging-face-evaluation/scripts/evaluation_manager.py.

CLI Orchestration Layer

The main() function serves as the entry point, handling argument parsing via argparse and loading environment variables through python-dotenv. It supports four primary commands: inspect-tables, extract-readme, get-prs, and the internal YAML generation logic. By default, all extraction commands run in dry-run mode, printing YAML to stdout without modifying the remote repository.

Table Discovery and Parsing Layer

The system downloads README content using huggingface_hub.ModelCard.load(), then processes the markdown through markdown-it-py (a GitHub-flavored markdown parser). The extract_tables_with_parser() function isolates table structures while ignoring code fences and inline markup. Subsequent processing by detect_table_format() classifies each table into one of three categories:

• rows – Benchmarks occupy rows, with each model represented as a column (most common format)
• columns – Benchmarks occupy columns, with each model represented as a row
• transposed – Models are rows, benchmarks are columns (requires row-based extraction)

This classification relies on header keyword analysis, numeric density calculations, and column count heuristics.

Model-Specific Extraction and YAML Generation Layer

Once a table format is identified, the pipeline executes extract_evaluations_from_readme(), which normalizes model names by lower-casing, stripping punctuation, and collapsing hyphens/underscores into spaces. The find_main_model_column() or find_main_model_row() functions perform exact token set matching against these normalized names to locate the correct data slice.

The extract_metrics_from_table() function then extracts numeric benchmark values, constructs metric dictionaries, and wraps them in the canonical model-index structure. Finally, PyYAML serializes the data via yaml.dump().

Key Design Choices

Exact Token Matching for Model Identification

Rather than using partial string matching or fuzzy logic, the implementation requires the complete tokenized model name to match the column or row header exactly. This prevents accidental extraction of scores from similarly named models in multi-model comparison tables. The normalization process removes markdown markup, converts to lowercase, and treats - and _ as equivalent whitespace before comparison.

Automatic Format Detection

The detect_table_format() heuristic eliminates manual configuration for most README styles. It analyzes header content for benchmark keywords (e.g., "MMLU", "HumanEval"), calculates the ratio of numeric cells per column, and evaluates column naming patterns to determine whether the table uses a row-major or column-major layout for benchmarks.

Safety-First Workflow

The CLI defaults to read-only operations. Users must explicitly add --apply to push changes directly to a repository they own, or --create-pr to open a pull request against third-party models. The get-prs command checks for existing open PRs before extraction to prevent duplicate submissions.

Environment Variable Management

Sensitive credentials like HF_TOKEN and AA_API_KEY load automatically from a .env file when present, keeping secrets out of shell history and CI logs.

Command-Line Usage

Inspecting README Tables

First, identify which tables contain evaluation data and locate your model's column index:

uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model"

This outputs table numbers, detected formats, and the indices of columns matching the normalized model name.

Extracting and Previewing YAML

Generate the model-index block for review without modifying the repository:

uv run scripts/evaluation_manager.py extract-readme \
    --repo-id "username/model" \
    --table 1 \
    --model-column-index 3

Applying Changes

Push directly to your own repository:

uv run scripts/evaluation_manager.py extract-readme \
    --repo-id "username/model" \
    --table 1 \
    --model-column-index 3 \
    --apply

Or create a pull request for community models:

uv run scripts/evaluation_manager.py extract-readme \
    --repo-id "username/model" \
    --table 1 \
    --model-column-index 3 \
    --create-pr

The system automatically merges new metrics with existing model-index entries rather than overwriting them.

Programmatic Python API

Import the extraction functions directly for custom workflows:

from evaluation_manager import (
    extract_tables_with_parser,
    detect_table_format,
    extract_metrics_from_table,
)
import yaml

# Load README content

readme_md = """## Evaluation Results

| Model | MMLU | HumanEval |
|-------|------|-----------|
| my-model | 0.85 | 0.92 |
| other-model | 0.80 | 0.88 |
"""

# Extract all tables

tables = extract_tables_with_parser(readme_md)

# Process the first evaluation table

for tbl in tables:
    format_info = detect_table_format(tbl, repo_id="org/my-model")
    if format_info["format"] != "unknown":
        header, rows = tbl["headers"], tbl["rows"]
        metrics = extract_metrics_from_table(
            header,
            rows,
            model_name="my-model",
            model_column_index=None,  # Auto-detect using token matching

        )
        
        # Generate YAML output

        model_index = {
            "model-index": [{
                "name": "my-model",
                "results": [{
                    "task": {"type": "text-generation"},
                    "dataset": {"name": "Benchmarks", "type": "benchmark"},
                    "metrics": metrics,
                    "source": {
                        "name": "Model README",
                        "url": "https://huggingface.co/org/my-model"
                    }
                }]
            }]
        }
        print(yaml.dump(model_index))
        break

The script supports PEP 723 inline dependencies, allowing direct execution via uv run without separate package installation.

Summary

• The huggingface/skills repository provides a complete CLI tool in evaluation_manager.py for converting README evaluation tables to YAML.
• Exact token matching ensures scores are extracted only from the correct model column or row, preventing attribution errors.
• Automatic format detection handles benchmarks-as-rows, benchmarks-as-columns, and transposed table layouts without manual configuration.
• The pipeline uses markdown-it-py for robust parsing and PyYAML for standard-compliant YAML generation.
• Safety features include dry-run defaults, duplicate PR prevention via get-prs, and .env file support for credentials.

Frequently Asked Questions

How does the tool handle different table formats in READMEs?

The detect_table_format() function analyzes table headers and cell content to classify structures into three categories: rows (benchmarks as rows, models as columns), columns (benchmarks as columns, models as rows), or transposed (models as rows, benchmarks as columns). It examines keyword presence, numeric density, and column naming patterns to determine the correct extraction strategy automatically.

What prevents the extractor from grabbing scores from the wrong model?

The implementation uses exact token set matching after normalizing model names. It lower-cases the model identifier, removes punctuation, and treats hyphens and underscores as spaces, then compares the complete token set against table headers. Only columns or rows with identical normalized token sets are selected for extraction, eliminating partial match errors in multi-model comparison tables.

Can I run this without pushing changes to Hugging Face?

Yes. By default, all extraction commands operate in preview mode. The extract-readme command prints the generated YAML to stdout without network modifications unless you explicitly add --apply for direct pushes or --create-pr for pull requests. You can also use the Python API to process README strings locally without any repository interaction.

Which dependencies are required to run the extraction pipeline?

The script requires huggingface_hub for model card retrieval, markdown-it-py for parsing markdown tables, python-dotenv for environment variable management, and PyYAML for YAML serialization. These dependencies are declared in the PEP 723 header of evaluation_manager.py, enabling direct execution with uv run without manual installation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}