Adding Evaluation Results to Model Cards Using Model-Index YAML: A Complete Guide

You can automatically populate Hugging Face model cards with structured benchmark scores by extracting markdown tables from READMEs or importing Artificial Analysis API data into the model-index YAML section using the evaluation_manager.py CLI.

The hugging-face-evaluation skill in the huggingface/skills repository automates the tedious process of formatting benchmark results according to the Papers with Code specification. This tool parses evaluation tables, detects their layout structure, and generates the required model-index metadata that powers the Hugging Face evaluation widget.

How the Evaluation Skill Works

The skill is implemented as a standalone Python CLI in skills/hugging-face-evaluation/scripts/evaluation_manager.py. It uses markdown-it-py for robust table parsing and the huggingface_hub library to interact with model repositories.

Core Components

• Table Extraction: The extract_tables_with_parser() function pulls raw markdown tables from a model's README using markdown-it-py, handling GFM tables, code fences, and links accurately.
• Layout Classification: detect_table_format(), is_transposed_table(), find_main_model_column(), and find_main_model_row() identify whether data is organized by rows, columns, or transposed, then locate the target model via exact-token matching in normalize_model_name.
• Metric Extraction: extract_metrics_from_table() converts parsed tables into metric dictionaries, supporting optional --model-column-index overrides for non-standard layouts.
• Model-Index Generation: extract_evaluations_from_readme() wraps metrics in the required YAML structure with task, dataset, metrics, and source fields.
• Hub Integration: update_model_card_with_evaluations() merges new results with existing entries (lines 10-21 of the function preserve earlier data) and pushes updates via ModelCard.push_to_hub().

Extracting Evaluation Results from README Tables

Before modifying any model card, inspect the tables to verify detection accuracy.

Inspecting Tables Before Extraction

Run the inspect-tables command to see how the parser interprets your README:

uv run scripts/evaluation_manager.py inspect-tables \
  --repo-id "meta-llama/Llama-3.3-70B-Instruct"

This outputs each table's format, column indices, and which columns match the model name using the exact-token matching logic in find_main_model_column().

Dry-Run vs. Apply

Always perform a dry run first to validate the generated YAML structure:

uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "your-username/your-model-7b" \
  --dry-run

The generated model-index YAML prints to stdout without modifying the repository. Once validated, apply the changes directly:

uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "your-username/your-model-7b" \
  --apply

Creating Pull Requests for Community Models

For repositories where you lack direct write access, use the --create-pr flag:

uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "community/awesome-7b" \
  --create-pr

The script checks for open PRs via list_open_prs() before creating a new one with your evaluation updates.

Importing from Artificial Analysis API

You can also import benchmarks from the Artificial Analysis API instead of parsing README tables. The import_aa_evaluations() function calls get_aa_model_data() and transforms the JSON response using aa_data_to_model_index():

AA_API_KEY="my-aa-key" uv run scripts/evaluation_manager.py import-aa \
  --creator-slug "anthropic" \
  --model-name "claude-sonnet-4" \
  --repo-id "your-username/claude-mirror" \
  --create-pr

This bypasses table extraction entirely, converting API responses directly into the model-index format.

Updating Model Cards Programmatically

For CI/CD pipelines or custom workflows, invoke the CLI via Python subprocess:

from pathlib import Path
import subprocess, json, os

repo = "username/model"

# Generate model-index YAML

proc = subprocess.run(
    ["uv", "run", "scripts/evaluation_manager.py", "extract-readme",
     "--repo-id", repo, "--dry-run"],
    capture_output=True, text=True, check=True
)

yaml_output = proc.stdout
print("Generated model-index:\n", yaml_output)

# Push changes to hub

subprocess.run(
    ["uv", "run", "scripts/evaluation_manager.py", "extract-readme",
     "--repo-id", repo, "--apply"],
    check=True
)

The script header includes a /// script block that automatically installs dependencies (huggingface_hub, markdown-it-py, pyyaml, requests, python-dotenv), so no manual environment setup is required when using uv run.

Summary

• The evaluation_manager.py CLI in huggingface/skills automates adding evaluation results to model cards using the model-index YAML format.
• It supports two input sources: README markdown tables (parsed via markdown-it-py) and the Artificial Analysis API.
• Table layouts are automatically classified as rows, columns, or transposed, with exact-token matching locating the correct model entry.
• The merge logic in update_model_card_with_evaluations() preserves existing evaluations while appending new results.
• All dependencies are handled automatically via the script header when using uv run, enabling immediate execution without setup.

Frequently Asked Questions

What is the model-index format?

The model-index format is a YAML structure following the Papers with Code specification. It contains a name field and a results array where each entry specifies the task, dataset, metrics (with type and value), and source. The Hugging Face Hub reads this section to render the evaluation widget on model pages.

How does the skill handle existing evaluations?

The skill never overwrites existing model-index entries. In update_model_card_with_evaluations() (specifically lines 10-21 of the implementation), new results are merged with existing data, ensuring that previously recorded benchmarks remain intact while additional scores are appended.

Can I use this without installing dependencies manually?

Yes. The script uses a /// script header that declares required packages (huggingface_hub, markdown-it-py, pyyaml, requests, python-dotenv). When you run the script with uv run, these dependencies are installed automatically in an isolated environment without polluting your system Python.

How does table layout detection work?

The skill uses detect_table_format() to classify tables as row-based, column-based, or transposed. It then applies find_main_model_column() or find_main_model_row() with normalize_model_name() to perform exact-token matching against the model identifier. You can override automatic detection using the --model-column-index parameter if the parser selects the wrong column.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}