Importing Benchmark Scores from Artificial Analysis API to Hugging Face Model Cards

The hugging-face-evaluation skill automates the process of fetching benchmark results from the Artificial Analysis API and embedding them into your model's model-index.yml metadata, enabling automatic updates to Hugging Face model card leaderboards.

The huggingface/skills repository includes a dedicated evaluation skill that streamlines importing benchmark scores from Artificial Analysis into model cards. This workflow eliminates manual data entry by programmatically retrieving performance metrics, transforming them into the Hugging Face model-index format, and optionally opening pull requests to update your repository.

How the Automated Import Works

The import workflow orchestrates data flow from the Artificial Analysis API to your Hugging Face model repository through five distinct stages. All logic resides in skills/hugging-face-evaluation/scripts/evaluation_manager.py.

1. Authentication Configuration

The system reads your Artificial Analysis API key from the AA_API_KEY environment variable. As documented in examples/.env.example, you must set this variable before running any import commands. The script passes this key in the Authorization header when making HTTP requests to the Artificial Analysis endpoint.

2. Data Retrieval from Artificial Analysis

The get_aa_model_data(creator_slug, model_name) function constructs the API endpoint https://api.artificialanalysis.ai/v1/models/{creator_slug}/{model_name} and uses the requests library to fetch the JSON payload containing benchmark scores. This function, located at lines 853-862 of evaluation_manager.py, handles network errors, 404 responses, and non-200 status codes by returning None and logging clear error messages.

3. Schema Transformation

The aa_data_to_model_index() function (lines 893-914) converts the raw Artificial Analysis JSON into the Hugging Face model-index schema. It generates a structured dictionary containing name, description, datasets, metrics, and results fields. The function specifically sets the dataset_name to "Artificial Analysis Benchmarks" and injects a source entry pointing back to the Artificial Analysis website for traceability.

4. Repository Update and PR Creation

The import_aa_evaluations() function (lines 943-966) loads any existing model-index.yml from your target repository via huggingface_hub, merges the new evaluation entry while preserving existing data, and writes the updated file to a temporary branch. When the --create-pr flag is used, the function automatically opens a pull request with a descriptive commit message such as "Add Artificial Analysis evaluations for [model-name]".

5. Duplicate Prevention

Before creating any pull request, the skill executes the get-prs sub-command to verify that no open PR already contains the same evaluation. This safety check, emphasized in the SKILL documentation, prevents duplicate efforts and repository clutter.

Core Implementation Functions

Three primary functions in evaluation_manager.py handle the end-to-end integration:

• get_aa_model_data (lines 853-862): Calls the Artificial Analysis API and returns raw JSON or None on failure
• aa_data_to_model_index (lines 893-914): Transforms API payloads into the Hugging Face model-index format
• import_aa_evaluations (lines 943-966): Drives the complete import workflow, handles merging, and manages PR creation

Prerequisites and Dependencies

The skill requires specific Python packages declared in the skill's dependency list:

uv pip install requests>=2.32.5 huggingface_hub>=0.26.0 python-dotenv

You must configure the AA_API_KEY environment variable by creating a .env file in your working directory:

echo "AA_API_KEY=your_api_key_here" > .env

Usage Examples

Run the import directly using the evaluation manager CLI:

uv run scripts/evaluation_manager.py import-aa \
    --repo-id "username/model-name" \
    --creator-slug "username" \
    --model-name "model-name" \
    --create-pr

Alternatively, use the provided example script at skills/hugging-face-evaluation/examples/artificial_analysis_to_hub.py:

uv run skills/hugging-face-evaluation/examples/artificial_analysis_to_hub.py \
    --repo-id "username/model-name" \
    --creator-slug "username" \
    --model-name "model-name"

Both methods support the --create-pr flag to automatically open pull requests with properly formatted descriptions linking back to the Artificial Analysis dashboard.

Key Architecture Features

Modular CLI Design: The evaluation_manager.py script uses argparse to expose sub-commands (get-prs, inspect-tables, extract-readme, import-aa), making the Artificial Analysis integration composable with other evaluation workflows like vLLM or lighteval.

Robust Error Handling: Network failures, API errors, and missing models are caught and logged without aborting the entire workflow, allowing batch operations to continue processing remaining models.

Idempotent Operations: The duplicate PR check ensures repeated runs are safe and do not flood model repositories with redundant pull requests.

Summary

• The hugging-face-evaluation skill automates importing Artificial Analysis benchmark scores into Hugging Face model cards via the import-aa CLI command.
• The workflow requires setting the AA_API_KEY environment variable and uses three core functions in evaluation_manager.py to fetch, transform, and merge data.
• Raw API data converts to the Hugging Face model-index format with proper attribution to Artificial Analysis.
• The system prevents duplicate PRs by checking existing open pull requests before creating new ones.
• Both direct CLI usage and example scripts are available in the skills/hugging-face-evaluation directory.

Frequently Asked Questions

How do I obtain an Artificial Analysis API key?

You must sign up for an account at Artificial Analysis and generate an API key from your dashboard. Store this key in the AA_API_KEY environment variable or a .env file in your project root, as shown in the examples/.env.example file within the huggingface/skills repository.

What happens if the model doesn't exist in the Artificial Analysis database?

The get_aa_model_data function returns None and logs an error message when it encounters a 404 response or any non-200 status code. The workflow skips that specific model and continues processing any remaining items in your batch, ensuring network issues don't crash the entire import operation.

Can I run this import without creating a pull request?

Yes. Omit the --create-pr flag when running the import-aa command. The script will still fetch and transform the data, displaying the results locally without modifying your Hugging Face repository or opening any pull requests.

Which file in my repository gets modified by this skill?

The skill updates your repository's model-index.yml file, which stores the structured benchmark metadata that powers Hugging Face model card leaderboards. If this file doesn't exist, the skill creates it; if it exists, it merges new Artificial Analysis entries with your existing evaluation data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}