Checking for Existing PRs Before Creating New Evaluation PRs in Hugging Face Skills

Use the get_open_prs function in evaluation_manager.py to query the Hugging Face API for open pull requests before invoking any command with --create-pr.

The 🤗 Skills repository provides automation tools for extracting benchmark results and updating model cards on the Hugging Face Hub. Before creating new evaluation pull requests, you must verify that no existing open PR already contains the same changes to prevent duplicates and reduce repository noise.

How the PR Checking System Works

The repository implements a two-layer approach to PR discovery: a core API function for programmatic access and a CLI wrapper for human-readable output.

The Core API Function: get_open_prs

Located in skills/hugging-face-evaluation/scripts/evaluation_manager.py at lines 887–904, the get_open_prs function queries the Hugging Face Discussions API to retrieve open pull requests for a specific model repository.

def get_open_prs(repo_id: str) -> List[Dict[str, Any]]:
    """
    Fetch open pull requests for a Hugging Face model repository.
    """
    requests = require_requests()
    url = f"https://huggingface.co/api/models/{repo_id}/discussions"

    try:
        response = requests.get(url, timeout=30, allow_redirects=True)
        response.raise_for_status()
        data = response.json()
        discussions = data.get("discussions", [])
        # Keep only open pull‑request discussions

        open_prs = [
            {
                "num": d["num"],
                "title": d["title"],
                "author": d["author"]["name"],
                "createdAt": d.get("createdAt", "unknown"),
            }
            for d in discussions
            if d.get("status") == "open" and d.get("isPullRequest")
        ]
        return open_prs
    except requests.RequestException as e:
        print(f"Error fetching PRs from Hugging Face: {e}")
        return []

The function filters the API response to include only discussions where status equals "open" and isPullRequest is true, returning a list of dictionaries containing the PR number, title, author, and creation timestamp.

CLI Interface: get-prs and list_open_prs

For command-line usage, the script exposes a get-prs sub-command defined at lines 1275–1290 in evaluation_manager.py. This command invokes list_open_prs (lines 926–944), which formats the API results for human readability.

def list_open_prs(repo_id: str) -> None:
    prs = get_open_prs(repo_id)
    print("\n" + "=" * 70)
    print(f"Open Pull Requests for: {repo_id}")
    print("=" * 70)

    if not prs:
        print("\nNo open pull requests found.")
    else:
        print(f"\nFound {len(prs)} open PR(s):\n")
        for pr in prs:
            print(f"  PR #{pr['num']} - {pr['title']}")
            print(f"     Author: {pr['author']}")
            print(f"     Created: {pr['createdAt']}")
            print(f"     URL: https://huggingface.co/{repo_id}/discussions/{pr['num']}")
            print()
    print("=" * 70 + "\n")

The output includes direct URLs to each PR discussion page, allowing you to quickly verify whether an existing evaluation update is already pending.

Practical Usage Examples

Command Line Workflow

Before creating any evaluation PR, run the get-prs command to check for existing open pull requests:

uv run scripts/evaluation_manager.py get-prs --repo-id "my-org/my-model"

Sample output:

======================================================================
Open Pull Requests for: my-org/my-model
======================================================================

Found 1 open PR(s):

  PR #42 - Update evaluation tables for MMLU
     Author: alice
     Created: 2024-09-12T14:03:00Z
     URL: https://huggingface.co/my-org/my-model/discussions/42

======================================================================

If this command returns no results, you can safely proceed with --create-pr.

Python API Integration

You can also integrate PR checking directly into custom automation scripts:

from skills.hugging_face_evaluation.scripts.evaluation_manager import get_open_prs

repo_id = "my-org/my-model"
open_prs = get_open_prs(repo_id)

if open_prs:
    print(f"There are already {len(open_prs)} open PR(s):")
    for pr in open_prs:
        print(f"  #{pr['num']}: {pr['title']} (by {pr['author']})")
else:
    print("No open PRs – safe to create a new one.")

This pattern is particularly useful when building batch processing pipelines that evaluate multiple models sequentially.

Full Evaluation Pipeline

Here is the complete recommended workflow for adding evaluation tables to a model card:

# 1️⃣ Inspect tables to identify the right table number

uv run scripts/evaluation_manager.py inspect-tables --repo-id my-org/my-model

# 2️⃣ Verify no open PR already exists

uv run scripts/evaluation_manager.py get-prs --repo-id my-org/my-model

# 3️⃣ Extract tables and push a PR (if safe)

uv run scripts/evaluation_manager.py extract-readme \
    --repo-id my-org/my-model \
    --table 2 \
    --model-column-index 1 \
    --create-pr

If step 2 reports an existing PR, either wait for it to merge or coordinate with the existing PR author before proceeding.

Implementation Details and Source Files

The PR checking functionality is implemented across several components in the repository:

File	Purpose	Direct link
skills/hugging-face-evaluation/scripts/evaluation_manager.py	Core CLI tool; implements get_open_prs, list_open_prs, and the get-prs sub‑command.	view on GitHub
apps/evals-leaderboard/collect_evals.py	Demonstrates PR fetching in a different context (leaderboard aggregation) via _fetch_pull_requests.	view on GitHub
skills/hugging-face-evaluation/examples/artificial_analysis_to_hub.py	Example script that can optionally create a PR (--create-pr). Shows the importance of the PR‑check step.	view on GitHub
skills/hugging-face-paper-publisher/scripts/paper_manager.py	Similar PR handling (--create-pr) for publishing papers; mirrors the same design pattern.	view on GitHub

The get_open_prs function specifically queries the endpoint https://huggingface.co/api/models/{repo_id}/discussions and filters for discussions where status == "open" and isPullRequest is true, as implemented in evaluation_manager.py lines 887–904.

Summary

• Always check first: Use get_open_prs or the get-prs CLI command before running any tool with --create-pr to prevent duplicate pull requests.
• API endpoint: The function queries https://huggingface.co/api/models/{repo_id}/discussions and filters for open pull requests using the status and isPullRequest fields.
• CLI integration: The evaluation_manager.py script exposes this functionality via the get-prs sub-command, which formats results with direct URLs to discussion pages.
• Cross-component reuse: The same PR checking pattern appears in collect_evals.py for leaderboard aggregation and in paper publishing workflows, demonstrating a consistent architectural approach across the repository.

Frequently Asked Questions

How do I check for existing PRs before creating a new evaluation PR?

Run the get-prs command from evaluation_manager.py with your repository ID:

uv run scripts/evaluation_manager.py get-prs --repo-id "org/model-name"

If the command returns "No open pull requests found," you can safely proceed with --create-pr. If it lists existing PRs, review them to ensure you are not submitting duplicate changes.

What API endpoint does the PR checker use?

The get_open_prs function in evaluation_manager.py queries the Hugging Face Hub Discussions API at:

https://huggingface.co/api/models/{repo_id}/discussions

It filters the response to include only discussions where status equals "open" and isPullRequest is true, returning metadata including the PR number, title, author, and creation date.

Can I use the PR checking functionality in my own Python scripts?

Yes, you can import get_open_prs directly from the evaluation manager module:

from skills.hugging_face_evaluation.scripts.evaluation_manager import get_open_prs

existing_prs = get_open_prs("my-org/my-model")
if not existing_prs:
    # Safe to create new PR

    pass

This allows you to integrate duplicate PR prevention into custom automation pipelines or batch processing workflows.

What happens if I try to create a PR when one already exists?

If you attempt to use --create-pr without first checking for existing open PRs, the Hugging Face Hub API will raise a duplicate PR error. The evaluation_manager.py script includes help text explicitly recommending that you run get-prs before using --create-pr to avoid this failure and prevent repository noise.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}