How llmfit Community Benchmark Submission Works Without CLI or GitHub Accounts

llmfit enables anonymous benchmark contributions through local JSON storage, OAuth device flow authentication, and direct GitHub API calls—no gh CLI or pre-existing account required.

The llmfit open-source project simplifies community-driven LLM benchmarking by removing traditional barriers to participation. Instead of requiring users to install the GitHub CLI or maintain active accounts, the tool implements a self-contained submission flow using standard HTTP requests and OAuth device authentication. This design makes community benchmark submission accessible from CI pipelines, containerized environments, and headless machines.

Local Benchmark Storage First

Every benchmark run begins with local persistence. The store_local function in llmfit-core/src/share.rs writes results as JSON payloads to the user's data directory.

// Record benchmark results locally (automatic hardware detection)
let results = vec![bench_result];
let specs   = SystemSpecs::detect()?;               // probes CPU, GPU, memory
let path = llmfit_core::share::store_local(&results, &specs)?;

Results land in $XDG_DATA_HOME/llmfit/benchmarks/pending/*.json (typically ~/.local/share/llmfit/benchmarks/pending/ on Linux). This decouples benchmark execution from submission timing—users can run benchmarks offline and share results later.

Key implementation details from llmfit-core/src/share.rs lines 95-111:

• Each payload includes detected hardware specs, timestamps, and benchmark metrics
• Filenames use stable hashes to prevent duplicate storage
• The function returns a PathBuf for verification or manual inspection

Authentication Without Existing Credentials

The resolve_token_noninteractive function implements a three-tier authentication strategy that requires zero pre-configuration:

1. Environment variables — Checks GITHUB_TOKEN or GH_TOKEN first
2. Cached tokens — Reuses tokens from previous device-flow logins via write_cached_token
3. OAuth device flow — Initiates fresh authentication when no token exists

When device flow begins, llmfit prints instructions to stderr (preserving stdout for piping):

To share benchmarks, please authenticate with GitHub:

1. Visit: https://github.com/login/device
2. Enter code: XXXX-XXXX

Waiting for authorization...

The flow uses GitHub's public OAuth device flow with DEFAULT_CLIENT_ID (llmfit-core/src/share.rs lines 39-65). After user authorization, device_flow_poll exchanges the device code for an access token. The token caches to disk for subsequent non-interactive runs.

This design means first-time contributors need only a browser—no GitHub account setup, no SSH keys, no CLI installation.

Payload Preparation and Idempotency

Before upload, llmfit reprocesses stored benchmarks to ensure clean, deduplicated submissions (llmfit-core/src/share.rs lines 58-76 and 160-178):

• Re-timestamping — Updates timestamps to submission time
• Hardware slug generation — payload_slug creates normalized identifiers like nvidia-rtx-4090-linux-x86_64
• Stable filenames — short_hash derives consistent names from content hash

These steps guarantee that re-running llmfit bench --share won't create duplicate files if previous submissions failed mid-way.

GitHub API Integration Without External Tools

All repository operations use ureq (a lightweight Rust HTTP client) rather than shelling out to gh. The flow in llmfit-core/src/share.rs lines 84-131 and 160-176:

Step	Function	API Endpoint
Fork creation	ensure_fork	POST /repos/{owner}/{repo}/forks
Branch creation	create_branch	POST /repos/{user}/{repo}/git/refs
File upload	put_file	PUT /repos/{user}/{repo}/contents/{path}
PR discovery	find_open_bench_pr	GET /repos/{owner}/{repo}/pulls
PR creation	open_pr	POST /repos/{owner}/{repo}/pulls

Branch names follow the pattern bench/{hardware-slug}-{hash}, making PRs discoverable and organized. Existing files are skipped during upload, and new benchmarks append to open PRs rather than spawning duplicates.

The generated PR includes a markdown table summarizing all contributed results for maintainer review.

Hardware Validation Gate

To maintain leaderboard quality, llmfit rejects submissions from unidentified hardware. The hardware_is_identifiable check (llmfit-core/src/share.rs lines 85-92) blocks uploads when:

• GPU detection returns "N/A" or empty strings
• Critical system specs are missing

This prevents polluted benchmarks from misconfigured containers or VMs without proper GPU passthrough.

Complete Sharing Workflow

// Share all pending benchmarks with automatic authentication
let opts = llmfit_core::share::ShareOptions { 
    dry_run: false,       // set true to preview without uploading
    assume_yes: true,     // skip interactive prompts
};

match llmfit_core::share::share_all_pending(&opts, None) {
    Ok(Some(outcome)) => println!("PR opened: {}", outcome.pr_url),
    Ok(None) => println!("No pending benchmarks or dry-run mode."),
    Err(e) => eprintln!("Share failed: {e}"),
}

Running this from a CI job triggers device flow on first execution, caches the token, and proceeds automatically thereafter. The binary handles the entire lifecycle—no container modifications or secret mounting required beyond standard environment variable support.

Key Source Files

• llmfit-core/src/share.rs — Core implementation: local storage, authentication, payload preparation, GitHub submission
• llmfit-tui/src/tui_app.rs (lines 335-388) — TUI integration providing the --share checkbox
• scripts/validate_community_benchmarks.py — CI helper for pre-submission JSON validation
• Cargo.toml — Declares ureq and base64 powering API calls

Summary

• Local-first design — Benchmarks store as JSON before any network operation, enabling offline workflows
• Zero-dependency authentication — OAuth device flow replaces CLI tools; environment variables skip interaction entirely
• Self-contained GitHub operations — ureq HTTP client handles all API calls without external binaries
• Deduplication and safety — Content hashing, hardware validation, and PR appending prevent noisy submissions
• CI-friendly — Headless operation with cached tokens supports automated benchmark pipelines

Frequently Asked Questions

Does llmfit require a GitHub account to submit benchmarks?

No pre-existing account is required for the initial device flow, but you will need to create a GitHub account during the OAuth authorization step if you don't have one. The tool guides you through this process in the browser. After first authentication, cached tokens enable fully non-interactive submissions.

Can I use llmfit in CI pipelines without interactive prompts?

Yes. Set GITHUB_TOKEN or GH_TOKEN as an environment variable, or run the device flow once locally to cache a token, then copy the cache file to your CI environment. Use assume_yes: true in ShareOptions to suppress all interactive prompts.

What happens if my hardware isn't detected properly?

Submissions are rejected locally before any network request. Ensure your GPU drivers are correctly installed and accessible from the runtime environment. Containerized workloads need proper GPU device passthrough for detection to succeed.

Is there a way to preview what would be submitted?

Set dry_run: true in ShareOptions. This validates all pending benchmarks, shows the generated payloads and branch names, and reports what PR operations would occur—without creating any forks, branches, or pull requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}