How llmfit Authenticates Benchmark Submissions to GitHub Without the gh CLI

The llmfit benchmark sharing feature authenticates directly against the GitHub REST API using the OAuth Device Flow and environment variable tokens, completely eliminating the need for the gh CLI tool.

The llmfit bench --share command enables developers to submit benchmark results as pull requests to a central repository. Unlike many GitHub-integrated tools that shell out to the gh command-line interface, llmfit implements a pure Rust authentication layer that communicates directly with GitHub's REST API. This article breaks down exactly how the benchmark submission flow handles authentication without external CLI dependencies.

Token Resolution and Validation

Before initiating any GitHub API calls, the code in llmfit-core/src/share.rs attempts to resolve a valid authentication token through a structured hierarchy.

Environment Variable Lookup

The primary authentication method checks for pre-existing credentials in environment variables. The function resolve_token_noninteractive (lines 30-41) searches for GITHUB_TOKEN or GH_TOKEN, allowing CI pipelines and scripts to inject credentials securely without interactive prompts.

If neither environment variable is present, the system falls back to read_cached_token (lines 36-41), which retrieves tokens stored from previous device-flow authentications in the user's local cache.

Token Validation via the User Endpoint

Once a candidate token is found, validate_token (lines 47-64) verifies its validity by sending a GET request to https://api.github.com/user. This ensures that expired or revoked tokens trigger the fallback authentication flow rather than failing during subsequent API operations.

OAuth Device Flow Implementation

When no valid token exists in the environment or cache, preflight_auth initiates the OAuth Device Flow—a protocol designed for input-constrained devices that cannot easily handle traditional OAuth redirects.

Device Flow Initiation

The authentication sequence begins with device_flow_start (lines 79-96), which sends a POST request to https://github.com/login/device/code. This request uses the public OAuth App client ID stored in DEFAULT_CLIENT_ID, though users can override this by setting the LLMFIT_GH_CLIENT_ID environment variable (lines 39-45).

The function returns a DeviceAuth struct containing the verification_uri and user_code that the CLI displays to the user.

User Authorization and Polling

After displaying the authentication URL and code, the system enters device_flow_poll (lines 109-123). This function repeatedly queries GitHub's authorization server until the user completes the browser-based authorization or the device code expires. Upon successful authorization, the polling returns DevicePoll::Token(tok), which is immediately persisted to the local cache via write_cached_token for future use.

Authenticated API Operations

With a validated token in hand, all subsequent GitHub interactions inject the credential via the api helper function (lines 73-108). This generic HTTP client automatically adds an Authorization: Bearer <token> header to every request.

The benchmark submission flow invokes this authenticated client through several specialized functions:

• ensure_fork – Creates a fork of the upstream repository if one does not exist
• create_branch – Generates a new branch for the benchmark data
• put_file – Uploads the benchmark JSON files via the GitHub Contents API
• open_pr – Opens the final pull request with the benchmark results

Complete Workflow Examples

Interactive Benchmark Sharing

For local development environments, trigger the device flow interactively:

$ llmfit bench --share

# CLI outputs:

# Please visit: https://github.com/login/device

# Enter code: ABCD-1234

# Waiting for authorization...

CI/CD Non-Interactive Authentication

In automated environments, bypass the interactive flow by providing a personal access token:

export GITHUB_TOKEN=ghp_XXXXXXXXXXXXXXXXXXXX
llmfit bench --share --assume-yes

Programmatic Authentication

Custom binaries can leverage the same authentication primitives:

use llmfit_core::share::{preflight_auth, submit_stored, ShareOptions, StoredBenchmark};

fn share_results(benchmarks: Vec<StoredBenchmark>) -> Result<(), String> {
    // Resolves token via env → cache → device flow
    let token = preflight_auth()?;
    
    // Submit benchmarks using authenticated API calls
    let outcome = submit_stored(&benchmarks, &token)?;
    println!("Pull request created: {:?}", outcome.pr_url);
    Ok(())
}

Summary

• No external dependencies: The entire authentication flow lives in llmfit-core/src/share.rs without invoking the gh binary.
• Triple fallback strategy: The system checks GITHUB_TOKEN/GH_TOKEN environment variables first, falls back to cached credentials, then initiates OAuth Device Flow as a last resort.
• Direct API communication: All operations use the GitHub REST API with Authorization: Bearer headers generated by the internal api helper.
• Persistent caching: Successfully acquired tokens are cached locally to avoid repeated device-flow prompts across multiple benchmark submissions.
• Configurable OAuth: Developers can override the default client ID using the LLMFIT_GH_CLIENT_ID environment variable for custom GitHub Apps.

Frequently Asked Questions

Does llmfit require the gh CLI to be installed?

No. According to the source code in llmfit-core/src/share.rs, llmfit implements its own GitHub API client and authentication handlers. The gh CLI is never invoked; all fork creation, file uploads, and pull request operations use direct HTTPS calls to api.github.com.

How long does the cached authentication token last?

The token cache persists indefinitely on the local filesystem until the GitHub token expires or is revoked. The validate_token function checks token validity before use, and invalid tokens trigger a fresh device-flow authentication automatically.

Can I use a GitHub App instead of a personal access token?

Yes. While the default implementation uses a public OAuth App ID (DEFAULT_CLIENT_ID), you can override this by setting the LLMFIT_GH_CLIENT_ID environment variable. However, the current implementation in share.rs expects OAuth App tokens rather than GitHub App installation tokens.

Is the device flow secure for CI environments?

The device flow requires interactive browser authentication, making it unsuitable for headless CI pipelines. For automated environments, provide a GITHUB_TOKEN environment variable containing a fine-grained personal access token or classic token with repo and fork permissions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}