How to Serve the llmfit REST API: Complete Setup and Endpoint Guide

The llmfit REST API is served by the llmfit-tui binary which starts an Axum-based HTTP server via the run_serve function in llmfit-tui/src/serve_api.rs, exposing JSON endpoints for model discovery, system specs, and deployment planning on TCP or Unix sockets.

The AlexsJones/llmfit repository provides a comprehensive hardware compatibility analyzer for Large Language Models. Beyond its terminal interface, llmfit embeds a production-ready REST API server that exposes the core analysis engine via HTTP, enabling programmatic access to model fitting algorithms, hardware detection, and deployment planning without launching the interactive TUI.

Starting the llmfit REST API Server

The server is embedded within the llmfit-tui crate and activated through command-line flags. You can bind to a TCP address for network access or use a Unix domain socket for local-only communication.

Start on TCP (default):

cargo run --bin llmfit-tui -- --host 0.0.0.0 --port 8080

Start on Unix socket (Linux/macOS):

cargo run --bin llmfit-tui -- --unix-socket /tmp/llmfit.sock

Once started, the binary prints the listening address and sample URLs, such as http://0.0.0.0:8080/api/v1/models?limit=20&min_fit=marginal&sort=score.

Server Architecture and Startup Flow

When run_serve is invoked in llmfit-tui/src/serve_api.rs, it executes a five-phase initialization sequence:

1. Hardware Detection – Calls super::detect_specs to generate a SystemSpecs struct containing CPU, RAM, and GPU capabilities.
2. Catalog Loading – Instantiates a ModelDatabase and clones the complete model list into memory.
3. State Construction – Builds an AppState containing node information, hardware specs, the model list, optional context limits, and a thread-safe download tracker.
4. Router Assembly – Invokes build_router(state) to wire Axum routes for all public endpoints.
5. Runtime Binding – Launches an async Tokio runtime and binds either a TCP listener or Unix domain socket.

All handlers return either Json responses or an ApiError that implements IntoResponse, ensuring errors are always serialized as {"error":"..."} with appropriate HTTP status codes.

REST API Endpoint Reference

The build_router function in llmfit-tui/src/serve_api.rs exposes the following endpoints, organized by functional area:

Health and System Information

• GET /health – Simple liveness check returning operational status and node metadata.
• GET /api/v1/system – Returns detailed hardware detection results (CPU, RAM, VRAM). Supports hardware overrides via query parameters (ram_gb, vram_gb, cpu_cores) merged by the effective_specs function for on-the-fly "what-if" analysis.

Example with hardware overrides:

curl "http://localhost:8080/api/v1/system?ram_gb=64&vram_gb=24&cpu_cores=16" | jq

Model Discovery and Ranking

• GET /api/v1/models – Returns a paginated list of model fits with extensive filtering capabilities (provider, minimum fit level, use case).
• GET /api/v1/models/top – Shortcut endpoint for retrieving the top N models by a specific sort criterion.
• GET /api/v1/models/{name} – Retrieves detailed fit information for a specific model by exact name match.

Filter for Ollama-compatible coding models:

curl "http://localhost:8080/api/v1/models?provider=ollama&min_fit=good&use_case=coding" | jq

Runtime and Installation Status

• GET /api/v1/runtimes – Detects which provider runtimes (Ollama, Mlx, LlamaCpp, etc.) are installed on the host system.
• GET /api/v1/installed – Lists specific models already installed per detected runtime.

Download and Deployment Operations

• POST /api/v1/download – Initiates a model pull operation. Restricted to localhost connections only for security.
• GET /api/v1/download/{id}/status – Polls the progress of an active download.
• POST /api/v1/plan – Generates a deployment plan estimating context limits, quantization levels, and expected tokens-per-second for specified hardware constraints.

Start a download (localhost only):

curl -X POST http://localhost:8080/api/v1/download \
    -H "Content-Type: application/json" \
    -d '{"model":"llama-2-7b","runtime":"ollama"}' | jq

Estimate a deployment plan:

curl -X POST http://localhost:8080/api/v1/plan \
    -H "Content-Type: application/json" \
    -d '{
          "model":"llama-2-7b",
          "context":2048,
          "quant":"q4_0",
          "target_tps":30,
          "ram_gb":48,
          "vram_gb":12,
          "cpu_cores":8
        }' | jq

Security Model and Localhost Restrictions

Because download endpoints manipulate the local filesystem, they implement strict localhost validation. The handlers extract ConnectInfo<SocketAddr> and verify addr.ip().is_loopback(); requests from remote addresses return 403 FORBIDDEN. This security boundary is enforced in serve_api.rs alongside the route definitions.

Hardware Override for "What-If" Analysis

The API supports temporary hardware specification overrides without modifying system detection. When query parameters like ram_gb=128 or vram_gb=48 are present, the effective_specs function (lines 331-365 in serve_api.rs) merges these values into the base SystemSpecs, allowing you to simulate deployment scenarios on different hardware configurations via API calls alone.

Embedded React Dashboard

All static web assets for the React-based dashboard are compiled into the binary via include!(concat!(env!("OUT_DIR"), "/web_assets.rs")). The fallback route /* serves index.html, while serve_web_path handles content-type detection and caching headers for assets under /assets/*. Simply navigate to the root URL to access the graphical interface.

Summary

• The llmfit REST API is served by the llmfit-tui binary using Axum and Tokio, defined primarily in llmfit-tui/src/serve_api.rs.
• Start the server with --host and --port for TCP or --unix-socket for local socket access.
• Endpoints cover health checks, system specs, model discovery, runtime detection, downloads, and deployment planning.
• Download endpoints are strictly limited to localhost connections for security.
• Hardware overrides enable simulated "what-if" analysis by passing query parameters to relevant endpoints.
• A full React dashboard is embedded in the binary and served via fallback routes.

Frequently Asked Questions

How do I start the llmfit REST API on a custom port?

Pass the --port flag when running the llmfit-tui binary. For example, cargo run --bin llmfit-tui -- --host 0.0.0.0 --port 9000 binds the server to port 9000 on all interfaces. You may also specify --unix-socket /path/to/socket for Unix domain socket access instead of TCP.

Can I access the llmfit API from remote machines?

Yes, the TCP listener respects the --host binding (e.g., 0.0.0.0 for all interfaces). However, specific endpoints like POST /api/v1/download enforce localhost-only access by checking the source IP address; these will return a 403 error if accessed remotely regardless of the host binding.

What framework powers the llmfit REST API?

The server is built on Axum, part of the Tokio ecosystem, as implemented in llmfit-tui/src/serve_api.rs. It uses standard Tower middleware for routing and axum::extract::ConnectInfo for client IP detection. All JSON serialization relies on shared conversion utilities in llmfit-tui/src/serve_shared.rs.

How do I perform "what-if" hardware analysis via the API?

Append hardware override parameters to qualifying endpoints like /api/v1/system or /api/v1/plan. Valid parameters include ram_gb, vram_gb, and cpu_cores. The effective_specs function merges these overrides with detected system specs, returning results as if the hardware matched your specified values without requiring physical hardware changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}