How to Use llmfit's JSON API for Scripting and Automation

llmfit exposes a built‑in HTTP server via llmfit serve that provides a stateless, JSON‑first REST API for automated model selection, hardware simulation, and deployment planning.

The llmfit CLI tool ships with an embedded Axum HTTP server (implemented in llmfit-tui/src/serve_api.rs) designed specifically for integration into scripts, CI pipelines, and automation workflows. By querying endpoints like /api/v1/models, /api/v1/system, and /api/v1/plan, you can programmatically determine which LLMs fit your hardware constraints—without manual interaction. This guide covers every endpoint, request patterns, and working code examples for both shell scripts and Python.

Starting the llmfit API Server

Before issuing requests, you must start the server. llmfit provides two equivalent methods:

• CLI command: llmfit serve (listens on 0.0.0.0:8080 by default)
• Direct invocation: call the run_serve function from Rust code

The server is stateless aside from download tracking, making it safe to restart or scale behind a load balancer.

Understanding the API Response Format

Every endpoint returns a consistent ApiEnvelope struct defined at lines 94‑102 in serve_api.rs. This envelope normalizes responses and carries metadata about the node, effective system specs, and result set.

{
  "node": { "name": "workstation-01", "os": "linux" },
  "system": { "ram_gb": 64, "vram_gb": 24, "cpu_cores": 16 },
  "total_models": 847,
  "returned_models": 20,
  "filters": { "ram_gb": "64", "use_case": "coding" },
  "models": [ /* array of ModelFit objects */ ]
}

Key fields in the envelope:

• node: Host identification for distributed environments
• system: Hardware specs actually used for the query (after overrides applied)
• total_models vs returned_models: Pagination awareness
• filters: Echo of query parameters for debugging

Core Endpoints for Automation

The router registration in build_router (lines 48‑64) wires these paths to handler functions. Below are the endpoints most useful for scripting, with implementation locations and use cases.

GET /health – Liveness Probes

Defined at lines 66‑74. Returns minimal JSON for load balancers and monitoring:

curl -s http://localhost:8080/health | jq .

{ "status": "ok" }

Use this in Kubernetes liveness probes or systemd health checks.

GET /api/v1/system – Hardware Discovery with Simulation

Implementation at lines 76‑89. Returns detected hardware unless overridden by query parameters.

Hardware override parameters (parsed by HardwareOverrideQuery at lines 56‑63):

Parameter	Type	Description
ram_gb	integer	Simulated system RAM
vram_gb	integer	Simulated GPU VRAM
cpu_cores	integer	Simulated CPU core count

Example: Simulate a high‑end workstation without owning one:

curl -s "http://localhost:8080/api/v1/system?ram_gb=128&vram_gb=48&cpu_cores=32" | jq '.system'

The effective_specs logic (lines 31‑65) merges overrides with auto‑detection, enabling "what‑if" scenarios in automation.

GET /api/v1/models – Filtered Model Enumeration

Core logic at lines 91‑117. Supports extensive filtering, sorting, and pagination.

Query parameters:

• ram_gb, vram_gb, cpu_cores – hardware overrides
• provider – filter by source (ollama, huggingface, mlx, etc.)
• use_case – "coding", "chat", "embedding", etc.
• sort – "score", "ram", "vram", "name"
• limit, offset – pagination

curl -s "http://localhost:8080/api/v1/models?use_case=coding&sort=score&limit=10" | jq '.models[].name'

GET /api/v1/models/top – Shortcut for Ranked Results

Lines 119‑145. Forces sort=score and descending rank, useful for "best N" queries:

curl -s "http://localhost:8080/api/v1/models/top?limit=3&vram_gb=16" | jq '.models[] | {name, score, fit_level}'

GET /api/v1/models/{name} – Single Model Lookup

Lines 147‑173. Retrieve fit analysis for a specific model. Supports partial name matching.

curl -s "http://localhost:8080/api/v1/models/Llama-3.1-8B" | jq '.models[0].estimated_tps'

Combine with hardware overrides to evaluate fit on different target machines:

curl -s "http://localhost:8080/api/v1/models/Mixtral-8x7B?ram_gb=64&vram_gb=48" | jq '.models[0].fit_level'

GET /api/v1/runtimes – Runtime Discovery

Lines 175‑203. Enumerates which inference engines are installed locally (Ollama, MLX, LlamaCpp, Docker, LMStudio, vLLM). The provider detection logic lives in llmfit-core/src/providers.rs.

curl -s http://localhost:8080/api/v1/runtimes | jq '.runtimes[]'

Use this to conditionally deploy models only where their runtime exists.

GET /api/v1/installed – Local Model Inventory

Lines 205‑233. Lists models already pulled/installed per runtime—useful for avoiding redundant downloads.

curl -s http://localhost:8080/api/v1/installed | jq '.installed'

POST /api/v1/plan – Deployment Estimation

Lines 235‑285. The most powerful endpoint for automation: given a model and target parameters, returns a deployment plan with quantization recommendation, context sizing, and throughput estimate.

Request body fields:

Field	Required	Description
model	yes	Model name or identifier
context	no	Desired context window (tokens)
quant	no	Preferred quantization (q4_0, q5_k_m, fp16, etc.)
target_tps	no	Desired tokens‑per‑second throughput
ram_gb, vram_gb, cpu_cores	no	Target hardware specs

curl -X POST http://localhost:8080/api/v1/plan \
  -H "Content-Type: application/json" \
  -d '{
        "model": "Llama-3.1-70B",
        "context": 8192,
        "target_tps": 50,
        "vram_gb": 80,
        "quant": "q4_k_m"
      }' | jq '.plan'

Complete Automation Examples

Shell Script: Select Best Model for CI Runner

#!/bin/bash
set -euo pipefail

LLMFIT_HOST="${LLMFIT_HOST:-http://localhost:8080}"
USE_CASE="${1:-coding}"
LIMIT="${2:-3}"

# Detect runner hardware

HW=$(curl -s "${LLMFIT_HOST}/api/v1/system")
RAM_GB=$(echo "$HW" | jq '.system.ram_gb')
VRAM_GB=$(echo "$HW" | jq '.system.vram_gb // 0')

echo "Runner specs: ${RAM_GB}GB RAM, ${VRAM_GB}GB VRAM"

# Get top models

MODELS=$(curl -s "${LLMFIT_HOST}/api/v1/models/top?limit=${LIMIT}&use_case=${USE_CASE}&ram_gb=${RAM_GB}&vram_gb=${VRAM_GB}")

echo "Recommended models:"
echo "$MODELS" | jq -r '.models[] | "\(.score)\t\(.name)\t\(.fit_level)"'

Python: Integration with Requests

import requests
import os

BASE = os.getenv("LLMFIT_URL", "http://localhost:8080/api/v1")

class LlmfitClient:
    def __init__(self, base_url: str = BASE):
        self.base = base_url
        self.session = requests.Session()

    def system_specs(self, **overrides) -> dict:
        """Get system info with optional hardware overrides."""
        resp = self.session.get(f"{self.base}/system", params=overrides)
        resp.raise_for_status()
        return resp.json()["system"]

    def best_models(
        self,
        use_case: str = "chat",
        limit: int = 5,
        **hardware
    ) -> list[dict]:
        """Return top-scoring models matching constraints."""
        params = {"use_case": use_case, "limit": limit, **hardware}
        resp = self.session.get(f"{self.base}/models/top", params=params)
        resp.raise_for_status()
        return resp.json()["models"]

    def deployment_plan(self, model: str, **kwargs) -> dict:
        """Generate a deployment plan for automation."""
        payload = {"model": model, **kwargs}
        resp = self.session.post(f"{self.base}/plan", json=payload)
        resp.raise_for_status()
        return resp.json()["plan"]

# Usage

client = LlmfitClient()

# Simulate cloud instance specs

specs = client.system_specs(ram_gb=32, vram_gb=0, cpu_cores=8)
print(f"Effective specs: {specs}")

# Find lightweight coding models

candidates = client.best_models(use_case="coding", limit=3, **specs)
for m in candidates:
    print(f"{m['name']}: score={m['score']:.2f}, fit={m['fit_level']}")

# Plan deployment for best candidate

if candidates:
    plan = client.deployment_plan(
        candidates[0]["name"],
        context=4096,
        target_tps=100
    )
    print(f"Recommended quant: {plan['quantization']}")

Ansible: Dynamic Inventory Integration

# llmfit_inventory.yml

# Custom Ansible inventory script using llmfit API

- name: Query llmfit for available runtimes
  uri:
    url: "{{ llmfit_host }}/api/v1/runtimes"
    return_content: yes
  register: runtime_check

- name: Filter hosts by LLM capability
  set_fact:
    llm_capable: "{{ runtime_check.json.runtimes | selectattr('available') | list }}"

Key Implementation Files

Understanding these source files helps debug API behavior and extend functionality:

File	Purpose	Key Symbols
llmfit-tui/src/serve_api.rs	Axum router, handlers, ApiEnvelope	build_router, run_serve, HardwareOverrideQuery
llmfit-tui/src/serve_shared.rs	JSON serialization helpers	system_json, fit_to_json
llmfit-core/src/hardware.rs	SystemSpecs struct, detection logic	SystemSpecs, hardware override application
llmfit-core/src/models.rs	Model metadata and parameter formulas	Model definitions, RAM/VRAM calculation
llmfit-core/src/fit.rs	Fit analysis engine	ModelFit, scoring algorithm
llmfit-core/src/providers.rs	Runtime provider definitions	Provider enum, availability detection

Performance and Reliability Considerations

• Stateless design: Safe to restart, scale horizontally, or run behind reverse proxies
• No authentication: Deploy behind VPN or add proxy auth for exposed instances
• Default port: 8080; override via environment or CLI flags in custom builds
• Response size: Model list responses can be large (~100KB); use limit and offset for pagination
• Hardware override overhead: Negligible; overrides bypass physical detection entirely

Summary

• Run llmfit serve to start the JSON API on port 8080
• All endpoints accept ram_gb, vram_gb, cpu_cores to simulate hardware without physical changes
• Use /api/v1/models/top for "best N" queries and /api/v1/plan for deployment estimation
• The ApiEnvelope wrapper provides consistent metadata across all responses
• Core implementation lives in serve_api.rs with serialization helpers in serve_shared.rs
• Python, shell, and infrastructure‑as‑code tools integrate via standard HTTP/JSON

Frequently Asked Questions

What port does llmfit serve run on by default?

The server listens on 0.0.0.0:8080 by default when started with llmfit serve. You can modify this by setting environment variables or adjusting the source in run_serve before compiling a custom build.

Can I query the API without starting the server manually?

No—the llmfit serve subcommand or direct run_serve invocation is required. The API is not embedded in the standard CLI operations; it runs as a separate asynchronous HTTP server using Axum.

How do hardware overrides affect the response?

Query parameters ram_gb, vram_gb, and cpu_cores override auto‑detected values via HardwareOverrideQuery (lines 56‑63). The effective_specs logic (lines 31‑65) substitutes these into SystemSpecs, causing all downstream fit calculations to use simulated hardware. This enables capacity planning for machines you do not yet possess.

Is the API suitable for production automation?

The API is intentionally simple and stateless, making it suitable for CI/CD pipelines and internal tooling. However, it ships without built‑in authentication or rate limiting—deploy it behind a reverse proxy (nginx, Traefik, cloud load balancer) with appropriate security controls for production exposure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}