# How to Use llmfit's JSON API for Scripting and Automation

> Automate model selection and deployment with llmfit's JSON API. Leverage its REST API for scripting and seamless integration.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: how-to-guide
- Published: 2026-08-20

---

**llmfit exposes a built‑in HTTP server via `llmfit serve` that provides a stateless, JSON‑first REST API for automated model selection, hardware simulation, and deployment planning.**

The llmfit CLI tool ships with an embedded Axum HTTP server (implemented in [`llmfit-tui/src/serve_api.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/serve_api.rs)) designed specifically for integration into scripts, CI pipelines, and automation workflows. By querying endpoints like `/api/v1/models`, `/api/v1/system`, and `/api/v1/plan`, you can programmatically determine which LLMs fit your hardware constraints—without manual interaction. This guide covers every endpoint, request patterns, and working code examples for both shell scripts and Python.

## Starting the llmfit API Server

Before issuing requests, you must start the server. llmfit provides two equivalent methods:

- **CLI command**: `llmfit serve` (listens on `0.0.0.0:8080` by default)
- **Direct invocation**: call the `run_serve` function from Rust code

The server is stateless aside from download tracking, making it safe to restart or scale behind a load balancer.

## Understanding the API Response Format

Every endpoint returns a consistent **ApiEnvelope** struct defined at lines 94‑102 in [`serve_api.rs`](https://github.com/AlexsJones/llmfit/blob/main/serve_api.rs). This envelope normalizes responses and carries metadata about the node, effective system specs, and result set.

```json
{
  "node": { "name": "workstation-01", "os": "linux" },
  "system": { "ram_gb": 64, "vram_gb": 24, "cpu_cores": 16 },
  "total_models": 847,
  "returned_models": 20,
  "filters": { "ram_gb": "64", "use_case": "coding" },
  "models": [ /* array of ModelFit objects */ ]
}

```

Key fields in the envelope:

- **node**: Host identification for distributed environments
- **system**: Hardware specs actually used for the query (after overrides applied)
- **total_models** vs **returned_models**: Pagination awareness
- **filters**: Echo of query parameters for debugging

## Core Endpoints for Automation

The router registration in `build_router` (lines 48‑64) wires these paths to handler functions. Below are the endpoints most useful for scripting, with implementation locations and use cases.

### GET /health – Liveness Probes

Defined at lines 66‑74. Returns minimal JSON for load balancers and monitoring:

```bash
curl -s http://localhost:8080/health | jq .

```

```json
{ "status": "ok" }

```

Use this in Kubernetes liveness probes or systemd health checks.

### GET /api/v1/system – Hardware Discovery with Simulation

Implementation at lines 76‑89. Returns detected hardware unless overridden by query parameters.

**Hardware override parameters** (parsed by `HardwareOverrideQuery` at lines 56‑63):

| Parameter | Type | Description |
|-----------|------|-------------|
| `ram_gb` | integer | Simulated system RAM |
| `vram_gb` | integer | Simulated GPU VRAM |
| `cpu_cores` | integer | Simulated CPU core count |

Example: Simulate a high‑end workstation without owning one:

```bash
curl -s "http://localhost:8080/api/v1/system?ram_gb=128&vram_gb=48&cpu_cores=32" | jq '.system'

```

The `effective_specs` logic (lines 31‑65) merges overrides with auto‑detection, enabling "what‑if" scenarios in automation.

### GET /api/v1/models – Filtered Model Enumeration

Core logic at lines 91‑117. Supports extensive filtering, sorting, and pagination.

**Query parameters:**

- `ram_gb`, `vram_gb`, `cpu_cores` – hardware overrides
- `provider` – filter by source (ollama, huggingface, mlx, etc.)
- `use_case` – "coding", "chat", "embedding", etc.
- `sort` – "score", "ram", "vram", "name"
- `limit`, `offset` – pagination

```bash
curl -s "http://localhost:8080/api/v1/models?use_case=coding&sort=score&limit=10" | jq '.models[].name'

```

### GET /api/v1/models/top – Shortcut for Ranked Results

Lines 119‑145. Forces `sort=score` and descending rank, useful for "best N" queries:

```bash
curl -s "http://localhost:8080/api/v1/models/top?limit=3&vram_gb=16" | jq '.models[] | {name, score, fit_level}'

```

### GET /api/v1/models/{name} – Single Model Lookup

Lines 147‑173. Retrieve fit analysis for a specific model. Supports partial name matching.

```bash
curl -s "http://localhost:8080/api/v1/models/Llama-3.1-8B" | jq '.models[0].estimated_tps'

```

Combine with hardware overrides to evaluate fit on different target machines:

```bash
curl -s "http://localhost:8080/api/v1/models/Mixtral-8x7B?ram_gb=64&vram_gb=48" | jq '.models[0].fit_level'

```

### GET /api/v1/runtimes – Runtime Discovery

Lines 175‑203. Enumerates which inference engines are installed locally (Ollama, MLX, LlamaCpp, Docker, LMStudio, vLLM). The provider detection logic lives in [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs).

```bash
curl -s http://localhost:8080/api/v1/runtimes | jq '.runtimes[]'

```

Use this to conditionally deploy models only where their runtime exists.

### GET /api/v1/installed – Local Model Inventory

Lines 205‑233. Lists models already pulled/installed per runtime—useful for avoiding redundant downloads.

```bash
curl -s http://localhost:8080/api/v1/installed | jq '.installed'

```

### POST /api/v1/plan – Deployment Estimation

Lines 235‑285. The most powerful endpoint for automation: given a model and target parameters, returns a deployment plan with quantization recommendation, context sizing, and throughput estimate.

**Request body fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `model` | yes | Model name or identifier |
| `context` | no | Desired context window (tokens) |
| `quant` | no | Preferred quantization (q4_0, q5_k_m, fp16, etc.) |
| `target_tps` | no | Desired tokens‑per‑second throughput |
| `ram_gb`, `vram_gb`, `cpu_cores` | no | Target hardware specs |

```bash
curl -X POST http://localhost:8080/api/v1/plan \
  -H "Content-Type: application/json" \
  -d '{
        "model": "Llama-3.1-70B",
        "context": 8192,
        "target_tps": 50,
        "vram_gb": 80,
        "quant": "q4_k_m"
      }' | jq '.plan'

```

## Complete Automation Examples

### Shell Script: Select Best Model for CI Runner

```bash
#!/bin/bash
set -euo pipefail

LLMFIT_HOST="${LLMFIT_HOST:-http://localhost:8080}"
USE_CASE="${1:-coding}"
LIMIT="${2:-3}"

# Detect runner hardware

HW=$(curl -s "${LLMFIT_HOST}/api/v1/system")
RAM_GB=$(echo "$HW" | jq '.system.ram_gb')
VRAM_GB=$(echo "$HW" | jq '.system.vram_gb // 0')

echo "Runner specs: ${RAM_GB}GB RAM, ${VRAM_GB}GB VRAM"

# Get top models

MODELS=$(curl -s "${LLMFIT_HOST}/api/v1/models/top?limit=${LIMIT}&use_case=${USE_CASE}&ram_gb=${RAM_GB}&vram_gb=${VRAM_GB}")

echo "Recommended models:"
echo "$MODELS" | jq -r '.models[] | "\(.score)\t\(.name)\t\(.fit_level)"'

```

### Python: Integration with Requests

```python
import requests
import os

BASE = os.getenv("LLMFIT_URL", "http://localhost:8080/api/v1")

class LlmfitClient:
    def __init__(self, base_url: str = BASE):
        self.base = base_url
        self.session = requests.Session()

    def system_specs(self, **overrides) -> dict:
        """Get system info with optional hardware overrides."""
        resp = self.session.get(f"{self.base}/system", params=overrides)
        resp.raise_for_status()
        return resp.json()["system"]

    def best_models(
        self,
        use_case: str = "chat",
        limit: int = 5,
        **hardware
    ) -> list[dict]:
        """Return top-scoring models matching constraints."""
        params = {"use_case": use_case, "limit": limit, **hardware}
        resp = self.session.get(f"{self.base}/models/top", params=params)
        resp.raise_for_status()
        return resp.json()["models"]

    def deployment_plan(self, model: str, **kwargs) -> dict:
        """Generate a deployment plan for automation."""
        payload = {"model": model, **kwargs}
        resp = self.session.post(f"{self.base}/plan", json=payload)
        resp.raise_for_status()
        return resp.json()["plan"]

# Usage

client = LlmfitClient()

# Simulate cloud instance specs

specs = client.system_specs(ram_gb=32, vram_gb=0, cpu_cores=8)
print(f"Effective specs: {specs}")

# Find lightweight coding models

candidates = client.best_models(use_case="coding", limit=3, **specs)
for m in candidates:
    print(f"{m['name']}: score={m['score']:.2f}, fit={m['fit_level']}")

# Plan deployment for best candidate

if candidates:
    plan = client.deployment_plan(
        candidates[0]["name"],
        context=4096,
        target_tps=100
    )
    print(f"Recommended quant: {plan['quantization']}")

```

### Ansible: Dynamic Inventory Integration

```yaml

# llmfit_inventory.yml

# Custom Ansible inventory script using llmfit API

- name: Query llmfit for available runtimes
  uri:
    url: "{{ llmfit_host }}/api/v1/runtimes"
    return_content: yes
  register: runtime_check

- name: Filter hosts by LLM capability
  set_fact:
    llm_capable: "{{ runtime_check.json.runtimes | selectattr('available') | list }}"

```

## Key Implementation Files

Understanding these source files helps debug API behavior and extend functionality:

| File | Purpose | Key Symbols |
|------|---------|-------------|
| [`llmfit-tui/src/serve_api.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/serve_api.rs) | Axum router, handlers, `ApiEnvelope` | `build_router`, `run_serve`, `HardwareOverrideQuery` |
| [`llmfit-tui/src/serve_shared.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/serve_shared.rs) | JSON serialization helpers | `system_json`, `fit_to_json` |
| [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs) | `SystemSpecs` struct, detection logic | `SystemSpecs`, hardware override application |
| [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) | Model metadata and parameter formulas | Model definitions, RAM/VRAM calculation |
| [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) | Fit analysis engine | `ModelFit`, scoring algorithm |
| [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs) | Runtime provider definitions | Provider enum, availability detection |

## Performance and Reliability Considerations

- **Stateless design**: Safe to restart, scale horizontally, or run behind reverse proxies
- **No authentication**: Deploy behind VPN or add proxy auth for exposed instances
- **Default port**: 8080; override via environment or CLI flags in custom builds
- **Response size**: Model list responses can be large (~100KB); use `limit` and `offset` for pagination
- **Hardware override overhead**: Negligible; overrides bypass physical detection entirely

## Summary

- Run `llmfit serve` to start the JSON API on port 8080
- All endpoints accept `ram_gb`, `vram_gb`, `cpu_cores` to simulate hardware without physical changes
- Use `/api/v1/models/top` for "best N" queries and `/api/v1/plan` for deployment estimation
- The `ApiEnvelope` wrapper provides consistent metadata across all responses
- Core implementation lives in [`serve_api.rs`](https://github.com/AlexsJones/llmfit/blob/main/serve_api.rs) with serialization helpers in [`serve_shared.rs`](https://github.com/AlexsJones/llmfit/blob/main/serve_shared.rs)
- Python, shell, and infrastructure‑as‑code tools integrate via standard HTTP/JSON

## Frequently Asked Questions

### What port does llmfit serve run on by default?

The server listens on **0.0.0.0:8080** by default when started with `llmfit serve`. You can modify this by setting environment variables or adjusting the source in `run_serve` before compiling a custom build.

### Can I query the API without starting the server manually?

No—the `llmfit serve` subcommand or direct `run_serve` invocation is required. The API is not embedded in the standard CLI operations; it runs as a separate asynchronous HTTP server using Axum.

### How do hardware overrides affect the response?

Query parameters `ram_gb`, `vram_gb`, and `cpu_cores` override auto‑detected values via `HardwareOverrideQuery` (lines 56‑63). The `effective_specs` logic (lines 31‑65) substitutes these into `SystemSpecs`, causing all downstream fit calculations to use simulated hardware. This enables capacity planning for machines you do not yet possess.

### Is the API suitable for production automation?

The API is intentionally simple and stateless, making it suitable for CI/CD pipelines and internal tooling. However, it ships **without built‑in authentication or rate limiting**—deploy it behind a reverse proxy (nginx, Traefik, cloud load balancer) with appropriate security controls for production exposure.