How llmfit Integrates with Ollama, llama.cpp, MLX, and vLLM Runtimes

llmfit unifies four local LLM inference backends—Ollama, llama.cpp, MLX, and vLLM—through a common ModelProvider trait that handles detection, model listing, and pulling across all runtimes.

The llmfit project eliminates the fragmentation of local AI inference by providing a single interface for discovering, downloading, and running models across the most popular open-source runtimes. Whether you're on Apple Silicon with MLX, running quantized GGUF files through llama.cpp, or serving production workloads with vLLM, llmfit abstracts the differences into a consistent CLI and TUI experience.

The ModelProvider Trait: A Unified Interface

At the heart of llmfit's runtime integration sits the ModelProvider trait defined in llmfit-core/src/providers.rs. Every runtime provider implements three core capabilities required by the CLI and TUI:

• Availability detection – Probes the runtime through HTTP APIs, local binaries, or cache inspection
• Model listing – Returns normalized "stems" of installable models (e.g., llama3.1:8b)
• Pull/download initiation – Starts background threads that stream PullEvent progress

This abstraction allows llmfit-tui/src/main.rs to parse flags like --runtime ollama or --runtime vllm and instantiate the appropriate concrete provider without duplicating logic.

Ollama Integration

The OllamaProvider connects to the Ollama daemon through its HTTP API, with automatic fallback from localhost to 127.0.0.1 if needed.

Detection and Model Discovery

The detect_with_installed method at line 85 in providers.rs queries Ollama's /api/tags endpoint to build a HashSet<String> of installed model stems. These are normalized to lowercase <model>:<size> format.

Pulling Models

The start_pull implementation (lines 89-151) POSTs to /api/pull and streams JSON status lines for real-time progress updates. Cloud-hosted models receive a -cloud tag and are excluded from the installed set via OllamaModel::is_cloud.

# Pull a model with streaming progress

$ llmfit pull llama3.1:8b
Downloading llama3.1:8b…  0%
Downloading llama3.1:8b… 45%
Downloading llm3.1:8b… 100%
✔ Pull complete

llama.cpp Integration

Unlike Ollama's client-server model, LlamaCppProvider manages local binaries and direct GGUF file downloads.

Binary Detection and Health Probes

The impl Default for LlamaCppProvider block starting at line 80 searches for llama-cli or llama-server binaries. If neither is found, it falls back to a health probe against LLAMA_SERVER_PORT (default 8080).

Model Management

Model discovery uses list_gguf_files to scan for local .gguf files. The download_gguf method (lines 1085-1145) constructs HuggingFace URLs and streams downloads directly, including support for sharded GGUF models.

# Download a specific GGUF quantization directly from HuggingFace

$ llmfit download "bartowski/Llama-3.1-8B-Instruct-GGUF" "Llama-3.1-8B-Instruct-Q4_K_M.gguf"
Downloading 0.5/0.5 GB… 100%
✔ Saved Llama-3.1-8B-Instruct-Q4_K_M.gguf

MLX Integration

The MlxProvider targets Apple's MLX framework, with platform-gated availability.

macOS-Only Runtime

On non-macOS systems, the provider automatically reports as unavailable. Detection occurs through detect_with_installed at line 100, which attempts both local HTTP server queries (MLX_DEFAULT_SERVER_URL or OMLX_DEFAULT_SERVER_URL) and HuggingFace cache scanning via scan_hf_cache_for_mlx.

Model Pulling via HuggingFace CLI

The start_pull method (lines 808-851) invokes the hf CLI tool to fetch complete repositories, rather than individual files. Model identifiers use HuggingFace's owner/model format.

# Run inference on MLX without explicit pull (model fetched on demand)

$ llmfit run --runtime mlx --model mlx-community/phi-2
Running inference on MLX… (GPU: Apple Silicon)

vLLM Integration

VllmProvider handles production-grade inference with important operational constraints.

Server-Dependent Architecture

Unlike other providers, vLLM cannot pull models at runtime. The start_pull method (lines 77-82) immediately returns an informational error directing users to pre-load models via vllm serve <model>.

Detection and Model Listing

Detection at line 2639 uses detect_with_installed to query the vLLM server, with special handling to distinguish from the omlx variant through endpoint_has_omlx_status. Model names use full HuggingFace identifiers like meta-llama/Llama-3.1-8B-Instruct.

# Attempting to pull with vLLM fails—models must be pre-loaded

$ llmfit run --runtime vllm --model facebook/opt-125m-awq
Error: vLLM does not support downloading models at runtime.

# Correct workflow: start server first, then run

$ vllm serve facebook/opt-125m-awq

Runtime Selection and Hardware Fitting

The llmfit-core/src/fit.rs module contains the RunMode enum and selection logic that determines which runtime suits a given model and hardware configuration. Key RunMode variants include:

• Gpu – Full GPU acceleration
• CpuOffload – Partial CPU fallback for large models
• TensorParallel – Multi-GPU distribution

The selection logic references each provider's installed_models() and is_available() methods. Pre-quantized AWQ/GPTQ models are forced to vLLM when no GGUF conversion exists.

Benchmarking Across Runtimes

llmfit-core/src/bench.rs provides unified benchmarking through OpenAI-compatible endpoints. All compliant runtimes—Ollama, MLX, and vLLM—accept requests to /v1/chat/completions, enabling direct performance comparisons.

User Interface Integration

The TUI in llmfit-tui/src/tui_ui.rs (line 226) displays runtime status with visual indicators:

$ llmfit system
Ollama:    ✓ (42 models)
MLX:       ✗
llama.cpp: ✓ (7 models)
vLLM:      ✓ (12 models)

The App struct in tui_app.rs maintains an installed aggregator that caches model counts per provider for responsive UI updates.

Summary

• Unified trait architecture: The ModelProvider trait in providers.rs abstracts all four runtimes behind three core methods
• Ollama: Full HTTP API integration with streaming pulls and cloud-model filtering
• llama.cpp: Binary discovery, health probes, and direct HuggingFace GGUF downloads
• MLX: macOS-gated with HuggingFace CLI integration for repository-fetching
• vLLM: Server-preload requirement with full HF identifier support and omlx-variant detection
• Hardware-aware selection: fit.rs matches models to optimal runtimes based on quantization format and available hardware

Frequently Asked Questions

What happens if multiple runtimes are installed?

llmfit lists all available runtimes in llmfit system output. You can force a specific runtime with --runtime ollama|mlx|llama.cpp|vllm, or let fit.rs select automatically based on model format and hardware capabilities.

Why can't I pull models directly into vLLM?

According to the llmfit source code, vLLM's architecture requires models to be loaded at server startup via vllm serve. The VllmProvider::start_pull implementation intentionally returns an error rather than attempting unsupported runtime behavior.

How does llmfit handle llama.cpp's optional server mode?

If llama-cli and llama-server binaries aren't found in PATH, the LlamaCppProvider attempts a health probe against LLAMA_SERVER_PORT (default 8080). This allows connection to externally managed llama.cpp servers.

Can I use MLX on non-Apple hardware?

No. The MlxProvider automatically reports is_available() = false on non-macOS platforms. Detection occurs early in detect_with_installed, preventing incompatible operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}