# How llmfit Integrates with Ollama, llama.cpp, MLX, and vLLM Runtimes

> Discover how llmfit unifies Ollama, llama.cpp, MLX, and vLLM runtimes with a common ModelProvider trait. Seamlessly integrate local LLM inference across multiple backends.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-20

---

**llmfit unifies four local LLM inference backends—Ollama, llama.cpp, MLX, and vLLM—through a common `ModelProvider` trait that handles detection, model listing, and pulling across all runtimes.**

The [llmfit](https://github.com/AlexsJones/llmfit) project eliminates the fragmentation of local AI inference by providing a single interface for discovering, downloading, and running models across the most popular open-source runtimes. Whether you're on Apple Silicon with MLX, running quantized GGUF files through llama.cpp, or serving production workloads with vLLM, llmfit abstracts the differences into a consistent CLI and TUI experience.

## The ModelProvider Trait: A Unified Interface

At the heart of llmfit's runtime integration sits the `ModelProvider` trait defined in [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs). Every runtime provider implements three core capabilities required by the CLI and TUI:

- **Availability detection** – Probes the runtime through HTTP APIs, local binaries, or cache inspection
- **Model listing** – Returns normalized "stems" of installable models (e.g., `llama3.1:8b`)
- **Pull/download initiation** – Starts background threads that stream `PullEvent` progress

This abstraction allows [`llmfit-tui/src/main.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/main.rs) to parse flags like `--runtime ollama` or `--runtime vllm` and instantiate the appropriate concrete provider without duplicating logic.

## Ollama Integration

The `OllamaProvider` connects to the Ollama daemon through its HTTP API, with automatic fallback from `localhost` to `127.0.0.1` if needed.

### Detection and Model Discovery

The `detect_with_installed` method at line 85 in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs) queries Ollama's `/api/tags` endpoint to build a `HashSet<String>` of installed model stems. These are normalized to lowercase `<model>:<size>` format.

### Pulling Models

The `start_pull` implementation (lines 89-151) POSTs to `/api/pull` and streams JSON status lines for real-time progress updates. Cloud-hosted models receive a `-cloud` tag and are excluded from the installed set via `OllamaModel::is_cloud`.

```bash

# Pull a model with streaming progress

$ llmfit pull llama3.1:8b
Downloading llama3.1:8b…  0%
Downloading llama3.1:8b… 45%
Downloading llm3.1:8b… 100%
✔ Pull complete

```

## llama.cpp Integration

Unlike Ollama's client-server model, `LlamaCppProvider` manages local binaries and direct GGUF file downloads.

### Binary Detection and Health Probes

The `impl Default for LlamaCppProvider` block starting at line 80 searches for `llama-cli` or `llama-server` binaries. If neither is found, it falls back to a health probe against `LLAMA_SERVER_PORT` (default 8080).

### Model Management

Model discovery uses `list_gguf_files` to scan for local `.gguf` files. The `download_gguf` method (lines 1085-1145) constructs HuggingFace URLs and streams downloads directly, including support for sharded GGUF models.

```bash

# Download a specific GGUF quantization directly from HuggingFace

$ llmfit download "bartowski/Llama-3.1-8B-Instruct-GGUF" "Llama-3.1-8B-Instruct-Q4_K_M.gguf"
Downloading 0.5/0.5 GB… 100%
✔ Saved Llama-3.1-8B-Instruct-Q4_K_M.gguf

```

## MLX Integration

The `MlxProvider` targets Apple's MLX framework, with platform-gated availability.

### macOS-Only Runtime

On non-macOS systems, the provider automatically reports as unavailable. Detection occurs through `detect_with_installed` at line 100, which attempts both local HTTP server queries (`MLX_DEFAULT_SERVER_URL` or `OMLX_DEFAULT_SERVER_URL`) and HuggingFace cache scanning via `scan_hf_cache_for_mlx`.

### Model Pulling via HuggingFace CLI

The `start_pull` method (lines 808-851) invokes the `hf` CLI tool to fetch complete repositories, rather than individual files. Model identifiers use HuggingFace's `owner/model` format.

```bash

# Run inference on MLX without explicit pull (model fetched on demand)

$ llmfit run --runtime mlx --model mlx-community/phi-2
Running inference on MLX… (GPU: Apple Silicon)

```

## vLLM Integration

`VllmProvider` handles production-grade inference with important operational constraints.

### Server-Dependent Architecture

Unlike other providers, vLLM **cannot pull models at runtime**. The `start_pull` method (lines 77-82) immediately returns an informational error directing users to pre-load models via `vllm serve <model>`.

### Detection and Model Listing

Detection at line 2639 uses `detect_with_installed` to query the vLLM server, with special handling to distinguish from the *omlx* variant through `endpoint_has_omlx_status`. Model names use full HuggingFace identifiers like `meta-llama/Llama-3.1-8B-Instruct`.

```bash

# Attempting to pull with vLLM fails—models must be pre-loaded

$ llmfit run --runtime vllm --model facebook/opt-125m-awq
Error: vLLM does not support downloading models at runtime.

# Correct workflow: start server first, then run

$ vllm serve facebook/opt-125m-awq

```

## Runtime Selection and Hardware Fitting

The [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) module contains the `RunMode` enum and selection logic that determines which runtime suits a given model and hardware configuration. Key `RunMode` variants include:

- **Gpu** – Full GPU acceleration
- **CpuOffload** – Partial CPU fallback for large models
- **TensorParallel** – Multi-GPU distribution

The selection logic references each provider's `installed_models()` and `is_available()` methods. Pre-quantized AWQ/GPTQ models are forced to vLLM when no GGUF conversion exists.

## Benchmarking Across Runtimes

[`llmfit-core/src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/bench.rs) provides unified benchmarking through OpenAI-compatible endpoints. All compliant runtimes—Ollama, MLX, and vLLM—accept requests to `/v1/chat/completions`, enabling direct performance comparisons.

## User Interface Integration

The TUI in [`llmfit-tui/src/tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_ui.rs) (line 226) displays runtime status with visual indicators:

```bash
$ llmfit system
Ollama:    ✓ (42 models)
MLX:       ✗
llama.cpp: ✓ (7 models)
vLLM:      ✓ (12 models)

```

The `App` struct in [`tui_app.rs`](https://github.com/AlexsJones/llmfit/blob/main/tui_app.rs) maintains an `installed` aggregator that caches model counts per provider for responsive UI updates.

## Summary

- **Unified trait architecture**: The `ModelProvider` trait in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs) abstracts all four runtimes behind three core methods
- **Ollama**: Full HTTP API integration with streaming pulls and cloud-model filtering
- **llama.cpp**: Binary discovery, health probes, and direct HuggingFace GGUF downloads
- **MLX**: macOS-gated with HuggingFace CLI integration for repository-fetching
- **vLLM**: Server-preload requirement with full HF identifier support and omlx-variant detection
- **Hardware-aware selection**: [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs) matches models to optimal runtimes based on quantization format and available hardware

## Frequently Asked Questions

### What happens if multiple runtimes are installed?

llmfit lists all available runtimes in `llmfit system` output. You can force a specific runtime with `--runtime ollama|mlx|llama.cpp|vllm`, or let [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs) select automatically based on model format and hardware capabilities.

### Why can't I pull models directly into vLLM?

According to the llmfit source code, vLLM's architecture requires models to be loaded at server startup via `vllm serve`. The `VllmProvider::start_pull` implementation intentionally returns an error rather than attempting unsupported runtime behavior.

### How does llmfit handle llama.cpp's optional server mode?

If `llama-cli` and `llama-server` binaries aren't found in PATH, the `LlamaCppProvider` attempts a health probe against `LLAMA_SERVER_PORT` (default 8080). This allows connection to externally managed llama.cpp servers.

### Can I use MLX on non-Apple hardware?

No. The `MlxProvider` automatically reports `is_available() = false` on non-macOS platforms. Detection occurs early in `detect_with_installed`, preventing incompatible operations.