# How to Set Up llmfit with llama.cpp: Complete Integration Guide

> Easily set up llmfit with llama.cpp using our complete integration guide. Discover how to drive recommendations with the --force-runtime llamacpp flag for seamless performance.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: how-to-guide
- Published: 2026-08-19

---

**You can set up llmfit with llama.cpp by installing the llama-cli or llama-server binaries on your PATH, or by running llama-server on port 8080 (configurable via LLAMA_SERVER_PORT), then using the --force-runtime llamacpp flag to drive recommendations.**

Setting up llmfit with llama.cpp enables hardware-aware model recommendations and direct GGUF inference without requiring Ollama or cloud APIs. The integration supports both local binary execution and HTTP server modes, automatically detecting your environment through the LlamaCppProvider implementation in the llmfit-core crate.

## Understanding llmfit's llama.cpp Integration Modes

The integration operates through two distinct detection strategies implemented in [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs) (lines 860-900).

### Static Binary Mode

When `llama-cli` or `llama-server` exists on your `$PATH`, llmfit immediately binds to these executables. The `LlamaCppProvider::default()` method calls `find_binary("llama-cli")` and `find_binary("llama-server")` to locate the binaries during initialization. This mode provides the lowest latency for model execution since llmfit spawns processes directly without HTTP overhead.

### Server-Probe Mode

If no binaries are detected, llmfit falls back to `probe_llama_server`, attempting to connect to an existing llama-server instance on the default port 8080. You can customize the port by setting the `LLAMA_SERVER_PORT` environment variable before running llmfit commands. This mode is ideal for remote or containerized deployments where the server runs independently of the llmfit client.

## Installing llama.cpp for llmfit

Before configuring detection, you must obtain the llama.cpp binaries compatible with your hardware.

### macOS and Linux via Homebrew

The fastest installation method uses the official Homebrew formula:

```bash
brew install llama.cpp

```

After installation, verify the binaries are accessible:

```bash
which llama-cli
which llama-server

```

### Building from Source

For specific hardware optimizations (CUDA, Metal, Vulkan), compile from the official repository:

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make

```

Move the resulting `llama-cli` and `llama-server` binaries to a directory on your `$PATH`, such as `/usr/local/bin/` or `~/.local/bin/`.

## Configuring llmfit to Detect llama.cpp

Once installed, llmfit automatically discovers the runtime through the detection logic in `LlamaCppProvider`.

### Binary Discovery Process

During provider initialization, llmfit executes the following sequence defined in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs):

1. Constructs the model cache directory via `llamacpp_models_dir()` (default: `~/.cache/llmfit/models/`)
2. Searches for `llama-cli` using the internal `find_binary` utility
3. Searches for `llama-server` as a fallback
4. Stores discovered paths for subsequent command execution

You can verify detection by running:

```bash
llmfit doctor

```

Look for the line `llama.cpp: ✓` in the TUI output, which indicates successful provider initialization (see rendering logic in [`llmfit-tui/src/tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_ui.rs), lines 178-185).

### Server Health Probe Configuration

When running llama-server manually, ensure the port matches llmfit's expectations:

```bash
export LLAMA_SERVER_PORT=8080  # Change if using non-default port

llama-server -m ~/.cache/llmfit/models/your-model.gguf -c 2048 &

```

The `probe_llama_server` function checks connectivity before marking the provider as available.

## Running LLM Recommendations with llama.cpp

With the provider configured, you can force llmfit to use llama.cpp for model analysis and recommendations.

### Forcing llama.cpp Runtime

To bypass automatic runtime selection and explicitly use the llama.cpp backend, use the `--force-runtime` flag defined in [`llmfit-tui/src/main.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/main.rs) (lines 30-65):

```bash
llmfit recommend --force-runtime llamacpp

```

This command routes all inference through the `LlamaCppProvider` regardless of platform defaults.

### Generating Ready-to-Run Commands

For automation and scripting, export the exact command strings needed to run recommended models:

```bash
llmfit recommend --output-llamacpp

```

This adds a `llamacpp_command` field to the JSON output, generated by `display::gen_llamacpp_command` in [`llmfit-tui/src/display.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) (lines 640-660), allowing you to pipe the output directly into shell execution scripts.

## Managing GGUF Models

The llmfit workflow includes integrated GGUF downloading optimized for llama.cpp consumption.

### Downloading Models

Use the download subcommand to pull quantized models from Hugging Face repositories:

```bash
llmfit download <repo-id-or-search-term>          # Pulls best-fit GGUF

llmfit download <repo-id> --quant Q4_K_M         # Explicit quantization

llmfit download <repo-id> --list                  # List available GGUF files

```

The download logic lives in `LlamaCppProvider::start_pull` ([`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs), lines 845-855), which selects appropriate file variants based on your hardware profile and the requested quantization level.

### Model Cache Location

Downloaded files persist in the directory returned by `llamacpp_models_dir()`, defaulting to `~/.cache/llmfit/models/`. Both the `--output-llamacpp` generator and the `run` subcommand reference this location when constructing file paths.

## Interactive Model Execution

After downloading, execute models directly through llmfit's abstraction layer or the underlying binaries.

### Running Models Locally

The `Run` subcommand (defined in [`llmfit-tui/src/main.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/main.rs), lines 880-910) validates binary presence and launches interactive chat:

```bash
llmfit run <model-name-or-path>

```

This command verifies that `llama-cli` is present in the detected provider, then constructs the appropriate arguments for context size and model path before spawning the process.

### Server Mode Operation

To use llama-server instead of the CLI for persistent inference:

```bash
llmfit run <model-name-or-path> --server

```

This flag switches the execution target to `llama-server`, maintaining the connection to the existing server process rather than spawning a new CLI instance.

## Summary

- **Dual detection strategy**: llmfit finds llama.cpp via `$PATH` binaries or server health probes on port 8080 (configurable via `LLAMA_SERVER_PORT`)
- **Core implementation**: The `LlamaCppProvider` in [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs) handles all discovery, downloading, and execution logic
- **Force runtime**: Use `--force-runtime llamacpp` to bypass automatic selection and explicitly use llama.cpp for recommendations
- **Model management**: The `download` command stores GGUF files in `~/.cache/llmfit/models/` with automatic quantization selection
- **Command generation**: The `--output-llamacpp` flag produces ready-to-execute command strings via `gen_llamacpp_command` in [`display.rs`](https://github.com/AlexsJones/llmfit/blob/main/display.rs)
- **Verification**: Run `llmfit doctor` to confirm the TUI shows `llama.cpp: ✓` indicating successful provider initialization

## Frequently Asked Questions

### How does llmfit choose between llama-cli and llama-server?

llmfit prefers `llama-cli` for direct execution and `llama-server` when the `--server` flag is passed to the `run` command. The detection occurs in `LlamaCppProvider::default()` in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs), which first attempts to locate `llama-cli` via `find_binary()`, then falls back to `llama-server`. If neither binary exists on `$PATH`, it attempts to connect to a running server via `probe_llama_server` on the configured port.

### Can I use llmfit with llama.cpp if the binaries are not on my PATH?

Yes. If `find_binary` fails to locate the executables, llmfit enters server-probe mode. Start `llama-server` manually with your model loaded, ensure it is accessible on the default port 8080 (or set `LLAMA_SERVER_PORT` to your custom port), and llmfit will communicate via HTTP. This is useful for Docker deployments or remote inference hosts.

### Where does llmfit store downloaded GGUF models?

All downloaded models are cached in the directory returned by `llamacpp_models_dir()`, which defaults to `~/.cache/llmfit/models/`. The `LlamaCppProvider::start_pull` method in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs) (lines 845-855) manages these downloads, organizing files by repository ID and quantization level for easy retrieval by the `run` command.

### How do I verify that llmfit has successfully detected my llama.cpp installation?

Run `llmfit doctor` or start any recommendation with `llmfit recommend`. The TUI status bar (rendered in [`llmfit-tui/src/tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_ui.rs), lines 178-185) displays `llama.cpp: ✓` when the provider successfully initializes. If detection fails, check that `llama-cli` is accessible in your shell `$PATH` or that your `llama-server` is responding on the correct port specified by `LLAMA_SERVER_PORT`.