How to Set Up llmfit with llama.cpp: Complete Integration Guide

You can set up llmfit with llama.cpp by installing the llama-cli or llama-server binaries on your PATH, or by running llama-server on port 8080 (configurable via LLAMA_SERVER_PORT), then using the --force-runtime llamacpp flag to drive recommendations.

Setting up llmfit with llama.cpp enables hardware-aware model recommendations and direct GGUF inference without requiring Ollama or cloud APIs. The integration supports both local binary execution and HTTP server modes, automatically detecting your environment through the LlamaCppProvider implementation in the llmfit-core crate.

Understanding llmfit's llama.cpp Integration Modes

The integration operates through two distinct detection strategies implemented in llmfit-core/src/providers.rs (lines 860-900).

Static Binary Mode

When llama-cli or llama-server exists on your $PATH, llmfit immediately binds to these executables. The LlamaCppProvider::default() method calls find_binary("llama-cli") and find_binary("llama-server") to locate the binaries during initialization. This mode provides the lowest latency for model execution since llmfit spawns processes directly without HTTP overhead.

Server-Probe Mode

If no binaries are detected, llmfit falls back to probe_llama_server, attempting to connect to an existing llama-server instance on the default port 8080. You can customize the port by setting the LLAMA_SERVER_PORT environment variable before running llmfit commands. This mode is ideal for remote or containerized deployments where the server runs independently of the llmfit client.

Installing llama.cpp for llmfit

Before configuring detection, you must obtain the llama.cpp binaries compatible with your hardware.

macOS and Linux via Homebrew

The fastest installation method uses the official Homebrew formula:

brew install llama.cpp

After installation, verify the binaries are accessible:

which llama-cli
which llama-server

Building from Source

For specific hardware optimizations (CUDA, Metal, Vulkan), compile from the official repository:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make

Move the resulting llama-cli and llama-server binaries to a directory on your $PATH, such as /usr/local/bin/ or ~/.local/bin/.

Configuring llmfit to Detect llama.cpp

Once installed, llmfit automatically discovers the runtime through the detection logic in LlamaCppProvider.

Binary Discovery Process

During provider initialization, llmfit executes the following sequence defined in providers.rs:

1. Constructs the model cache directory via llamacpp_models_dir() (default: ~/.cache/llmfit/models/)
2. Searches for llama-cli using the internal find_binary utility
3. Searches for llama-server as a fallback
4. Stores discovered paths for subsequent command execution

You can verify detection by running:

llmfit doctor

Look for the line llama.cpp: ✓ in the TUI output, which indicates successful provider initialization (see rendering logic in llmfit-tui/src/tui_ui.rs, lines 178-185).

Server Health Probe Configuration

When running llama-server manually, ensure the port matches llmfit's expectations:

export LLAMA_SERVER_PORT=8080  # Change if using non-default port

llama-server -m ~/.cache/llmfit/models/your-model.gguf -c 2048 &

The probe_llama_server function checks connectivity before marking the provider as available.

Running LLM Recommendations with llama.cpp

With the provider configured, you can force llmfit to use llama.cpp for model analysis and recommendations.

Forcing llama.cpp Runtime

To bypass automatic runtime selection and explicitly use the llama.cpp backend, use the --force-runtime flag defined in llmfit-tui/src/main.rs (lines 30-65):

llmfit recommend --force-runtime llamacpp

This command routes all inference through the LlamaCppProvider regardless of platform defaults.

Generating Ready-to-Run Commands

For automation and scripting, export the exact command strings needed to run recommended models:

llmfit recommend --output-llamacpp

This adds a llamacpp_command field to the JSON output, generated by display::gen_llamacpp_command in llmfit-tui/src/display.rs (lines 640-660), allowing you to pipe the output directly into shell execution scripts.

Managing GGUF Models

The llmfit workflow includes integrated GGUF downloading optimized for llama.cpp consumption.

Downloading Models

Use the download subcommand to pull quantized models from Hugging Face repositories:

llmfit download <repo-id-or-search-term>          # Pulls best-fit GGUF

llmfit download <repo-id> --quant Q4_K_M         # Explicit quantization

llmfit download <repo-id> --list                  # List available GGUF files

The download logic lives in LlamaCppProvider::start_pull (providers.rs, lines 845-855), which selects appropriate file variants based on your hardware profile and the requested quantization level.

Model Cache Location

Downloaded files persist in the directory returned by llamacpp_models_dir(), defaulting to ~/.cache/llmfit/models/. Both the --output-llamacpp generator and the run subcommand reference this location when constructing file paths.

Interactive Model Execution

After downloading, execute models directly through llmfit's abstraction layer or the underlying binaries.

Running Models Locally

The Run subcommand (defined in llmfit-tui/src/main.rs, lines 880-910) validates binary presence and launches interactive chat:

llmfit run <model-name-or-path>

This command verifies that llama-cli is present in the detected provider, then constructs the appropriate arguments for context size and model path before spawning the process.

Server Mode Operation

To use llama-server instead of the CLI for persistent inference:

llmfit run <model-name-or-path> --server

This flag switches the execution target to llama-server, maintaining the connection to the existing server process rather than spawning a new CLI instance.

Summary

• Dual detection strategy: llmfit finds llama.cpp via $PATH binaries or server health probes on port 8080 (configurable via LLAMA_SERVER_PORT)
• Core implementation: The LlamaCppProvider in llmfit-core/src/providers.rs handles all discovery, downloading, and execution logic
• Force runtime: Use --force-runtime llamacpp to bypass automatic selection and explicitly use llama.cpp for recommendations
• Model management: The download command stores GGUF files in ~/.cache/llmfit/models/ with automatic quantization selection
• Command generation: The --output-llamacpp flag produces ready-to-execute command strings via gen_llamacpp_command in display.rs
• Verification: Run llmfit doctor to confirm the TUI shows llama.cpp: ✓ indicating successful provider initialization

Frequently Asked Questions

How does llmfit choose between llama-cli and llama-server?

llmfit prefers llama-cli for direct execution and llama-server when the --server flag is passed to the run command. The detection occurs in LlamaCppProvider::default() in providers.rs, which first attempts to locate llama-cli via find_binary(), then falls back to llama-server. If neither binary exists on $PATH, it attempts to connect to a running server via probe_llama_server on the configured port.

Can I use llmfit with llama.cpp if the binaries are not on my PATH?

Yes. If find_binary fails to locate the executables, llmfit enters server-probe mode. Start llama-server manually with your model loaded, ensure it is accessible on the default port 8080 (or set LLAMA_SERVER_PORT to your custom port), and llmfit will communicate via HTTP. This is useful for Docker deployments or remote inference hosts.

Where does llmfit store downloaded GGUF models?

All downloaded models are cached in the directory returned by llamacpp_models_dir(), which defaults to ~/.cache/llmfit/models/. The LlamaCppProvider::start_pull method in providers.rs (lines 845-855) manages these downloads, organizing files by repository ID and quantization level for easy retrieval by the run command.

How do I verify that llmfit has successfully detected my llama.cpp installation?

Run llmfit doctor or start any recommendation with llmfit recommend. The TUI status bar (rendered in llmfit-tui/src/tui_ui.rs, lines 178-185) displays llama.cpp: ✓ when the provider successfully initializes. If detection fails, check that llama-cli is accessible in your shell $PATH or that your llama-server is responding on the correct port specified by LLAMA_SERVER_PORT.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}