How to Add Support for a New Model Architecture in llama.cpp: A Complete Guide

To add a new model architecture to llama.cpp, you must extend the llm_arch enumeration in src/llama-arch.h, register the architecture name and tensor mapping in src/llama-arch.cpp, parse custom GGUF key-values in src/llama-model.cpp, and implement a graph builder in src/models/<arch>.cpp that constructs the computation graph.

Integrating a custom transformer architecture into llama.cpp requires modifying the core enumeration system, loader logic, and graph construction pipeline. This guide walks through the exact files and functions you need to touch in the ggml-org/llama.cpp repository, providing runnable code examples for a hypothetical "MyArch" implementation.

Overview of the Integration Layers

Adding support for a new model architecture involves three logical layers: identification, loading, and execution. The codebase uses a centralized architecture enum to route logic through switch statements in the loader and graph builder.

• Enumeration layer: Defines a unique symbolic ID (LLM_ARCH_MYARCH) in src/llama-arch.h
• Mapping layer: Connects the string name ("myarch") to the enum and defines expected tensors in src/llama-arch.cpp
• Execution layer: Parses hyperparameters and constructs the computation graph in src/llama-model.cpp and src/models/

Step-by-Step Implementation Guide

1. Extend the Architecture Enumeration

First, add your architecture to the llm_arch enum in src/llama-arch.h. Insert the new entry before LLM_ARCH_UNKNOWN to ensure the compiler assigns a valid numeric ID.

// src/llama-arch.h
enum llm_arch {
    // ... existing architectures ...
    LLM_ARCH_KIMI_LINEAR,
    LLM_ARCH_MYARCH,           // ← add your new architecture here
    LLM_ARCH_UNKNOWN,
};

This symbolic constant acts as the primary key for routing architecture-specific behavior throughout the codebase.

2. Register the Architecture Name

In src/llama-arch.cpp, add the human-readable name to the LLM_ARCH_NAMES map. This enables GGUF files to store "myarch" and allows command-line tools to recognize the architecture via string parsing.

// src/llama-arch.cpp
static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
    // ... existing entries ...
    { LLM_ARCH_KIMI_LINEAR, "kimi-linear" },
    { LLM_ARCH_MYARCH,      "myarch" },      // ← add this line
    { LLM_ARCH_UNKNOWN,     "(unknown)" },
};

3. Define the Tensor Set

Still in src/llama-arch.cpp, extend the llm_get_tensor_names() function to return the set of tensors your architecture expects. This validates GGUF files during loading and ensures all required weights are present.

// src/llama-arch.cpp (function llm_get_tensor_names)
static std::set<llm_tensor> llm_get_tensor_names(llm_arch arch) {
    switch (arch) {
        // ... existing cases ...
        case LLM_ARCH_MYARCH: {
            return {
                LLM_TENSOR_TOKEN_EMBD,
                LLM_TENSOR_OUTPUT_NORM,
                LLM_TENSOR_OUTPUT,
                LLM_TENSOR_ROPE_FREQS,
                LLM_TENSOR_ATTN_NORM,
                LLM_TENSOR_ATTN_Q,
                LLM_TENSOR_ATTN_K,
                LLM_TENSOR_ATTN_V,
                LLM_TENSOR_ATTN_OUT,
                LLM_TENSOR_FFN_NORM,
                LLM_TENSOR_FFN_DOWN,
                LLM_TENSOR_FFN_UP,
                // Add custom tensors here (e.g., MoE routing gates)
            };
        } break;
        // ...
    }
}

Include only the tensors your model actually uses. Standard transformer architectures typically reuse the attention and FFN tensor groups shown above.

4. Parse Architecture-Specific Parameters

In src/llama-model.cpp, locate the switch (arch) statement (around line 440) and add a case for LLM_ARCH_MYARCH. This is where you read custom GGUF key-value pairs and set model defaults.

// src/llama-model.cpp
switch (arch) {
    // ... existing cases ...
    case LLM_ARCH_MYARCH: {
        // Example: MyArch uses RMS normalization
        ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);

        // Read custom KV entries if needed:
        // ml.get_key(LLM_KV_MY_CUSTOM_PARAM, hparams.my_custom, /*required=*/false);

        // Map layer count to model type (example heuristic)
        switch (hparams.n_layer) {
            case 24: type = LLM_TYPE_7B; break;
            case 48: type = LLM_TYPE_13B; break;
            default: type = LLM_TYPE_UNKNOWN;
        }
    } break;
    // ...
}

Handle any unique hyperparameters your architecture requires, such as custom rotary embedding configurations or Mixture-of-Experts (MoE) settings.

5. Implement the Graph Builder

Create a new file src/models/myarch.cpp (and optional header) that implements the computation graph. Follow the pattern of existing builders like llm_build_llama or llm_build_falcon.

// src/models/myarch.cpp
#include "models.h"

//
// MyArch graph builder - inherits standard Llama implementation
// Override methods here for custom attention or FFN logic
//
struct llm_build_myarch : public llm_build_llama<false> {
    llm_build_myarch(const llama_model & model, const llm_graph_params & params)
        : llm_build_llama<false>(model, params) {}

    // Optional: override build_attn() for custom attention patterns
    // Optional: override build_ffn() for custom feed-forward networks
};

Inherit from an existing builder if your architecture is a variant of standard transformers. For radically different architectures, inherit from llm_graph and implement the virtual build() method directly.

6. Register the Builder

Return to src/llama-model.cpp and register your builder in the instantiation section (around line 860). This connects the architecture enum to the actual graph construction logic.

// src/llama-model.cpp (graph instantiation)
switch (arch) {
    // ... existing cases ...
    case LLM_ARCH_MYARCH:
        graph = std::make_unique<llm_build_myarch>(model, params);
        break;
    // ...
}

7. Optional: Quantization and Tokenizer Support

If your architecture requires custom quantization logic (e.g., non-standard weight layouts), modify src/llama-quant.cpp:

// src/llama-quant.cpp
if (arch == LLM_ARCH_MYARCH && nx % qk_k != 0) {
    // Custom quantization handling
}

For custom tokenizers, extend src/llama-tokenizer.cpp and set the LLM_KV_TOKENIZER_MODEL key in your GGUF file to trigger the appropriate parsing logic.

Complete Code Example

Below is a unified diff showing the minimal changes required to add a functional "MyArch" architecture that behaves like Llama-2 but is selectable via "myarch" in GGUF metadata:

diff --git a/src/llama-arch.h b/src/llama-arch.h
index 7f8c9e2..e3b1a45 100644
--- a/src/llama-arch.h
+++ b/src/llama-arch.h
@@ -30,6 +30,7 @@ enum llm_arch {
     LLM_ARCH_KIMI_LINEAR,
+    LLM_ARCH_MYARCH,
     LLM_ARCH_UNKNOWN,
 };
 
diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
index 1a2c3d4..5e6f7b8 100644
--- a/src/llama-arch.cpp
+++ b/src/llama-arch.cpp
@@ -15,6 +15,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
     { LLM_ARCH_KIMI_LINEAR, "kimi-linear" },
+    { LLM_ARCH_MYARCH,      "myarch" },
     { LLM_ARCH_UNKNOWN,     "(unknown)" },
 };
@@ -450,6 +451,20 @@ static std::set<llm_tensor> llm_get_tensor_names(llm_arch arch) {
         } break;
+        case LLM_ARCH_MYARCH: {
+            return {
+                LLM_TENSOR_TOKEN_EMBD,
+                LLM_TENSOR_OUTPUT_NORM,
+                LLM_TENSOR_OUTPUT,
+                LLM_TENSOR_ATTN_NORM,
+                LLM_TENSOR_ATTN_Q,
+                LLM_TENSOR_ATTN_K,
+                LLM_TENSOR_ATTN_V,
+                LLM_TENSOR_ATTN_OUT,
+                LLM_TENSOR_FFN_NORM,
+                LLM_TENSOR_FFN_DOWN,
+                LLM_TENSOR_FFN_UP,
+            };
+        } break;
         case LLM_ARCH_UNKNOWN:

// src/models/myarch.cpp
#include "models.h"

struct llm_build_myarch : public llm_build_llama<false> {
    llm_build_myarch(const llama_model & model, const llm_graph_params & params)
        : llm_build_llama<false>(model, params) {}
};

After compiling with make, load your model using a GGUF file containing general.architecture = "myarch".

Key Files Reference

• src/llama-arch.h: Declares the llm_arch enumeration
• src/llama-arch.cpp: Maps enums to strings (LLM_ARCH_NAMES) and defines tensor sets (llm_get_tensor_names)
• src/llama-model.cpp: Parses GGUF KV pairs and instantiates graph builders via switch (arch) blocks
• src/models/myarch.cpp: Implements the llm_build_myarch class for graph construction
• src/llama-quant.cpp: Optional custom quantization logic for the new architecture
• src/llama-tokenizer.cpp: Optional custom tokenizer implementation

Summary

• Enumeration: Add LLM_ARCH_MYARCH to src/llama-arch.h to create a unique architecture ID
• Registration: Map the string name in src/llama-arch.cpp via LLM_ARCH_NAMES and define expected tensors in llm_get_tensor_names()
• Loading: Handle custom GGUF key-values in the switch (arch) block within src/llama-model.cpp
• Execution: Implement llm_build_myarch in src/models/myarch.cpp and register it in src/llama-model.cpp
• Extensions: Add quantization rules in src/llama-quant.cpp and tokenizer support in src/llama-tokenizer.cpp if your architecture requires them

Frequently Asked Questions

How do I handle architectures with custom attention mechanisms?

Implement a custom graph builder class in src/models/myarch.cpp that inherits from llm_build_llama or the base llm_graph class. Override the build_attn() method to inject your custom attention logic while reusing existing tensor management and memory allocation infrastructure.

Can I support quantization for my new architecture immediately?

Yes, but you must verify that your tensor layouts align with existing quantization assumptions. If your architecture uses standard tensor names and shapes, llama-quant will work automatically. For non-standard layouts, add conditional logic in src/llama-quant.cpp to handle your LLM_ARCH_MYARCH case specifically.

What is the minimum code required to test a new architecture?

You need four changes: (1) add the enum value in src/llama-arch.h, (2) add the name mapping in src/llama-arch.cpp, (3) return a tensor set in llm_get_tensor_names(), and (4) instantiate a builder (even a stub) in src/llama-model.cpp. This allows the loader to recognize GGUF files with your architecture string and attempt graph construction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}