GPULlama3.java Model Architectures: Supported LLMs Beyond Llama 3 (Mistral, Qwen, Phi-3, Granite)

GPULlama3.java supports seven distinct transformer architectures—including Mistral, Qwen 2/3, Phi-3, Granite, and DeepSeek-R1-Distill-Qwen—through a unified ModelType enum that delegates to family-specific loaders, tokenizers, and optimized inference cores.

GPULlama3.java from beehive-lab/gpullama3.java implements a model-agnostic inference engine designed to run multiple large language model families on both CPU and GPU. While the repository name references Llama 3, the source code reveals a pluggable architecture where each supported model family is encapsulated in dedicated classes for configuration, tokenization, and weight loading. This design allows developers to switch between architectures like Mistral or Granite without changing the high-level inference API.

Supported Model Architectures in GPULlama3.java

The central enumeration ModelType in src/main/java/org/beehive/gpullama3/model/ModelType.java defines all supported model families. Each enum constant implements a loadModel(...) method that instantiates the appropriate Model, Configuration, Tokenizer, and Loader classes for that architecture.

The following model families are fully implemented:

• Llama 3 (LLAMA_3): The reference implementation using Llama, LlamaConfiguration, and LlamaTokenizer. Uses standard RoPE positional encoding.
• Mistral (MISTRAL): Shares the transformer stack with Llama but employs distinct attention scaling and the MistralTokenizer for processing.
• Qwen 2 (QWEN_2): Supports the Qwen-2 series via Qwen2 and Qwen2Configuration. Specifically omits the <|beginoftext|> token and re-uses Qwen3Tokenizer for encoding.
• Qwen 3 (QWEN_3): Dedicated Qwen3 implementation mirroring Qwen-2 logic but with separate configuration handling and the Qwen3Tokenizer.
• Phi-3 (PHI_3): Uses Phi3 and Phi3Configuration with a specialized forward pass (forwardJavaPhi3) and no begin-of-text token.
• Granite (GRANITE): IBM's Granite architecture implemented in Granite with GraniteConfiguration. Uses a BOS token of 0 (<|end_of_text|>) and applies per-layer scaling factors during the forward pass.
• DeepSeek-R1-Distill-Qwen (DEEPSEEK_R1_DISTILL_QWEN): Treated as a Qwen-2 variant, reusing Qwen2 and Qwen2Configuration but overriding chat-prompt rules for reasoning tasks.

Attempting to load an unsupported architecture returns ModelType.UNKNOWN, which throws an UnsupportedOperationException during initialization.

Architectural Implementation Details

Each model family in GPULlama3.java follows a consistent pattern while retaining architecture-specific optimizations.

Model-Specific Forward Passes

The InferenceCore class provides dedicated forward implementations for each architecture to enable fine-grained optimizations. For example, forwardJavaPhi3 handles the unique attention mechanisms in Phi-3, while forwardGranite applies custom embedding, residual, attention, and logit scaling factors specific to IBM's model. When GPU acceleration is enabled via TornadoVMMasterPlan, the engine selects the appropriate TornadoVM kernel instead of the Java fallback.

Tokenizer and Chat Format Abstraction

Every family ships with a specialized tokenizer class—such as MistralTokenizer, Phi3Tokenizer, or GraniteTokenizer—that parses model-specific special token mappings from GGUF metadata. Chat formatting logic is encapsulated in corresponding classes like GraniteChatFormat and Phi3ChatFormat, selected automatically by ChatFormat.create(...) based on the active model type.

Weight Loading and GGUF Support

Architecture-specific loaders—including MistralModelLoader, Qwen3ModelLoader, and GraniteLoader—parse GGUF files and construct the appropriate Weights implementation. These loaders respect quantization types such as FP16 and Q8_0, and can build both standard Java weight objects or TornadoVM-optimized buffers for GPU execution.

Practical Code Examples

Loading a Mistral Model

The following example demonstrates loading a Mistral model from a GGUF file using the ModelType enum:

// Path to a GGUF model file
Path modelPath = Paths.get("models/mistral-7b-instruct.gguf");

// Open the file channel
try (FileChannel fc = FileChannel.open(modelPath, StandardOpenOption.READ)) {
    // Parse GGUF metadata
    GGUF gguf = GGUF.load(fc);

    // Choose the architecture via the enum
    ModelType type = ModelType.MISTRAL;

    // Load the model (CPU implementation)
    Model mistral = type.loadModel(fc, gguf, /*contextLength*/ 4096, /*useTornadoVM*/ false);

    // Optionally enable GPU acceleration
    // Model mistral = type.loadModel(fc, gguf, 4096, true);
}

This delegates to MistralModelLoader and instantiates the Mistral class defined in src/main/java/org/beehive/gpullama3/model/mistral/Mistral.java.

Running Granite on GPU with TornadoVM

To leverage GPU acceleration for IBM Granite models, pass true to the loadModel method and provide a TornadoVMMasterPlan:

// Instantiate a Granite model (GPU‑enabled)
Model granite = ModelType.GRANITE.loadModel(fc, gguf, 4096, true);

// Create a state (batch size 1)
State state = granite.createNewState();

// Prompt tokenisation (GraniteTokenizer)
List<Integer> prompt = granite.tokenizer().encode("Explain the difference between CPU and GPU.");

// Generate up to 128 tokens, using a top‑p sampler
Sampler sampler = new ToppSampler(0.9f);
List<Integer> output = granite.generateTokensGPU(
        state,
        /*startPosition=*/0,
        prompt,
        Set.of(granite.tokenizer().getSpecialTokens().get("<|end_of_text|>")), // stop token
        128,
        sampler,
        /*echo=*/true,
        token -> System.out.print(granite.tokenizer().decode(List.of(token))),
        new TornadoVMMasterPlan(/*plan config*/));

The generateTokensGPU method utilizes InferenceCore.forwardGranite optimized for TornadoVM execution.

Runtime Model Switching

You can dynamically select architectures at runtime without changing downstream logic:

ModelType[] supported = {
        ModelType.LLAMA_3,
        ModelType.MISTRAL,
        ModelType.QWEN_2,
        ModelType.QWEN_3,
        ModelType.PHI_3,
        ModelType.GRANITE
};

for (ModelType t : supported) {
    Model model = t.loadModel(fc, gguf, 4096, false);
    System.out.println("Loaded model type: " + t);
    // Run a quick sanity check (e.g., token count of a short prompt)
    List<Integer> ids = model.tokenizer().encode("Hello");
    System.out.println("Prompt token IDs: " + ids);
}

Each iteration invokes the specific loadModel implementation defined in the ModelType enum.

Summary

• GPULlama3.java supports seven model architectures through the ModelType enum: Llama 3, Mistral, Qwen 2, Qwen 3, Phi-3, Granite, and DeepSeek-R1-Distill-Qwen.
• Each architecture uses dedicated loader, tokenizer, and configuration classes (e.g., MistralModelLoader, GraniteConfiguration) to handle family-specific GGUF metadata and initialization.
• GPU acceleration is available for all models via TornadoVM, with architecture-specific forward passes like forwardGranite and forwardJavaPhi3 providing optimized inference paths.
• The engine is extensible: adding a new model requires only a new enum constant and corresponding implementation classes, while the sampler, state management, and token generation pipeline remain unchanged.

Frequently Asked Questions

How does GPULlama3.java handle different tokenizer formats?

Each model family implements a dedicated tokenizer class—such as Qwen3Tokenizer or Phi3Tokenizer—that reads special token mappings directly from the GGUF metadata. The ChatFormat.create(...) method then selects the appropriate formatting logic (e.g., GraniteChatFormat) to ensure correct prompt templating for that specific architecture.

Can I run Qwen and Phi-3 models on GPU using GPULlama3.java?

Yes. All supported architectures—including Qwen 2/3 and Phi-3—can execute on GPU by passing useTornadoVM=true to ModelType.loadModel(...). The engine automatically selects the TornadoVM-optimized forward pass (e.g., forwardJavaPhi3 adapted for GPU) and handles memory management through TornadoVMMasterPlan.

What is required to add a new model architecture to GPULlama3.java?

Adding support requires implementing four components: a new enum constant in ModelType.java with a loadModel implementation, a model class (e.g., NewModel.java), a configuration class, and a loader class following the existing patterns in src/main/java/org/beehive/gpullama3/model/loader/. The tokenizer and chat format classes must also be provided to handle special tokens.

Does GPULlama3.java support quantized models like Q8_0?

Yes. The architecture-specific loaders—including GraniteLoader and Qwen2ModelLoader—parse quantization metadata from GGUF files and construct weight objects accordingly. Both FP16 and Q8_0 quantization schemes are supported for CPU and TornadoVM GPU execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}