# How GPULlama3.java Enables LangChain4j Integration for GPU-Accelerated AI Applications > Discover how GPULlama3.java integrates with LangChain4j for GPU-accelerated AI. Stream tokens directly from Llama-3 runtime for faster application development and enhanced performance. - Repository: [Beehive lab/gpullama3.java](https://github.com/beehive-lab/gpullama3.java) - Tags: how-to-guide - Published: 2026-02-26 --- **GPULlama3.java integrates with LangChain4j by implementing the `ChatLanguageModel` interface through the `GPULlama3ChatModel` class, delegating inference calls to GPU-accelerated Llama-3 runtime methods that stream tokens back via LangChain4j-compatible callbacks.** The beehive-lab/gpullama3.java repository provides a high-performance Java inference engine for Llama-3 models that exposes a native LangChain4j provider. This integration allows developers to leverage GPU-accelerated inference through TornadoVM while writing idiomatic LangChain4j code for building AI applications. ## Architecture Overview The integration follows the **provider pattern** common in LangChain4j ecosystems. Rather than implementing the entire LangChain4j stack, GPULlama3.java exposes a specialized model provider that plugs into LangChain4j's `ChatLanguageModel` abstraction. The architecture centers on four primary components that bridge the GPULlama3 runtime with LangChain4j's expectations: - **Model Interface**: Defines the contract for inference operations - **GPULlama3ChatModel**: LangChain4j-facing implementation - **InferenceEngine**: Hardware abstraction layer (CPU/GPU) - **ModelLoader**: GGUF file handling with LangChain4j compatibility ## Core Integration Components ### GPULlama3ChatModel The `GPULlama3ChatModel` class in the published Maven artifact implements LangChain4j's `ChatLanguageModel` interface (v1.7.1+). This class acts as the primary entry point for LangChain4j applications. When instantiated through its fluent builder API, the class holds a reference to a GPULlama3 `Model` instance. All `generate()` calls from LangChain4j are forwarded to the underlying `Model.runInstructOnceLangChain4J` method, which handles the actual token generation. ### Model Interface The core abstraction resides in [`src/main/java/org/beehive/gpullama3/model/Model.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/Model.java), specifically lines 62-101. This interface defines the `runInstructOnceLangChain4J` default method, which serves as the bridge between GPULlama3's native inference and LangChain4j's streaming expectations. The method signature accepts: - `Sampler`: Controls token sampling strategy - `Options`: Configuration including GPU/CPU selection via `useTornadovm()` - `Consumer`: Token callback for streaming The `Consumer` callback is the critical integration point. As the `InferenceEngine` generates tokens (on GPU via TornadoVM or CPU), each decoded token is passed to this callback, which LangChain4j uses to assemble streaming responses. ### InferenceEngine Located in [`src/main/java/org/beehive/gpullama3/inference/InferenceEngine.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/inference/InferenceEngine.java), this component executes the actual model inference. The `runInstructOnceLangChain4J` method delegates to this engine based on the `Options` configuration. When `Options.useTornadovm()` returns `true`, the engine invokes `generateTokensGPU`, leveraging TornadoVM for GPU acceleration. Otherwise, it falls back to `generateTokens` for CPU execution. This hardware abstraction allows LangChain4j applications to switch between CPU and GPU inference through simple configuration changes. ### ModelLoader The `ModelLoader` class in [`src/main/java/org/beehive/gpullama3/model/loader/ModelLoader.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/loader/ModelLoader.java) (lines 101-103) provides GGUF model file loading with specific compatibility methods for LangChain4j and Quarkus integrations. This loader ensures that models initialized through LangChain4j's configuration mechanisms properly instantiate the underlying GPULlama3 runtime, maintaining the connection between LangChain4j's model references and the actual GPU-accelerated inference engine. ## Implementing the Integration ### Configuring the Model Builder To integrate GPULlama3.java with LangChain4j, instantiate the `GPULlama3ChatModel` using its builder pattern. This configuration specifies the GGUF model path, sampling parameters, and hardware acceleration preferences. ```java import org.beehive.gpullama3.GPULlama3ChatModel; // Build the LangChain4j-compatible model GPULlama3ChatModel model = GPULlama3ChatModel.builder() .modelPath("/path/to/llama3-8b.gguf") .temperature(0.9) .topP(0.9) .maxTokens(2048) .onGPU(Boolean.TRUE) // Enable TornadoVM GPU acceleration .build(); ``` ### Creating LangChain4j AI Services Once configured, the `GPULlama3ChatModel` implements `ChatLanguageModel`, allowing it to power LangChain4j's `AiService` interfaces. This enables declarative AI application development with GPU-accelerated inference. ```java import dev.langchain4j.model.chat.ChatLanguageModel; import dev.langchain4j.service.AiService; import dev.langchain4j.service.AiName; // Define the AI service interface @AiService public interface Assistant { @AiName("assistant") String chat(String userMessage); } // Wire the GPULlama3 model into the service ChatLanguageModel chatModel = model; // GPULlama3ChatModel Assistant assistant = AiService.builder(Assistant.class) .chatLanguageModel(chatModel) .build(); // Execute GPU-accelerated inference String response = assistant.chat("Explain GPU acceleration benefits"); System.out.println(response); ``` ### Streaming Tokens with Callbacks For real-time applications, GPULlama3.java supports token streaming through LangChain4j-compatible callbacks. The `generate` method accepts a `Consumer` that receives tokens as they are produced by the GPU or CPU inference engine. ```java // Stream tokens to LangChain4j handler model.generate( "Explain quantum computing in one sentence.", response -> System.out.print(response) // Callback receives each token ); ``` Under the hood, this lambda connects to the `Consumer` parameter in `Model.runInstructOnceLangChain4J`, which the `InferenceEngine` invokes for every decoded token during the generation loop in [`Model.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/Model.java). ## Summary - **GPULlama3.java** provides a native LangChain4j integration through the `GPULlama3ChatModel` class, implementing the `ChatLanguageModel` interface required by LangChain4j v1.7.1+. - The integration delegates inference to the **Model interface** ([`src/main/java/org/beehive/gpullama3/model/Model.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/Model.java)), specifically through the `runInstructOnceLangChain4J` method that streams tokens via `Consumer` callbacks. - **Hardware acceleration** is handled by the `InferenceEngine`, which routes execution to GPU (via TornadoVM) or CPU based on `Options.useTornadovm()` configuration. - Developers configure the integration using a **fluent builder API** to specify GGUF model paths, sampling parameters, and GPU preferences, then wire the model into LangChain4j `AiService` interfaces for declarative AI application development. ## Frequently Asked Questions ### How does GPULlama3.java handle token streaming in LangChain4j applications? GPULlama3.java implements token streaming through the `runInstructOnceLangChain4J` method in the `Model` interface, which accepts a `Consumer` callback parameter. As the `InferenceEngine` generates tokens—whether on GPU via TornadoVM or CPU—it invokes this callback for each decoded token, allowing LangChain4j to receive real-time streaming responses without waiting for complete generation. ### What configuration options are available when building a GPULlama3ChatModel? The `GPULlama3ChatModel.builder()` provides fluent configuration for model path (`modelPath`), sampling parameters (`temperature`, `topP`), generation limits (`maxTokens`), and hardware acceleration (`onGPU`). The `onGPU(Boolean.TRUE)` setting enables TornadoVM GPU acceleration, while `Boolean.FALSE` forces CPU execution through the standard `InferenceEngine` fallback path. ### Which LangChain4j interfaces does GPULlama3ChatModel implement? `GPULlama3ChatModel` implements LangChain4j's `ChatLanguageModel` interface (version 1.7.1 and above), making it compatible with LangChain4j's `AiService` builder patterns and declarative AI service definitions. This implementation allows the GPULlama3 runtime to serve as a drop-in replacement for other LangChain4j model providers while offering GPU-accelerated inference. ### Where is the LangChain4j compatibility code located in the GPULlama3.java repository? The primary LangChain4j integration code resides in [`src/main/java/org/beehive/gpullama3/model/Model.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/Model.java) (lines 62-101), which defines the `runInstructOnceLangChain4J` method. Additional compatibility utilities are found in [`src/main/java/org/beehive/gpullama3/model/loader/ModelLoader.java`](https://github.com/beehive-lab/gpullama3.java/blob/main/src/main/java/org/beehive/gpullama3/model/loader/ModelLoader.java) (lines 101-103), which provides LangChain4j and Quarkus-specific model loading methods. The public API class `GPULlama3ChatModel` is exposed through the main artifact and documented in the repository's README.md.