GPULlama3.java Interactive Mode vs Instruction Mode: A Technical Comparison

GPULlama3.java offers two distinct CLI execution styles: an interactive REPL chat mode that maintains conversation state across turns, and a single-prompt instruction mode that processes one request and exits immediately.

The beehive-lab/gpullama3.java repository provides a GPU-accelerated Java implementation of the Llama 3 model using TornadoVM. When launching the CLI via LlamaTornadoCli.java, users must choose between interactive mode for conversational workflows and instruction mode for one-off generation tasks. The selected mode determines how the system handles state persistence, GPU execution plans, and user input processing.

Core Architectural Differences

The primary distinction lies in how each mode manages the Model interface and underlying State objects throughout the program lifecycle.

Execution Flow and Entry Points

In LlamaTornadoCli.java, the main() method checks the options.interactive() boolean to determine which execution path to invoke.

Interactive mode calls model.runInteractive(sampler, options) at LlamaTornadoCli.java lines 106-114. This initializes a persistent REPL loop that prints a > prompt, reads lines from stdin, and continues until the user types quit or exit.

Instruction mode calls runSingleInstruction(model, sampler, options) at LlamaTornadoCli.java lines 70-78. This method executes a single prompt provided via the --prompt (or -p) flag and terminates immediately after generating the response.

State Management and Conversation History

Interactive mode maintains a persistent State object and conversation token list across turns, enabling coherent multi-turn dialogues. According to Model.java lines 71-84, the chat history accumulates with each user input, allowing the model to reference previous turns.

Instruction mode creates a fresh State for each one-shot generation at Model.java lines 88-95. No conversation history persists after the call completes, making this mode stateless by design.

GPU Acceleration and Performance Planning

When using TornadoVM acceleration (--use-tornadovm true), both modes create a TornadoVMMasterPlan, but their lifecycle management differs significantly.

Interactive mode initializes the execution plan once at the start of the chat session and reuses it for every turn, freeing resources only when the loop ends (Model.java lines 94-101 and 169-176). This reuse minimizes GPU initialization overhead during conversations.

Instruction mode lazily creates the plan for the single request but frees it immediately after the response is produced (Model.java lines 99-104). This approach ensures clean resource management for scripting scenarios where the process exits immediately.

User Interface and Streaming Behavior

Both modes support token streaming via the --stream flag, but the user experience differs.

Interactive mode provides continuous UI feedback with performance metrics printed after each turn when SHOW_PERF_INTERACTIVE is enabled (Model.java lines 26-30 and 66-70). The streaming output creates an "as-you-type" feel within the persistent chat interface.

Instruction mode streams tokens directly to console for the single call, but lacks the continuous prompt-and-response loop. Once generation completes, the program exits without awaiting further input.

How to Launch Each Mode

The --interactive (or -i) flag controls mode selection during option parsing at Options.java lines 85-86.

Starting an Interactive Chat Session

Launch a REPL-style conversation with persistent history:

jbang LlamaTornadoCli.java -m /models/llama-3-8B.gguf --interactive \
      --temperature 0.7 --top-p 0.9 --max-tokens 512

The console displays a banner and repeatedly prompts:

> Explain quantum computing simply.
[streaming response]
> Now give me a use case.
[response considers previous context]

Running a Single Instruction Prompt

Execute a one-off prompt and exit immediately:

jbang LlamaTornadoCli.java -m /models/llama-3-8B.gguf \
      -p "Summarize the plot of Hamlet in three sentences." \
      --temperature 0.3 --max-tokens 128

This command outputs only the generated summary without entering an interactive loop.

GPU Acceleration in Both Modes

Add --use-tornadovm true to either command above to enable GPU acceleration. The interactive session maintains the TornadoVM plan across turns, while the instruction mode initializes and destroys GPU resources within the single execution.

Implementation Details

CLI Dispatch Logic

The LlamaTornadoCli.java entry point implements a straightforward dispatch pattern:

// Simplified logic from lines 106-115
if (options.interactive()) {
    model.runInteractive(sampler, options);
} else {
    runSingleInstruction(model, sampler, options);
}

This boolean check occurs after Options.parseOptions(args) processes the command-line arguments.

Model Interface Design

Both modes share the same underlying Model interface but invoke different default methods. The runInteractive() method implements the while-loop with history handling, while runInstructOnce() (called by runSingleInstruction) performs a single forward pass without state retention.

The Options.java class defines the interactive flag and its parsing logic, supporting aliases including --chat and -i for consistency with common CLI conventions.

Summary

• Interactive mode launches a REPL chat loop via model.runInteractive(), maintains persistent State across turns, and reuses the TornadoVMMasterPlan for the entire session to minimize GPU overhead.
• Instruction mode executes a single prompt via runSingleInstruction(), creates fresh state for each call, and immediately frees GPU resources after generation.
• The --interactive (or -i) flag selects the mode during CLI initialization in LlamaTornadoCli.java.
• Both modes support --stream for token-by-token output, but only interactive mode provides continuous prompting and conversation history.
• Interactive mode suits conversational agents and debugging; instruction mode fits scripting and one-off completions.

Frequently Asked Questions

How do I switch between interactive and instruction mode in GPULlama3.java?

Add the --interactive or -i flag to enter interactive mode. Omit this flag and provide a prompt via -p or --prompt to run in instruction mode. The Options.java class parses these flags at lines 85-86, setting the boolean that LlamaTornadoCli.java checks at lines 106-115 to determine the execution path.

Does interactive mode use more GPU memory than instruction mode?

Yes, interactive mode typically consumes GPU memory for the duration of the chat session because it retains the TornadoVMMasterPlan and conversation state across turns (Model.java lines 94-101). Instruction mode frees these resources immediately after the single generation completes (Model.java lines 99-104), making it more memory-efficient for batch processing.

Can I use streaming output in both GPULlama3.java modes?

Yes, both modes support the --stream flag. In interactive mode, streaming provides real-time token display within the REPL loop along with optional performance metrics (Model.java lines 26-30). In instruction mode, streaming outputs tokens directly to console before the program exits, without the interactive prompt infrastructure.

Why does my conversation history reset when I restart the CLI?

GPULlama3.java does not persist conversation history to disk. Interactive mode maintains history only in memory during the active session via the State object and token list (Model.java lines 71-84). Once you exit the program, all conversation context is lost, requiring state externalization if you need persistent conversations across sessions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}