How the LLVM JIT Compiler Works: Embedding ORC v2 in C++ Applications

The LLVM JIT compiler uses the ORC v2 infrastructure with LLJIT and LLLazyJIT classes to compile LLVM IR into machine code at runtime, allowing you to embed just-in-time compilation into your application by creating a JIT instance, adding modules via ThreadSafeModule, and looking up symbols to obtain callable function pointers.

The llvm/llvm-project repository provides a modern Just-In-Time (JIT) compilation framework built on the On-Request Compilation (ORC) v2 architecture. This system moves beyond traditional JIT approaches by treating the JIT as a dynamic linker, enabling features like lazy compilation, concurrent compilation, and code removal at runtime.

ORC v2 Architecture and Core Components

The LLVM JIT compiler is structured around several key abstractions defined in llvm/include/llvm/ExecutionEngine/Orc/LLJIT.h and related headers.

ExecutionSession and JITDylib

At the foundation lies the ExecutionSession, which owns the entire JIT session and manages symbol strings, error reporting, and synchronization. Within this session, JITDylib instances act as dynamic libraries inside the JIT—each maintains a symbol table and can declare dependencies on other JITDylibs to control symbol resolution order.

When you embed the LLVM JIT compiler, you interact with these components through higher-level wrappers. The ExecutionSession locks during symbol lookup to gather required definitions, then unlocks to allow concurrent materialization.

The Layer Stack and Materialization

ORC v2 organizes compilation through a stack of layers that transform and compile program representations:

• IRCompileLayer – compiles LLVM IR modules to machine code
• IRTransformLayer – applies IR transformations before compilation
• RTDyldObjectLinkingLayer – links object files using the runtime dynamic linker
• CompileOnDemandLayer – enables lazy compilation by creating stubs (used by LLLazyJIT)

The MaterializationUnit serves as the generic compiler interface. When JIT->lookup("foo") is called, the system creates a query, identifies the relevant MaterializationUnit, and dispatches it for compilation—either immediately (eager) or deferred (lazy).

ThreadSafeModule and Memory Management

Before adding code to the JIT, you must wrap LLVM modules in a ThreadSafeModule, defined in llvm/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h. This pairs an LLVM Module with a ThreadSafeContext to guarantee safe concurrent access. The jitlink::JITLinkMemoryManager abstracts allocation of executable memory, ensuring proper permissions and lifecycle management.

Eager vs. Lazy Compilation Strategies

The LLVM JIT compiler provides two primary embedding strategies depending on your performance requirements.

LLJIT for Eager Compilation

LLJIT compiles symbols immediately upon lookup. When you call lookup() on a symbol in the main JITDylib, the system materializes the function right away, returning a ready-to-call address. This approach suits applications where startup latency matters less than consistent execution speed.

LLLazyJIT for On-Demand Compilation

LLLazyJIT defers compilation until the first function call. When you add a module via addLazyIRModule(), the system creates trampolines that trigger compilation via the CompileOnDemandLayer on first invocation. This reduces initial load times but introduces a pause on first execution. Configure concurrent compilation threads using setNumCompileThreads() to mitigate this latency.

How to Embed the LLVM JIT Compiler

Embedding requires linking against LLVMExecutionEngine or LLVMOrcJIT and following a specific initialization pattern demonstrated in llvm/examples/HowToUseLLJIT/HowToUseLLJIT.cpp.

Creating the JIT Instance

Use the builder pattern to construct your JIT stack. The LLJITBuilder automatically detects the host target and configures the layer stack:

#include "llvm/ExecutionEngine/Orc/LLJIT.h"

auto JITOrErr = LLJITBuilder().create();
if (!JITOrErr) {
    return JITOrErr.takeError();
}
auto &JIT = *JITOrErr;

For lazy compilation, substitute LLLazyJITBuilder and optionally configure compilation threads:

auto LazyJITOrErr = LLLazyJITBuilder()
                       .setNumCompileThreads(4)
                       .create();

Loading LLVM IR Modules

Convert your LLVM IR into a ThreadSafeModule before adding it to the JITDylib:

#include "llvm/ExecutionEngine/Orc/ThreadSafeModule.h"
#include "llvm/IRReader/IRReader.h"

ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
auto Mod = parseIRFile("module.ll", Err, *TSCtx.getContext());
if (!Mod) {
    return Err;
}
ThreadSafeModule TSM(std::move(Mod), std::move(TSCtx));

if (auto Err = JIT->addIRModule(std::move(TSM))) {
    return Err;
}

For lazy loading, call addLazyIRModule() instead, which creates the deferred compilation stubs.

Looking Up and Calling JIT-Compiled Functions

Symbol resolution follows the same linking rules as static linkers, respecting visibility and weak definitions. Retrieve function pointers by name and cast them to callable types:

auto SymOrErr = JIT->lookup("my_function");
if (!SymOrErr) {
    return SymOrErr.takeError();
}

using FuncTy = int(*)(int, int);
auto *FuncPtr = SymOrErr->getAddress().toPtr<FuncTy>();
int Result = FuncPtr(42, 100);

The address returned by getAddress() remains valid for the lifetime of the containing JITDylib or until explicitly removed via a ResourceTracker.

Thread Safety and Concurrency

The ORC v2 stack supports concurrent compilation through the ConcurrentIRCompiler class. When you configure multiple compilation threads via setNumCompileThreads(), materialization units dispatch to a thread pool while the calling thread waits for the specific symbol it requested. All JIT operations are thread-safe, though you must ensure your LLVMContext is wrapped in ThreadSafeContext as shown above.

The JIT-as-linker model enables reliable removal of code: ResourceTrackers track dependencies between symbols and materialization units, allowing you to unload specific modules without destroying the entire JIT session.

Summary

• The LLVM JIT compiler operates on the ORC v2 architecture, treating JIT compilation as dynamic linking.
• LLJIT provides eager compilation at lookup time, while LLLazyJIT defers compilation until first call via CompileOnDemandLayer.
• ExecutionSession manages the JIT lifecycle while JITDylib instances organize code into searchable symbol tables.
• Embed the JIT using LLJITBuilder, wrap modules in ThreadSafeModule, and retrieve functions via lookup() followed by getAddress().toPtr<>().
• Source files llvm/include/llvm/ExecutionEngine/Orc/LLJIT.h and llvm/examples/HowToUseLLJIT/HowToUseLLJIT.cpp provide the definitive reference implementation.

Frequently Asked Questions

What is the difference between LLJIT and LLLazyJIT?

LLJIT compiles code immediately when you call lookup(), ensuring the function is ready before execution begins. LLLazyJIT, as shown in llvm/examples/OrcV2Examples/LLJITWithLazyReexports.cpp, uses the CompileOnDemandLayer to create stubs that trigger compilation only when the function is first called, trading initial call latency for faster startup times.

How does symbol lookup work in the LLVM JIT?

According to the implementation in llvm/include/llvm/ExecutionEngine/Orc/ExecutionUtils.h, calling JIT->lookup("symbol") creates a lookup query that locks the ExecutionSession, gathers MaterializationUnits containing the requested symbol, unlocks the session, and dispatches those units for compilation. Once materialized, the address is cached and returned to the caller.

Is the LLVM JIT compiler thread-safe?

Yes. The ORC v2 infrastructure uses ThreadSafeContext and ThreadSafeModule to protect LLVM IR, while the ExecutionSession manages synchronization internally. You can safely call lookup() from multiple threads, and when using setNumCompileThreads(), compilation occurs in parallel on a thread pool without manual locking.

What headers and libraries are required to embed the LLVM JIT?

You need llvm/include/llvm/ExecutionEngine/Orc/LLJIT.h for the JIT classes, llvm/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h for concurrent module handling, and llvm/IRReader/IRReader.h if parsing IR from files. Link against LLVMOrcJIT and LLVMExecutionEngine libraries, and consult llvm/examples/HowToUseLLJIT/HowToUseLLJIT.cpp for a complete build example.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}