# How the KCL Compiler Pipeline Processes Code from Lexing to Evaluation

> Explore the KCL compiler pipeline stages transforming code from lexing to evaluation. Understand the KCL compilation process from source to executable configuration.

- Repository: [The KCL Programming Language/kcl](https://github.com/kcl-lang/kcl)
- Tags: internals
- Published: 2026-03-05

---

**The KCL compiler pipeline transforms source code into executable configuration through four distinct stages: lexical analysis in `crates/lexer`, parsing into an AST in `crates/parser`, semantic resolution and type checking in `crates/sema`, and final evaluation in `crates/evaluator`, all orchestrated by the driver in `crates/driver`.**

The KCL compiler pipeline is the core engine behind the KCL (Kubernetes Configuration Language) project, hosted at `kcl-lang/kcl`. It converts declarative configuration files into concrete JSON or YAML output through a rigorous multi-stage process. Understanding this pipeline—from raw text tokens to evaluated runtime values—is essential for debugging performance issues, extending the language, or integrating KCL into custom toolchains.

## 1. Lexical Analysis

The pipeline begins in [`crates/lexer/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/lexer/src/lib.rs), where the lexer scans source characters and categorizes them into a stream of `Token` objects.

Key components include:

- **`parse_token_streams`** (located in [`lexer/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/lexer/mod.rs)): The primary entry point invoked by the parser. It accepts a `ParseSession`, source string, and starting position, returning a vector of `Token` instances.
- **`Cursor`** (in [`cursor.rs`](https://github.com/kcl-lang/kcl/blob/main/cursor.rs)): Manages byte offsets and tracks line/column positions during scanning.
- **`TokenKind`**: An enum defining all lexical categories—identifiers, string literals, numeric literals, punctuation, keywords, and comment tokens.

The lexer performs minimal error recovery; invalid identifiers or malformed literals are flagged with specific token flags (e.g., `InvalidIdent`) rather than halting compilation.

## 2. Parsing

The token stream flows into [`crates/parser/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/parser/src/lib.rs), where the recursive-descent parser constructs an Abstract Syntax Tree (AST).

The parsing workflow:

1. Initialize a `ParseSession` containing diagnostics and source-map information.
2. Create a `Parser` instance via `Parser::new(&sess, stream)`.
3. Invoke **`parse_module`** to generate an `ast::Module` representing the top-level structure of a KCL file.

The resulting AST captures high-level constructs including import declarations, schema definitions, rule blocks, and configuration expressions. This tree serves as the input for the semantic analysis phase.

## 3. Semantic Analysis (Sema)

Semantic resolution occurs in [`crates/sema/src/resolver/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/sema/src/resolver/mod.rs). The resolver traverses the AST to bind identifiers to definitions, infer types, and validate constraints.

Core responsibilities:

- **Name Resolution**: Links variable references, schema names, and import paths to their declarations using scope-based symbol tables.
- **Type Inference & Checking**: Leverages the type system defined in [`crates/sema/src/ty/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/sema/src/ty/mod.rs) to compute concrete `Type` representations, unify generic parameters, and detect type mismatches.
- **Constraint Validation**: Evaluates `rule` blocks and schema constraints to ensure user-provided values satisfy defined invariants.
- **Plugin Loading**: Processes `kclplugin` imports through the plugin subsystem in `kcl_sema::plugin`.

The output is a **resolved program** (`kcl_sema::Program`) containing fully typed and validated definitions ready for execution.

## 4. Evaluation

The final stage executes the resolved program in [`crates/evaluator/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/evaluator/src/lib.rs). The evaluator interprets the semantic model to produce concrete configuration values.

Key structures:

- **`Context`**: Maintains runtime state including variable bindings, import caches, and schema instantiation records.
- **`Module`**: Represents an evaluated module containing resolved schemas, functions, and configuration data.
- **`Runtime`**: Executes statements, evaluates expressions, and applies schema defaults and overrides.
- **`Value`** (defined in [`value.rs`](https://github.com/kcl-lang/kcl/blob/main/value.rs)): A unified representation for all KCL runtime values including primitives, lists, dictionaries, and schema instances.

Evaluation produces a value tree that the runtime serializes to JSON or YAML output formats.

## 5. Driver Orchestration

The `crates/driver` crate coordinates the entire pipeline. The primary entry point is `kcl_driver::run`, implemented in [`crates/driver/src/toolchain.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/driver/src/toolchain.rs).

Orchestration flow:

1. **Load Files**: Initialize a `ParseSession` and load source files.
2. **Parse**: Invoke `parse_file_with_session` to generate the AST.
3. **Resolve**: Call `sema::resolver::Resolver::resolve` to perform semantic analysis.
4. **Evaluate**: Execute `evaluator::run_program` to generate runtime values.
5. **Return**: Deliver the final value or emit collected diagnostics.

The driver abstracts the complexity of stage management, error propagation, and incremental compilation support.

## Code Example: Running the Full Pipeline

You can invoke the KCL compiler pipeline programmatically using the driver API:

```rust
use kcl_driver::{run, DriverOptions};
use std::path::PathBuf;

fn main() -> anyhow::Result<()> {
    // 1️⃣ Configure driver options (type checking, plugins, etc.)
    let opts = DriverOptions::default();

    // 2️⃣ Specify the entry KCL file
    let entry = PathBuf::from("example.k");

    // 3️⃣ Execute the full pipeline: lexer → parser → sema → evaluator
    let result = run(&[entry], opts)?;

    // 4️⃣ Serialize the evaluated configuration to JSON
    let json = serde_json::to_string_pretty(&result.value)?;
    println!("{}", json);
    Ok(())
}

```

The `run` function handles all stages internally, returning a structured result containing the evaluated value and any compilation diagnostics.

## Key Source Files

| Stage | File | Description |
|-------|------|-------------|
| **Lexing** | [`crates/lexer/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/lexer/src/lib.rs) | Tokenization and lexical analysis |
| **Parsing** | [`crates/parser/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/parser/src/lib.rs) | AST construction from token streams |
| **Semantic Analysis** | [`crates/sema/src/resolver/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/sema/src/resolver/mod.rs) | Name resolution and type checking |
| **Type System** | [`crates/sema/src/ty/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/sema/src/ty/mod.rs) | Type definitions and inference logic |
| **Evaluation** | [`crates/evaluator/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/evaluator/src/lib.rs) | Runtime execution and value generation |
| **Driver** | [`crates/driver/src/toolchain.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/driver/src/toolchain.rs) | Pipeline orchestration and public API |

These files collectively implement the KCL compiler pipeline, from raw source text to executable configuration output.

## Summary

- The **KCL compiler pipeline** consists of four discrete stages: lexing, parsing, semantic analysis, and evaluation.
- **Lexical analysis** in `crates/lexer` converts source text into token streams via `parse_token_streams`.
- **Parsing** in `crates/parser` builds an AST using `Parser::new` and `parse_module`.
- **Semantic analysis** in `crates/sema` resolves names and types through `Resolver::resolve`, producing a typed `Program`.
- **Evaluation** in `crates/evaluator` executes the resolved program via `run_program`, generating concrete `Value` objects.
- The **driver** in `crates/driver` coordinates the entire flow through `kcl_driver::run`, providing the primary API for CLI and programmatic usage.

## Frequently Asked Questions

### What is the entry point for the KCL compiler pipeline?

The primary entry point is **`kcl_driver::run`** located in [`crates/driver/src/toolchain.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/driver/src/toolchain.rs). This function accepts source file paths and driver options, then orchestrates the complete pipeline from lexing through evaluation, returning the final configuration value or compilation errors.

### How does the KCL compiler handle type checking?

Type checking occurs during the **semantic analysis phase** in [`crates/sema/src/resolver/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/sema/src/resolver/mod.rs). The `Resolver` traverses the AST to infer types using the type system defined in [`crates/sema/src/ty/mod.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/sema/src/ty/mod.rs), unifying generic parameters and validating that all expressions conform to their declared schemas before evaluation begins.

### What crate handles the final evaluation of KCL code?

The **`crates/evaluator`** crate handles final evaluation, with its main interface in [`crates/evaluator/src/lib.rs`](https://github.com/kcl-lang/kcl/blob/main/crates/evaluator/src/lib.rs). It interprets the resolved semantic model to produce runtime values, executing schema definitions, applying defaults and overrides, and generating the final configuration output that can be serialized to JSON or YAML.

### How can I invoke the KCL compiler pipeline programmatically?

You can invoke the pipeline using the driver API by calling **`kcl_driver::run`** with a list of entry file paths and `DriverOptions`. This function abstracts the internal stages—internally calling `parse_file_with_session`, `Resolver::resolve`, and `evaluator::run_program`—and returns a structured result containing the evaluated value and any diagnostics.