How the KCL Compiler Pipeline Processes Code from Lexing to Evaluation

The KCL compiler pipeline transforms source code into executable configuration through four distinct stages: lexical analysis in crates/lexer, parsing into an AST in crates/parser, semantic resolution and type checking in crates/sema, and final evaluation in crates/evaluator, all orchestrated by the driver in crates/driver.

The KCL compiler pipeline is the core engine behind the KCL (Kubernetes Configuration Language) project, hosted at kcl-lang/kcl. It converts declarative configuration files into concrete JSON or YAML output through a rigorous multi-stage process. Understanding this pipeline—from raw text tokens to evaluated runtime values—is essential for debugging performance issues, extending the language, or integrating KCL into custom toolchains.

1. Lexical Analysis

The pipeline begins in crates/lexer/src/lib.rs, where the lexer scans source characters and categorizes them into a stream of Token objects.

Key components include:

• parse_token_streams (located in lexer/mod.rs): The primary entry point invoked by the parser. It accepts a ParseSession, source string, and starting position, returning a vector of Token instances.
• Cursor (in cursor.rs): Manages byte offsets and tracks line/column positions during scanning.
• TokenKind: An enum defining all lexical categories—identifiers, string literals, numeric literals, punctuation, keywords, and comment tokens.

The lexer performs minimal error recovery; invalid identifiers or malformed literals are flagged with specific token flags (e.g., InvalidIdent) rather than halting compilation.

2. Parsing

The token stream flows into crates/parser/src/lib.rs, where the recursive-descent parser constructs an Abstract Syntax Tree (AST).

The parsing workflow:

1. Initialize a ParseSession containing diagnostics and source-map information.
2. Create a Parser instance via Parser::new(&sess, stream).
3. Invoke parse_module to generate an ast::Module representing the top-level structure of a KCL file.

The resulting AST captures high-level constructs including import declarations, schema definitions, rule blocks, and configuration expressions. This tree serves as the input for the semantic analysis phase.

3. Semantic Analysis (Sema)

Semantic resolution occurs in crates/sema/src/resolver/mod.rs. The resolver traverses the AST to bind identifiers to definitions, infer types, and validate constraints.

Core responsibilities:

• Name Resolution: Links variable references, schema names, and import paths to their declarations using scope-based symbol tables.
• Type Inference & Checking: Leverages the type system defined in crates/sema/src/ty/mod.rs to compute concrete Type representations, unify generic parameters, and detect type mismatches.
• Constraint Validation: Evaluates rule blocks and schema constraints to ensure user-provided values satisfy defined invariants.
• Plugin Loading: Processes kclplugin imports through the plugin subsystem in kcl_sema::plugin.

The output is a resolved program (kcl_sema::Program) containing fully typed and validated definitions ready for execution.

4. Evaluation

The final stage executes the resolved program in crates/evaluator/src/lib.rs. The evaluator interprets the semantic model to produce concrete configuration values.

Key structures:

• Context: Maintains runtime state including variable bindings, import caches, and schema instantiation records.
• Module: Represents an evaluated module containing resolved schemas, functions, and configuration data.
• Runtime: Executes statements, evaluates expressions, and applies schema defaults and overrides.
• Value (defined in value.rs): A unified representation for all KCL runtime values including primitives, lists, dictionaries, and schema instances.

Evaluation produces a value tree that the runtime serializes to JSON or YAML output formats.

5. Driver Orchestration

The crates/driver crate coordinates the entire pipeline. The primary entry point is kcl_driver::run, implemented in crates/driver/src/toolchain.rs.

Orchestration flow:

1. Load Files: Initialize a ParseSession and load source files.
2. Parse: Invoke parse_file_with_session to generate the AST.
3. Resolve: Call sema::resolver::Resolver::resolve to perform semantic analysis.
4. Evaluate: Execute evaluator::run_program to generate runtime values.
5. Return: Deliver the final value or emit collected diagnostics.

The driver abstracts the complexity of stage management, error propagation, and incremental compilation support.

Code Example: Running the Full Pipeline

You can invoke the KCL compiler pipeline programmatically using the driver API:

use kcl_driver::{run, DriverOptions};
use std::path::PathBuf;

fn main() -> anyhow::Result<()> {
    // 1️⃣ Configure driver options (type checking, plugins, etc.)
    let opts = DriverOptions::default();

    // 2️⃣ Specify the entry KCL file
    let entry = PathBuf::from("example.k");

    // 3️⃣ Execute the full pipeline: lexer → parser → sema → evaluator
    let result = run(&[entry], opts)?;

    // 4️⃣ Serialize the evaluated configuration to JSON
    let json = serde_json::to_string_pretty(&result.value)?;
    println!("{}", json);
    Ok(())
}

The run function handles all stages internally, returning a structured result containing the evaluated value and any compilation diagnostics.

Key Source Files

Stage	File	Description
Lexing	crates/lexer/src/lib.rs	Tokenization and lexical analysis
Parsing	crates/parser/src/lib.rs	AST construction from token streams
Semantic Analysis	crates/sema/src/resolver/mod.rs	Name resolution and type checking
Type System	crates/sema/src/ty/mod.rs	Type definitions and inference logic
Evaluation	crates/evaluator/src/lib.rs	Runtime execution and value generation
Driver	crates/driver/src/toolchain.rs	Pipeline orchestration and public API

These files collectively implement the KCL compiler pipeline, from raw source text to executable configuration output.

Summary

• The KCL compiler pipeline consists of four discrete stages: lexing, parsing, semantic analysis, and evaluation.
• Lexical analysis in crates/lexer converts source text into token streams via parse_token_streams.
• Parsing in crates/parser builds an AST using Parser::new and parse_module.
• Semantic analysis in crates/sema resolves names and types through Resolver::resolve, producing a typed Program.
• Evaluation in crates/evaluator executes the resolved program via run_program, generating concrete Value objects.
• The driver in crates/driver coordinates the entire flow through kcl_driver::run, providing the primary API for CLI and programmatic usage.

Frequently Asked Questions

What is the entry point for the KCL compiler pipeline?

The primary entry point is kcl_driver::run located in crates/driver/src/toolchain.rs. This function accepts source file paths and driver options, then orchestrates the complete pipeline from lexing through evaluation, returning the final configuration value or compilation errors.

How does the KCL compiler handle type checking?

Type checking occurs during the semantic analysis phase in crates/sema/src/resolver/mod.rs. The Resolver traverses the AST to infer types using the type system defined in crates/sema/src/ty/mod.rs, unifying generic parameters and validating that all expressions conform to their declared schemas before evaluation begins.

What crate handles the final evaluation of KCL code?

The crates/evaluator crate handles final evaluation, with its main interface in crates/evaluator/src/lib.rs. It interprets the resolved semantic model to produce runtime values, executing schema definitions, applying defaults and overrides, and generating the final configuration output that can be serialized to JSON or YAML.

How can I invoke the KCL compiler pipeline programmatically?

You can invoke the pipeline using the driver API by calling kcl_driver::run with a list of entry file paths and DriverOptions. This function abstracts the internal stages—internally calling parse_file_with_session, Resolver::resolve, and evaluator::run_program—and returns a structured result containing the evaluated value and any diagnostics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}