# How Does the Nelson Interpreter Parse and Execute Code Internally

> Discover how the Nelson interpreter parses and executes code internally. Explore its lex-parse-evaluate pipeline, tokenization, AST construction, and tree-walking execution.

- Repository: [The Nelson Programming Language/nelson](https://github.com/nelson-lang/nelson)
- Tags: internals
- Published: 2026-03-08

---

**The Nelson interpreter processes source code through a three-stage lex-parse-evaluate pipeline, utilizing `LexerContext` for tokenization, Bison-generated parsers for AST construction, and the `Evaluator` class for tree-walking execution.**

The Nelson numerical computing environment (nelson-lang/nelson) implements a classic interpreter architecture to transform scripts into computational results. Understanding how the Nelson interpreter parses and executes code internally reveals a well-organized pipeline of lexical analysis, syntax parsing, and AST evaluation implemented in the `modules/interpreter` directory. The system combines Flex-generated scanners with Bison-generated parsers to build an intermediate representation that the evaluation engine traverses to perform operations.

## Lexical Analysis: Tokenizing Source Code with `LexerContext`

The first stage converts raw source text into a structured token stream. The **`LexerContext`** class, defined in [`modules/interpreter/src/include/LexerContext.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/LexerContext.hpp), manages the input buffer, cursor position, and line/column counters during scanning.

The `yylex` function serves as the entry point for the lexical analyzer:

```cpp
extern int yylex(Nelson::LexerContext &lexerContext);

```

This function, declared in [`NelSonParser.cpp`](https://github.com/nelson-lang/nelson/blob/main/NelSonParser.cpp), repeatedly reads characters from the `LexerContext` buffer and returns token identifiers (such as `IDENT`, `NUMERIC`, `IF`, or `FOR`) defined in the grammar specification `NelSonParser.yxx`. During each invocation, `yylex` updates the lexer state and populates the global `yylval` structure with token values—whether identifiers, literals, or operators—providing the parser with typed semantic values for AST construction.

## Parsing and AST Generation

Once tokenized, the stream feeds into the **Bison-generated parser** defined by `modules/interpreter/src/grammar/NelSonParser.yxx`. This grammar specification describes Nelson's language syntax and defines how to construct nodes for the **Abstract Syntax Tree (AST)**.

### Parser Interface and State

The [`modules/interpreter/src/include/ParserInterface.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/ParserInterface.hpp) header provides the public API for invoking the parser, while [`ParserState.hpp`](https://github.com/nelson-lang/nelson/blob/main/ParserState.hpp) enumerates possible outcomes: `ScriptBlock`, `FuncDef`, or `ParseError`. The typical parsing workflow follows this pattern:

```cpp
// Reset any previous parser state
Nelson::resetParser();

// Parse source string
Nelson::ParserState state = Nelson::parseString(evaluator->lexerContext, source);

if (state == Nelson::ParserState::ScriptBlock) {
    auto ast = Nelson::getParsedScriptBlock();
    evaluator->block(ast);  // Proceed to evaluation
} else if (state == Nelson::ParserState::FuncDef) {
    auto func = Nelson::getParsedFunctionDef();
    // Store macro-function definition in global function table
}

```

The parser constructs an AST consisting of typed nodes (`ident`, `num`, `assign`, `if`, `for`, `call`) with child pointers mapping to grammar rules. The **`AbstractSyntaxTree`** structure, defined in [`modules/interpreter/src/include/AbstractSyntaxTree.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/AbstractSyntaxTree.hpp), serves as the intermediate representation passed between the parsing and evaluation phases.

## Evaluating the Abstract Syntax Tree

The **`Evaluator`** class in [`modules/interpreter/src/include/Evaluator.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/Evaluator.hpp) implements the execution engine that traverses the AST. The primary entry point `evaluateString` orchestrates the full parse-to-execute workflow:

```cpp
bool Evaluator::evaluateString(const std::string& line, bool propagateException)
{
    resetParser();
    ParserState state = parseString(lexerContext, line);
    if (state != ParserState::ScriptBlock) return false;

    auto ast = getParsedScriptBlock();
    block(ast);  // Execute statements
    return true;
}

```

### Tree-Walking Execution Strategy

The evaluator implements a recursive descent pattern through several key methods:

- **`block`** – Iterates over top-level statement nodes in a script block.
- **`statement`** – Determines if a node represents a quiet statement, regular statement, or control-flow construct.
- **`statementType`** – Dispatches to specialized handlers based on the node's head token:
  - **`=`** → `assignStatement` (delegates to `simpleAssign`)
  - **`if`** → `ifStatement` (evaluates condition via `conditionedStatement`)
  - **`for`** → `forStatement` (manages iteration expressions)
  - **`while`** → `whileStatement`
  - **Function calls** → `functionExpression` or `specialFunctionCall`

Expression evaluation methods such as `expression`, `rhsExpression`, `plusOperator`, `colonOperator`, and `functionHandleAnonymousOperator` recursively evaluate sub-trees and return **`ArrayOf`** objects, which serve as Nelson's runtime value container.

### Execution State and Debugging

The interpreter maintains internal **state flags** (OK, BREAK, CONTINUE, RETURN, QUIT) and a **`CallStack`** to manage nested function calls. The `Evaluator` also integrates debugging support through methods like `addBreakpoint`, `onBreakpoint`, and `stepBreakpointExists`, which hook into the AST traversal to pause execution at specific nodes.

## End-to-End Execution Example

The following C++ example demonstrates how to instantiate the runtime environment and execute a Nelson script:

```cpp
#include "Evaluator.hpp"
#include "Context.hpp"
#include "Interface.hpp"

int main()
{
    // Initialize runtime components
    Nelson::Context ctx;       // Variable and function storage
    Nelson::Interface io;      // Console I/O abstraction
    Nelson::Evaluator eval(&ctx, &io, false, 1);

    // Nelson script as string
    std::string script = "a = 10; b = a * 2; disp(b);";

    // Parse and execute
    eval.evaluateString(script);
    
    return 0;
}

```

In this workflow, `Evaluator::evaluateString` triggers the complete pipeline: parsing via `parseString`, AST retrieval through `getParsedScriptBlock`, and execution via `block`. The `disp(b)` call routes to `Evaluator::display`, which writes output through the `Interface` abstraction layer defined in [`modules/interpreter/src/include/Interface.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/Interface.hpp).

## Key Source Files

The Nelson interpreter's implementation spans these critical components:

- **[`LexerContext.hpp`](https://github.com/nelson-lang/nelson/blob/main/LexerContext.hpp)** ([`modules/interpreter/src/include/LexerContext.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/LexerContext.hpp)) – Manages input buffers and scanner state for tokenization.
- **`NelSonParser.yxx`** (`modules/interpreter/src/grammar/NelSonParser.yxx`) – Bison grammar defining language syntax and AST construction rules.
- **[`NelSonParser.cpp`](https://github.com/nelson-lang/nelson/blob/main/NelSonParser.cpp)** ([`modules/interpreter/src/grammar/NelSonParser.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/grammar/NelSonParser.cpp)) – Generated C++ parser that drives `yylex` and builds the AST.
- **[`ParserInterface.hpp`](https://github.com/nelson-lang/nelson/blob/main/ParserInterface.hpp)** ([`modules/interpreter/src/include/ParserInterface.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/ParserInterface.hpp)) – Public API for parser reset, string/file parsing, and AST retrieval.
- **[`ParserState.hpp`](https://github.com/nelson-lang/nelson/blob/main/ParserState.hpp)** ([`modules/interpreter/src/include/ParserState.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/ParserState.hpp)) – Enumeration of parser outcomes (ScriptBlock, FuncDef, ParseError).
- **[`AbstractSyntaxTree.hpp`](https://github.com/nelson-lang/nelson/blob/main/AbstractSyntaxTree.hpp)** ([`modules/interpreter/src/include/AbstractSyntaxTree.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/AbstractSyntaxTree.hpp)) – Node structure definitions for the syntax tree.
- **[`Evaluator.hpp`](https://github.com/nelson-lang/nelson/blob/main/Evaluator.hpp)** ([`modules/interpreter/src/include/Evaluator.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/Evaluator.hpp)) – Core execution engine implementing statement and expression semantics.
- **[`Interface.hpp`](https://github.com/nelson-lang/nelson/blob/main/Interface.hpp)** ([`modules/interpreter/src/include/Interface.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interpreter/src/include/Interface.hpp)) – Abstract I/O layer for console, file, and debug panel communication.

## Summary

The Nelson interpreter parses and executes code through a well-defined three-layer architecture:

- **Lexical Analysis** – `LexerContext` and `yylex` convert source text into tokens according to the grammar specification.
- **Parsing** – The Bison-generated `NelSonParser` constructs an `AbstractSyntaxTree` from the token stream, accessible via the `ParserInterface`.
- **Evaluation** – The `Evaluator` traverses the AST using `block` and `statement` methods, dispatching to specialized handlers for assignments, control flow, and function calls while maintaining execution state and debugging information.

## Frequently Asked Questions

### How does the Nelson interpreter handle syntax errors during parsing?

When the Bison-generated parser encounters invalid syntax, it returns a `ParserState::ParseError` through the `ParserInterface`. The parser uses `resetParser()` to clear previous state before attempting new input, allowing the interpreter to report specific line and column information derived from the `LexerContext` without crashing the runtime environment.

### What data structure represents runtime values during AST evaluation?

Nelson uses the **`ArrayOf`** class as its universal runtime value container. During tree traversal, expression evaluation methods (such as `plusOperator` or `functionExpression`) return `ArrayOf` objects that encapsulate numerical arrays, strings, function handles, or other data types. These objects flow through the `Evaluator`'s recursive calls as the AST is executed.

### Can the Nelson interpreter evaluate code incrementally or only full scripts?

The interpreter supports incremental evaluation through `Evaluator::evaluateString`, which processes single lines or incomplete blocks. The `LexerContext` maintains input buffer state, allowing the parser to handle interactive sessions where statements are fed individually. The `block` method executes whatever valid AST nodes are produced, whether from a full script or a single statement parsed in isolation.

### How are breakpoints implemented in the Nelson execution engine?

The `Evaluator` class embeds breakpoint logic via methods like `addBreakpoint`, `onBreakpoint`, and `stepBreakpointExists`. Before executing a statement node during the AST walk, the evaluator checks whether the current line or function matches a registered breakpoint. If triggered, the interpreter pauses execution and yields control to the debugging interface, allowing inspection of the `CallStack` and variable contexts maintained by the `Context` object.