How Does the Nelson Interpreter Parse and Execute Code Internally

The Nelson interpreter processes source code through a three-stage lex-parse-evaluate pipeline, utilizing LexerContext for tokenization, Bison-generated parsers for AST construction, and the Evaluator class for tree-walking execution.

The Nelson numerical computing environment (nelson-lang/nelson) implements a classic interpreter architecture to transform scripts into computational results. Understanding how the Nelson interpreter parses and executes code internally reveals a well-organized pipeline of lexical analysis, syntax parsing, and AST evaluation implemented in the modules/interpreter directory. The system combines Flex-generated scanners with Bison-generated parsers to build an intermediate representation that the evaluation engine traverses to perform operations.

Lexical Analysis: Tokenizing Source Code with LexerContext

The first stage converts raw source text into a structured token stream. The LexerContext class, defined in modules/interpreter/src/include/LexerContext.hpp, manages the input buffer, cursor position, and line/column counters during scanning.

The yylex function serves as the entry point for the lexical analyzer:

extern int yylex(Nelson::LexerContext &lexerContext);

This function, declared in NelSonParser.cpp, repeatedly reads characters from the LexerContext buffer and returns token identifiers (such as IDENT, NUMERIC, IF, or FOR) defined in the grammar specification NelSonParser.yxx. During each invocation, yylex updates the lexer state and populates the global yylval structure with token values—whether identifiers, literals, or operators—providing the parser with typed semantic values for AST construction.

Parsing and AST Generation

Once tokenized, the stream feeds into the Bison-generated parser defined by modules/interpreter/src/grammar/NelSonParser.yxx. This grammar specification describes Nelson's language syntax and defines how to construct nodes for the Abstract Syntax Tree (AST).

Parser Interface and State

The modules/interpreter/src/include/ParserInterface.hpp header provides the public API for invoking the parser, while ParserState.hpp enumerates possible outcomes: ScriptBlock, FuncDef, or ParseError. The typical parsing workflow follows this pattern:

// Reset any previous parser state
Nelson::resetParser();

// Parse source string
Nelson::ParserState state = Nelson::parseString(evaluator->lexerContext, source);

if (state == Nelson::ParserState::ScriptBlock) {
    auto ast = Nelson::getParsedScriptBlock();
    evaluator->block(ast);  // Proceed to evaluation
} else if (state == Nelson::ParserState::FuncDef) {
    auto func = Nelson::getParsedFunctionDef();
    // Store macro-function definition in global function table
}

The parser constructs an AST consisting of typed nodes (ident, num, assign, if, for, call) with child pointers mapping to grammar rules. The AbstractSyntaxTree structure, defined in modules/interpreter/src/include/AbstractSyntaxTree.hpp, serves as the intermediate representation passed between the parsing and evaluation phases.

Evaluating the Abstract Syntax Tree

The Evaluator class in modules/interpreter/src/include/Evaluator.hpp implements the execution engine that traverses the AST. The primary entry point evaluateString orchestrates the full parse-to-execute workflow:

bool Evaluator::evaluateString(const std::string& line, bool propagateException)
{
    resetParser();
    ParserState state = parseString(lexerContext, line);
    if (state != ParserState::ScriptBlock) return false;

    auto ast = getParsedScriptBlock();
    block(ast);  // Execute statements
    return true;
}

Tree-Walking Execution Strategy

The evaluator implements a recursive descent pattern through several key methods:

• block – Iterates over top-level statement nodes in a script block.
• statement – Determines if a node represents a quiet statement, regular statement, or control-flow construct.
• statementType – Dispatches to specialized handlers based on the node's head token:
  • = → assignStatement (delegates to simpleAssign)
  • if → ifStatement (evaluates condition via conditionedStatement)
  • for → forStatement (manages iteration expressions)
  • while → whileStatement
  • Function calls → functionExpression or specialFunctionCall

Expression evaluation methods such as expression, rhsExpression, plusOperator, colonOperator, and functionHandleAnonymousOperator recursively evaluate sub-trees and return ArrayOf objects, which serve as Nelson's runtime value container.

Execution State and Debugging

The interpreter maintains internal state flags (OK, BREAK, CONTINUE, RETURN, QUIT) and a CallStack to manage nested function calls. The Evaluator also integrates debugging support through methods like addBreakpoint, onBreakpoint, and stepBreakpointExists, which hook into the AST traversal to pause execution at specific nodes.

End-to-End Execution Example

The following C++ example demonstrates how to instantiate the runtime environment and execute a Nelson script:

#include "Evaluator.hpp"
#include "Context.hpp"
#include "Interface.hpp"

int main()
{
    // Initialize runtime components
    Nelson::Context ctx;       // Variable and function storage
    Nelson::Interface io;      // Console I/O abstraction
    Nelson::Evaluator eval(&ctx, &io, false, 1);

    // Nelson script as string
    std::string script = "a = 10; b = a * 2; disp(b);";

    // Parse and execute
    eval.evaluateString(script);
    
    return 0;
}

In this workflow, Evaluator::evaluateString triggers the complete pipeline: parsing via parseString, AST retrieval through getParsedScriptBlock, and execution via block. The disp(b) call routes to Evaluator::display, which writes output through the Interface abstraction layer defined in modules/interpreter/src/include/Interface.hpp.

Key Source Files

The Nelson interpreter's implementation spans these critical components:

• LexerContext.hpp (modules/interpreter/src/include/LexerContext.hpp) – Manages input buffers and scanner state for tokenization.
• NelSonParser.yxx (modules/interpreter/src/grammar/NelSonParser.yxx) – Bison grammar defining language syntax and AST construction rules.
• NelSonParser.cpp (modules/interpreter/src/grammar/NelSonParser.cpp) – Generated C++ parser that drives yylex and builds the AST.
• ParserInterface.hpp (modules/interpreter/src/include/ParserInterface.hpp) – Public API for parser reset, string/file parsing, and AST retrieval.
• ParserState.hpp (modules/interpreter/src/include/ParserState.hpp) – Enumeration of parser outcomes (ScriptBlock, FuncDef, ParseError).
• AbstractSyntaxTree.hpp (modules/interpreter/src/include/AbstractSyntaxTree.hpp) – Node structure definitions for the syntax tree.
• Evaluator.hpp (modules/interpreter/src/include/Evaluator.hpp) – Core execution engine implementing statement and expression semantics.
• Interface.hpp (modules/interpreter/src/include/Interface.hpp) – Abstract I/O layer for console, file, and debug panel communication.

Summary

The Nelson interpreter parses and executes code through a well-defined three-layer architecture:

• Lexical Analysis – LexerContext and yylex convert source text into tokens according to the grammar specification.
• Parsing – The Bison-generated NelSonParser constructs an AbstractSyntaxTree from the token stream, accessible via the ParserInterface.
• Evaluation – The Evaluator traverses the AST using block and statement methods, dispatching to specialized handlers for assignments, control flow, and function calls while maintaining execution state and debugging information.

Frequently Asked Questions

How does the Nelson interpreter handle syntax errors during parsing?

When the Bison-generated parser encounters invalid syntax, it returns a ParserState::ParseError through the ParserInterface. The parser uses resetParser() to clear previous state before attempting new input, allowing the interpreter to report specific line and column information derived from the LexerContext without crashing the runtime environment.

What data structure represents runtime values during AST evaluation?

Nelson uses the ArrayOf class as its universal runtime value container. During tree traversal, expression evaluation methods (such as plusOperator or functionExpression) return ArrayOf objects that encapsulate numerical arrays, strings, function handles, or other data types. These objects flow through the Evaluator's recursive calls as the AST is executed.

Can the Nelson interpreter evaluate code incrementally or only full scripts?

The interpreter supports incremental evaluation through Evaluator::evaluateString, which processes single lines or incomplete blocks. The LexerContext maintains input buffer state, allowing the parser to handle interactive sessions where statements are fed individually. The block method executes whatever valid AST nodes are produced, whether from a full script or a single statement parsed in isolation.

How are breakpoints implemented in the Nelson execution engine?

The Evaluator class embeds breakpoint logic via methods like addBreakpoint, onBreakpoint, and stepBreakpointExists. Before executing a statement node during the AST walk, the evaluator checks whether the current line or function matches a registered breakpoint. If triggered, the interpreter pauses execution and yields control to the debugging interface, allowing inspection of the CallStack and variable contexts maintained by the Context object.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}