How to Integrate Python Code in Nelson Using the python_engine Module

Nelson's python_engine module bridges the Nelson interpreter with a native Python runtime through dynamic library loading, lifecycle management, and built-in commands like pyrun and py_invoke.

The nelson-lang/nelson repository provides a robust python_engine module that enables seamless execution of Python code within Nelson scripts. This integration allows you to leverage Python's extensive ecosystem while maintaining the numerical computing workflow of Nelson. The module handles interpreter initialization, code execution, object marshalling, and output capture through a well-defined C++ architecture.

Understanding the python_engine Architecture

The python_engine module operates through three distinct architectural layers that manage the integration between Nelson and Python.

Dynamic Python Library Loading

At the foundation, the PythonEnvironment class discovers the system Python installation and dynamically loads the shared library at runtime. The initializePythonEngine() function in modules/python_engine/src/cpp/PythonEngine.cpp orchestrates this process, attempting to locate and load the Python DLL or shared object. If the library cannot be loaded, the function raises an appropriate error to prevent further execution attempts.

Interpreter Lifecycle Management

The initializePythonEngine() function ensures the Python interpreter initializes only once through the PyIsInitializedInterpreter and PyInitializeInterpreter flags. During initialization, the engine injects a helper script named standardInOutRedirection.py that intercepts stdout and stderr streams. This redirection enables Nelson to retrieve Python output through getPythonStandardOutput() and getPythonStandardError() functions defined in modules/python_engine/src/include/PythonEngine.hpp.

Built-in Command Interface

The module registers high-level built-in commands through modules/python_engine/builtin/cpp/Gateway.cpp. These commands include pyrun, pyrunfile, py_class, py_get, and py_invoke. Each built-in acts as a wrapper that forwards calls to the core PythonRunner class and PyRun functions. The runner synchronizes execution, captures output streams, and converts Python results into Nelson ArrayOf objects for seamless integration.

Executing Python Code with pyrun and pyrunfile

The pyrun built-in executes Python code strings directly within the Nelson environment. The implementation in modules/python_engine/builtin/cpp/pyrunBuiltin.cpp parses arguments, builds a Python global dictionary, compiles the code, and executes it via PythonRunner::runPythonCode.

// Execute a Python expression and capture a variable
result = pyrun('import math; x = math.sqrt(9)', 'x');
// result now holds the numeric value 3.0

For executing external scripts, pyrunfile loads and runs Python files from disk. This command handles file path resolution and executes the script content within the initialized Python environment.

// Run a Python script file
pyrunfile('my_script.py');  // executes the file located in the current folder

Working with Python Objects in Nelson

The python_engine module supports object-oriented Python programming through handles that wrap Python objects as Nelson variables.

Creating Objects with py_class

The py_class built-in instantiates Python classes and returns a PythonObjectHandle that Nelson can manipulate. This handle stores a reference to the Python object and the interpreter context.

// Create a Python datetime object
dt = py_class('datetime.datetime', '2024', '3', '8');
// dt is now a handle to a Python datetime instance

Invoking Methods with py_invoke

The py_invoke built-in calls methods on Python object handles. The implementation in modules/python_engine/builtin/cpp/py_invokeBuiltin.cpp retrieves the handle, validates the object, and executes the method through PythonObjectHandle::invoke, which uses the stored interpreter pointer to perform the call.

// Call the strftime method on the datetime object
formatted = py_invoke(dt, 'strftime', '%Y-%m-%d');
// formatted contains the string "2024-03-08"

Capturing Python Output and Errors

Since the Python interpreter runs in a separate embedded context, the python_engine module captures standard streams to make them accessible from Nelson. The standardInOutRedirection.py helper script redirects stdout and stderr to internal buffers that Nelson can query.

// Execute Python code that prints output
pyrun('print("Hello from Python")');
out = getPythonStandardOutput();   // returns "Hello from Python\n"

// Capture error messages
pyrun('import sys; sys.stderr.write("Error message")');
err = getPythonStandardError();    // returns "Error message"

You can also pass Nelson variables into the Python global namespace using the pyrun argument syntax, enabling bidirectional data exchange.

// Pass Nelson variables as Python globals
a = 42;
b = pyrun('c = a * 2; c', [], 'a', a);   // b receives 84

Summary

• The python_engine module in nelson-lang/nelson enables bidirectional integration between Nelson and Python through dynamic library loading and interpreter management.
• Initialization occurs via initializePythonEngine() in modules/python_engine/src/cpp/PythonEngine.cpp, which handles Python runtime discovery and stdout/stderr redirection.
• Code execution uses pyrun for strings and pyrunfile for scripts, implemented in modules/python_engine/builtin/cpp/pyrunBuiltin.cpp and executed through the PythonRunner class.
• Object-oriented programming is supported via py_class for instantiation and py_invoke for method calls, with handles managed through PythonObjectHandle.
• Output capture functions getPythonStandardOutput() and getPythonStandardError() retrieve Python stream content after execution.

Frequently Asked Questions

How do I initialize the Python engine before running code?

The Python engine initializes automatically when you first call a Python-related built-in such as pyrun or py_class. The initializePythonEngine() function in modules/python_engine/src/cpp/PythonEngine.cpp handles dynamic library loading and interpreter startup, ensuring the Python runtime is ready before executing your code.

Can I pass Nelson variables into Python code?

Yes, the pyrun built-in accepts name-value pairs that inject Nelson variables into the Python global namespace. Use the syntax pyrun('python_code', [], 'var_name', nelson_variable) to make Nelson data available to Python, enabling seamless data exchange between the two environments.

How do I capture Python print statements in Nelson?

The python_engine module redirects Python's stdout and stderr streams through an internal helper script. After executing Python code, call getPythonStandardOutput() to retrieve printed text or getPythonStandardError() to capture error messages, both defined in modules/python_engine/src/include/PythonEngine.hpp.

What is the difference between pyrun and pyrunfile?

The pyrun built-in executes Python code strings directly within the Nelson interpreter, while pyrunfile loads and executes external .py script files from disk. Both functions utilize the PythonRunner class for execution, but pyrunfile handles file path resolution and reading before passing the code to the Python interpreter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}