# Nelson Engine API: How to Embed Nelson as a Computational Backend in C Applications

> Discover the Nelson Engine API to embed Nelson as a computational backend in C applications. Start Nelson processes, evaluate commands, and exchange data via shared memory IPC.

- Repository: [The Nelson Programming Language/nelson](https://github.com/nelson-lang/nelson)
- Tags: how-to-guide
- Published: 2026-03-08

---

**The Nelson Engine API is a MATLAB-compatible MEX interface that allows C programs to start a Nelson process, evaluate commands, and exchange data via shared memory IPC.**

The Nelson Engine API, implemented in the `nelson-lang/nelson` repository, provides a direct path to embed Nelson’s numerical computing capabilities into external C or C++ applications. By mirroring the MATLAB MEX Engine API, it enables existing MATLAB-centric codebases to migrate or integrate with Nelson while maintaining familiar function signatures and workflows.

## What Is the Nelson Engine API?

The Nelson Engine API is a C interface defined in [`modules/mex/src/include/engine.h`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/include/engine.h) and implemented in [`modules/mex/src/cpp/Engine.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/cpp/Engine.cpp). It exposes a set of functions that allow host applications to control a Nelson interpreter running in a separate process. The API is designed to be compatible with the MATLAB MEX Engine API, meaning functions like `engOpen`, `engEvalString`, and `engPutVariable` behave identically to their MATLAB counterparts.

The engine operates by launching a Nelson GUI or headless process and establishing communication through **inter-process shared memory**, **named mutexes**, and a lightweight **IPC protocol**. The host application receives an opaque `Engine*` handle, which it uses for all subsequent interactions.

## Architecture and IPC Mechanism

### Process Management and Shared Memory

When `engOpen` is invoked, the API performs several steps to establish the connection:

1. Creates a **Nelson-interprocess receiver** for the parent PID using `Nelson::createNelsonInterprocessReceiver`, which sets up the shared-memory segment for commands and data.
2. Checks for an existing GUI-enabled Nelson instance via `Nelson::getLatestPidWithModeInSharedMemory(NELSON_ENGINE_MODE::GUI)`.
3. If no instance exists, spawns a new Nelson GUI process using `start_child` (platform-specific: `CreateProcessW` on Windows or `fork/exec` on POSIX).
4. Waits for the child to signal readiness through the **NelsonReadyNamedMutex** (`waitUntilNelsonIsReady`).

The resulting `Engine` struct stores a pointer to the `PROCESS_CHILD` object and an `isSingleUse` flag that determines whether `engClose` should terminate the child process.

### Thread Safety and Reference Counting

All engine interactions are thread-safe. The implementation maintains a **reference count** (`countEngine`) to manage shared resources. When the last engine handle is closed via `engClose`, the API cleans up shared memory segments and mutexes, ensuring no resource leaks occur between multiple embedded instances.

## Core Engine API Functions

The Nelson Engine API provides the following functions for controlling the interpreter:

| Function | Purpose |
|----------|---------|
| `engOpen` / `engOpenSingleUse` | Start a Nelson process (shared or single-use) and return an `Engine*` handle |
| `engClose` | Close the engine connection and optionally terminate the child process |
| `engEvalString` | Execute a Nelson command string in the remote workspace |
| `engPutVariable` | Transfer a variable from C (`mxArray`) to Nelson's workspace |
| `engGetVariable` | Retrieve a variable from Nelson's workspace into a C `mxArray` |
| `engOutputBuffer` | Attach a character buffer to capture text output from the engine |
| `engSetVisible` / `engGetVisible` | Control or query the visibility of the Nelson GUI window |

Data marshalling between Nelson's internal `ArrayOf` type and MATLAB-compatible `mxArray` structures is handled by `ArrayOfToMxArray` and `MxArrayToArrayOf`, defined in [`modules/mex/src/include/mxArrayOf.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/include/mxArrayOf.hpp).

## Step-by-Step Implementation Guide

### Opening the Engine Connection

To begin embedding Nelson, include the engine header and open a connection:

```c
#include "engine.h"

Engine *ep = engOpen("");
if (ep == NULL) {
    fprintf(stderr, "Unable to start Nelson engine\n");
    return 1;
}

```

Passing an empty string to `engOpen` attempts to connect to an existing Nelson GUI instance or launches a new one. Use `engOpenSingleUse` to force a dedicated process that terminates when closed.

### Evaluating Commands and Exchanging Data

Once connected, execute Nelson commands and transfer data:

```c
/* Execute a command in Nelson's workspace */
engEvalString(ep, "x = 1:10; y = x.^2;");

/* Create a double array in C and send it to Nelson */
double data[] = {3.14, 2.71, 1.41};
mxArray *mx = mxCreateDoubleMatrix(1, 3, mxREAL);
memcpy(mxGetPr(mx), data, sizeof(data));
engPutVariable(ep, "constants", mx);
mxDestroyArray(mx);

/* Retrieve a variable from Nelson */
mxArray *result = engGetVariable(ep, "y");
if (result != NULL) {
    double *pr = mxGetPr(result);
    printf("First element of y: %f\n", pr[0]);
    mxDestroyArray(result);
}

```

### Closing the Engine Session

When finished, close the connection to release resources:

```c
engClose(ep);

```

If the engine was opened in shared mode (`engOpen`), the Nelson GUI process remains running for other applications to use. If opened with `engOpenSingleUse`, `engClose` terminates the dedicated process.

## Complete C Example: Embedding Nelson

The following complete example demonstrates starting the engine, evaluating expressions, transferring variables, and shutting down:

```c
#include <stdio.h>
#include "engine.h"        /* Nelson Engine API header */

int main(void)
{
    /* 1. Open a shared Nelson engine (reuse existing GUI if present) */
    Engine *ep = engOpen("");
    if (ep == NULL) {
        fprintf(stderr, "Failed to start Nelson engine.\n");
        return 1;
    }

    /* 2. Evaluate a command */
    if (engEvalString(ep, "a = 42;") != 0) {
        fprintf(stderr, "Eval failed.\n");
        engClose(ep);
        return 1;
    }

    /* 3. Create a numeric mxArray on the C side */
    double data[3] = {1.0, 2.0, 3.0};
    mxArray *mx = mxCreateDoubleMatrix(1, 3, mxREAL);
    memcpy(mxGetPr(mx), data, sizeof(data));

    /* 4. Put the array into Nelson's workspace as variable 'b' */
    if (engPutVariable(ep, "b", mx) != 0) {
        fprintf(stderr, "PutVariable failed.\n");
        mxDestroyArray(mx);
        engClose(ep);
        return 1;
    }
    mxDestroyArray(mx);  /* safe to free after transfer */

    /* 5. Retrieve variable 'a' from Nelson */
    mxArray *retr = engGetVariable(ep, "a");
    if (retr) {
        double a_val = *mxGetPr(retr);
        printf("Nelson returned a = %f\n", a_val);
        mxDestroyArray(retr);
    } else {
        fprintf(stderr, "GetVariable failed.\n");
    }

    /* 6. Close the engine (will not terminate the GUI because we used shared mode) */
    engClose(ep);
    return 0;
}

```

**Explanation of workflow**

| Step | API call | Purpose |
|------|----------|---------|
| 1 | `engOpen("")` | Attaches to existing Nelson GUI or launches new process; returns `Engine*` handle |
| 2 | `engEvalString` | Executes `a = 42;` in the remote workspace |
| 3‑4 | `mxCreateDoubleMatrix` + `engPutVariable` | Constructs C array, sends to Nelson as variable `b` |
| 5 | `engGetVariable` | Retrieves `a` from Nelson into C `mxArray` |
| 6 | `engClose` | Releases handle; shared GUI remains running |

Compile with the Nelson library (e.g., `-lnlsMex` on Linux or link `libnlsMex.dll` on Windows) and ensure the include path contains `modules/mex/src/include`.

## Key Source Files and Headers

| File | Role | GitHub Link |
|------|------|-------------|
| [`modules/mex/src/include/engine.h`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/include/engine.h) | Public C header exposing the Engine API | [engine.h](https://github.com/nelson-lang/nelson/blob/master/modules/mex/src/include/engine.h) |
| [`modules/mex/src/cpp/Engine.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/cpp/Engine.cpp) | Core implementation of engine functions, process handling, IPC, and data marshalling | [Engine.cpp](https://github.com/nelson-lang/nelson/blob/master/modules/mex/src/cpp/Engine.cpp) |
| [`modules/mex/src/include/mxArrayOf.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/include/mxArrayOf.hpp) | Conversion utilities between `mxArray` and Nelson’s `ArrayOf` | [mxArrayOf.hpp](https://github.com/nelson-lang/nelson/blob/master/modules/mex/src/include/mxArrayOf.hpp) |
| [`modules/mex/src/include/MxVariables.h`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/include/MxVariables.h) | Helper macros and definitions for MEX-compatible matrix types | [MxVariables.h](https://github.com/nelson-lang/nelson/blob/master/modules/mex/src/include/MxVariables.h) |
| [`modules/interprocess/src/include/NelsonInterprocess.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/interprocess/src/include/NelsonInterprocess.hpp) | Shared-memory segment, mutex handling, and command protocol | [NelsonInterprocess.hpp](https://github.com/nelson-lang/nelson/blob/master/modules/interprocess/src/include/NelsonInterprocess.hpp) |

These files constitute the complete backend that enables C applications to embed Nelson as a computational engine.

## Summary

- The **Nelson Engine API** provides MATLAB-compatible functions (`engOpen`, `engEvalString`, `engPutVariable`, etc.) to control a Nelson interpreter from C or C++.
- Communication occurs via **shared memory IPC** and named mutexes, with the engine running as a separate process managed through an opaque `Engine*` handle.
- **Thread-safe reference counting** ensures proper cleanup of shared resources when multiple applications or threads use the engine concurrently.
- Data exchange uses **MATLAB-compatible `mxArray` structures**, converted to Nelson’s internal `ArrayOf` type via `ArrayOfToMxArray` and `MxArrayToArrayOf`.
- The API supports both **shared mode** (attaches to existing GUI) and **single-use mode** (dedicated process), controlled via `engOpen` and `engOpenSingleUse`.

## Frequently Asked Questions

### How does the Nelson Engine API differ from MATLAB's Engine API?

The Nelson Engine API is designed to be **functionally compatible** with the MATLAB MEX Engine API, using identical function signatures for `engOpen`, `engEvalString`, `engPutVariable`, and related calls. However, Nelson’s implementation uses **shared memory IPC** rather than COM (Windows) or pipes (Unix) typically used by MATLAB, and it marshals data between `mxArray` and Nelson’s native `ArrayOf` type. This allows existing MATLAB engine code to compile against Nelson with minimal modifications while leveraging Nelson’s open-source architecture.

### Can I run multiple Nelson engine instances simultaneously from the same C application?

Yes, the Nelson Engine API supports **multiple concurrent instances** through thread-safe reference counting. You can call `engOpen` or `engOpenSingleUse` multiple times to obtain distinct `Engine*` handles, each representing a separate connection. The internal `countEngine` variable tracks active handles, and shared IPC resources are only cleaned up when the last engine is closed via `engClose`. This design allows complex applications to isolate computational contexts or distribute workloads across multiple Nelson processes.

### What data types can be transferred between C and Nelson using the Engine API?

The API supports all **MATLAB-compatible matrix types** through the `mxArray` interface, including `mxDOUBLE_ARRAY`, `mxSINGLE_ARRAY`, `mxINT8_ARRAY`, `mxUINT8_ARRAY`, `mxINT16_ARRAY`, `mxUINT16_ARRAY`, `mxINT32_ARRAY`, `mxUINT32_ARRAY`, `mxINT64_ARRAY`, `mxUINT64_ARRAY`, `mxCHAR_ARRAY`, `mxSTRUCT_CLASS`, and `mxCELL_CLASS`. Data conversion occurs in [`modules/mex/src/include/mxArrayOf.hpp`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/include/mxArrayOf.hpp), where `MxArrayToArrayOf` converts incoming `mxArray` structures to Nelson’s `ArrayOf` objects, and `ArrayOfToMxArray` handles the reverse direction when retrieving variables via `engGetVariable`.

### How do I capture text output from Nelson when using the Engine API?

Use the `engOutputBuffer` function to redirect Nelson’s standard output to a C character buffer. Before calling `engEvalString`, allocate a character array and register it with `engOutputBuffer(ep, buffer, bufferSize)`. When `engEvalString` completes, the buffer contains UTF-8 encoded text output from the command execution. The implementation in [`modules/mex/src/cpp/Engine.cpp`](https://github.com/nelson-lang/nelson/blob/main/modules/mex/src/cpp/Engine.cpp) (lines 486-499) converts internal wide strings to UTF-8 using `Nelson::wstring_to_utf8` before populating the buffer, ensuring cross-platform text compatibility.