# How PythonRunner Embeds and Executes Analytics Scripts from C++ in FinceptTerminal

> Discover how PythonRunner embeds and executes analytics scripts from C++ within FinceptTerminal. Learn about virtual environment management, concurrent execution, and async callbacks.

- Repository: [Fincept Corporation/FinceptTerminal](https://github.com/Fincept-Corporation/FinceptTerminal)
- Tags: internals
- Published: 2026-04-20

---

**PythonRunner is a singleton C++ wrapper that abstracts the full lifecycle of Python subprocesses—managing virtual environment discovery, argument serialization, concurrent execution queues, and JSON extraction—so FinceptTerminal analytics workflows can execute Python scripts via simple asynchronous callbacks.**

FinceptTerminal bridges quantitative Python analytics with a native Qt C++ interface through a specialized wrapper called **PythonRunner**. This component eliminates the complexity of directly embedding the Python interpreter by instead managing external Python processes, virtual environments, and real-time output streaming. Understanding how `PythonRunner` embeds and executes analytics scripts from C++ reveals the architecture that powers the terminal's technical indicator calculations and data processing nodes.

## Architecture and Singleton Design

The `PythonRunner` class follows the singleton pattern to ensure a single global instance manages all Python execution state across the application. When an analytics workflow node requires Python execution, it accesses the runner through `PythonRunner::instance()`.

This singleton holds critical runtime state including the discovered Python interpreter paths, the scripts directory location, a `QQueue<QueuedRequest>` for pending jobs, and concurrency counters. According to the implementation in [`fincept-qt/src/python/PythonRunner.h`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/fincept-qt/src/python/PythonRunner.h), this design prevents resource conflicts while allowing the Qt event loop to remain responsive during script execution.

## Python Interpreter Discovery and Virtual Environment Routing

Before executing any script, `PythonRunner` must locate a valid Python installation. The initialization logic in [`fincept-qt/src/python/PythonRunner.cpp`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/fincept-qt/src/python/PythonRunner.cpp) performs this discovery through two primary methods:

- **`find_python_sync()`**: Performs a fast, non-blocking search for managed virtual environments named `venv-numpy2` or `venv-numpy1` via `PythonSetupManager`
- **`find_python_async()`**: Falls back to an asynchronous system Python check using `python3 --version` if no managed venv is found

Once a base interpreter is located, `select_venv_for_script()` determines the appropriate virtual environment for the specific workload. Regular analytics scripts route to either NumPy 1 or NumPy 2 environments based on known script names, while inline notebook code always executes in the NumPy 2 venv. This routing ensures dependency compatibility without requiring C++ code changes.

## Script Discovery and Environment Preparation

The runner locates analytics scripts using `find_scripts_dir()`, which walks up from the executable directory searching for a `scripts/` folder containing expected entry points like [`yfinance_data.py`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/yfinance_data.py). This directory becomes the base path for all relative script references.

For each execution, `build_python_env()` constructs a specialized `QProcessEnvironment` that:
- Pins UTF-8 encoding and disables bytecode generation via `PYTHONDONTWRITEBYTECODE=1`
- Sets `FINCEPT_DATA_DIR` and `FINAGENT_DATA_DIR` environment variables
- Builds a `PYTHONPATH` starting with the discovered `scripts_dir`

When executing submodules (scripts containing `/` like [`indicators/sma.py`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/indicators/sma.py)), `PythonRunner` automatically prepends the parent-of-package directory to `PYTHONPATH` and invokes the module using `python -m indicators.sma`, ensuring relative imports resolve correctly.

## Executing Analytics Scripts

Analytics workflows invoke Python through the primary `run()` method. The helper function `analytics_run_python_json()` in [`fincept-qt/src/services/workflow/nodes/AnalyticsNodes.cpp`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/fincept-qt/src/services/workflow/nodes/AnalyticsNodes.cpp) demonstrates the typical invocation pattern:

```cpp
PythonRunner::instance().run("compute_technicals.py",
    {"--data", json_input, "--indicator", "SMA", "--period", "14"},
    [](const PythonResult& res) {
        if (!res.success) {
            qWarning() << "Python error:" << res.error;
            return;
        }
        QJsonDocument doc = QJsonDocument::fromJson(res.output.toUtf8());
        // Process JSON result...
    });

```

The `PythonResult` struct contains `success` (bool), `output` (JSON or raw stdout), `error` (stderr or process error), and `exit_code`. This callback-based approach keeps the UI thread non-blocking while the Python process executes quant calculations.

## Handling Raw Python Code

For notebook-style execution or dynamic code generation, `PythonRunner` provides `run_code()`. This method:
1. Writes the code to a temporary `.py` file
2. Routes through the same virtual environment selection and concurrency logic
3. Automatically deletes the temporary file upon completion

```cpp
QString code = R"(
import json
import pandas as pd
result = {"mean": pd.Series([1,2,3]).mean()}
print(json.dumps(result))
)";

PythonRunner::instance().run_code(code, [](const PythonResult& res) {
    if (res.success) {
        qDebug() << "Cell output:" << res.output;
    }
});

```

This path always uses the NumPy 2 virtual environment to ensure modern pandas and numpy compatibility for interactive analytics.

## Argument Spilling and Large Payload Handling

When analytics workflows pass large datasets (> 8KB) as arguments, `PythonRunner` implements argument spilling to avoid command-line length limits. In the `start_next()` method, oversized arguments are written to temporary JSON files and referenced using `@/path/to/file` syntax. The Python entry scripts recognize this `@-file` convention and expand the content before processing, enabling seamless handling of large historical price datasets or complex configuration objects without breaking process invocation limits.

## Concurrency Control and Live Output Streaming

`PythonRunner` maintains a `queue_` of pending requests and respects a `max_concurrent_` limit (default 3) to prevent system resource exhaustion. When a slot frees, `start_next()` dequeues the next request and spawns a `QProcess`.

Live output streaming occurs through Qt signals:
- `readyReadStandardOutput` and `readyReadStandardError` trigger the `drain_lines()` lambda
- This splits output on `\n` boundaries and fires optional `StreamCallback` lambdas per line
- Analytics nodes use this to update progress bars or log windows while indicators calculate

When the process exits, the finish handler flushes remaining buffers, removes any spilled temporary files, and invokes `extract_json()` to parse the first `{` or `[` block from stdout before delivering the final `PythonResult` to the caller's callback.

## Error Handling and Result Extraction

Process failures trigger the `errorOccurred` signal handler, which cleans up temporary files and invokes the callback with a failure status. Successful executions flow through `extract_json()` at the bottom of [`PythonRunner.cpp`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/PythonRunner.cpp), which scans stdout for JSON blocks. The runner also checks for Python-side error patterns like `{ "success": false, "error": "..." }` that analytics scripts emit to indicate calculation failures despite clean process exits.

## Summary

- **PythonRunner** is a singleton C++ wrapper in `fincept-qt/src/python/` that manages Python execution for FinceptTerminal
- It discovers and routes to appropriate virtual environments (`venv-numpy1` or `venv-numpy2`) using `select_venv_for_script()`
- Scripts are found via `find_scripts_dir()` and executed with prepared `PYTHONPATH` and environment variables
- The `run()` method accepts script names and argument lists, while `run_code()` handles temporary files for inline Python
- Argument spilling handles payloads >8KB by writing to temporary files referenced with `@` syntax
- Concurrent execution is limited to 3 processes by default with queuing managed in `start_next()`
- Live output streams via `drain_lines()` enable real-time UI updates during script execution
- Results are extracted via `extract_json()` and returned through asynchronous callbacks to keep the Qt event loop responsive

## Frequently Asked Questions

### How does PythonRunner handle different Python virtual environments?

`PythonRunner` uses `select_venv_for_script()` in [`PythonRunner.cpp`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/PythonRunner.cpp) to route requests to either a NumPy 1 or NumPy 2 virtual environment based on known script compatibility requirements. The `PythonSetupManager` class provides the actual filesystem paths to these environments, and the runner overrides the base interpreter path when spawning the `QProcess` for specific scripts.

### What happens when analytics scripts return large datasets?

When arguments exceed 8KB, `PythonRunner` automatically spills them to temporary JSON files in `start_next()`. These files are referenced using `@/path/to/tempfile` syntax on the command line, which the Python analytics scripts recognize and expand. This prevents command-line length errors while maintaining the same functional interface for the C++ calling code.

### Can PythonRunner execute Python code from strings rather than files?

Yes, the `run_code()` method accepts a `QString` containing Python source code. It creates a temporary `.py` file, executes it through the same concurrency and environment preparation pipeline as regular scripts, and deletes the temporary file upon completion. This method always routes to the NumPy 2 virtual environment for maximum compatibility with modern analytics libraries.

### How does the C++ side receive results from Python scripts?

Execution is asynchronous via callbacks. The `run()` method accepts a `std::function<void(const PythonResult&)>` callback that receives a struct containing `success`, `output`, `error`, and `exit_code`. The `extract_json()` function parses stdout to find JSON blocks, allowing C++ analytics nodes in [`AnalyticsNodes.cpp`](https://github.com/Fincept-Corporation/FinceptTerminal/blob/main/AnalyticsNodes.cpp) to deserialize results directly into `QJsonDocument` objects without manual string parsing.