How PythonRunner Embeds and Executes Analytics Scripts from C++ in FinceptTerminal

PythonRunner is a singleton C++ wrapper that abstracts the full lifecycle of Python subprocesses—managing virtual environment discovery, argument serialization, concurrent execution queues, and JSON extraction—so FinceptTerminal analytics workflows can execute Python scripts via simple asynchronous callbacks.

FinceptTerminal bridges quantitative Python analytics with a native Qt C++ interface through a specialized wrapper called PythonRunner. This component eliminates the complexity of directly embedding the Python interpreter by instead managing external Python processes, virtual environments, and real-time output streaming. Understanding how PythonRunner embeds and executes analytics scripts from C++ reveals the architecture that powers the terminal's technical indicator calculations and data processing nodes.

Architecture and Singleton Design

The PythonRunner class follows the singleton pattern to ensure a single global instance manages all Python execution state across the application. When an analytics workflow node requires Python execution, it accesses the runner through PythonRunner::instance().

This singleton holds critical runtime state including the discovered Python interpreter paths, the scripts directory location, a QQueue<QueuedRequest> for pending jobs, and concurrency counters. According to the implementation in fincept-qt/src/python/PythonRunner.h, this design prevents resource conflicts while allowing the Qt event loop to remain responsive during script execution.

Python Interpreter Discovery and Virtual Environment Routing

Before executing any script, PythonRunner must locate a valid Python installation. The initialization logic in fincept-qt/src/python/PythonRunner.cpp performs this discovery through two primary methods:

• find_python_sync(): Performs a fast, non-blocking search for managed virtual environments named venv-numpy2 or venv-numpy1 via PythonSetupManager
• find_python_async(): Falls back to an asynchronous system Python check using python3 --version if no managed venv is found

Once a base interpreter is located, select_venv_for_script() determines the appropriate virtual environment for the specific workload. Regular analytics scripts route to either NumPy 1 or NumPy 2 environments based on known script names, while inline notebook code always executes in the NumPy 2 venv. This routing ensures dependency compatibility without requiring C++ code changes.

Script Discovery and Environment Preparation

The runner locates analytics scripts using find_scripts_dir(), which walks up from the executable directory searching for a scripts/ folder containing expected entry points like yfinance_data.py. This directory becomes the base path for all relative script references.

For each execution, build_python_env() constructs a specialized QProcessEnvironment that:

• Pins UTF-8 encoding and disables bytecode generation via PYTHONDONTWRITEBYTECODE=1
• Sets FINCEPT_DATA_DIR and FINAGENT_DATA_DIR environment variables
• Builds a PYTHONPATH starting with the discovered scripts_dir

When executing submodules (scripts containing / like indicators/sma.py), PythonRunner automatically prepends the parent-of-package directory to PYTHONPATH and invokes the module using python -m indicators.sma, ensuring relative imports resolve correctly.

Executing Analytics Scripts

Analytics workflows invoke Python through the primary run() method. The helper function analytics_run_python_json() in fincept-qt/src/services/workflow/nodes/AnalyticsNodes.cpp demonstrates the typical invocation pattern:

PythonRunner::instance().run("compute_technicals.py",
    {"--data", json_input, "--indicator", "SMA", "--period", "14"},
    [](const PythonResult& res) {
        if (!res.success) {
            qWarning() << "Python error:" << res.error;
            return;
        }
        QJsonDocument doc = QJsonDocument::fromJson(res.output.toUtf8());
        // Process JSON result...
    });

The PythonResult struct contains success (bool), output (JSON or raw stdout), error (stderr or process error), and exit_code. This callback-based approach keeps the UI thread non-blocking while the Python process executes quant calculations.

Handling Raw Python Code

For notebook-style execution or dynamic code generation, PythonRunner provides run_code(). This method:

1. Writes the code to a temporary .py file
2. Routes through the same virtual environment selection and concurrency logic
3. Automatically deletes the temporary file upon completion

QString code = R"(
import json
import pandas as pd
result = {"mean": pd.Series([1,2,3]).mean()}
print(json.dumps(result))
)";

PythonRunner::instance().run_code(code, [](const PythonResult& res) {
    if (res.success) {
        qDebug() << "Cell output:" << res.output;
    }
});

This path always uses the NumPy 2 virtual environment to ensure modern pandas and numpy compatibility for interactive analytics.

Argument Spilling and Large Payload Handling

When analytics workflows pass large datasets (> 8KB) as arguments, PythonRunner implements argument spilling to avoid command-line length limits. In the start_next() method, oversized arguments are written to temporary JSON files and referenced using @/path/to/file syntax. The Python entry scripts recognize this @-file convention and expand the content before processing, enabling seamless handling of large historical price datasets or complex configuration objects without breaking process invocation limits.

Concurrency Control and Live Output Streaming

PythonRunner maintains a queue_ of pending requests and respects a max_concurrent_ limit (default 3) to prevent system resource exhaustion. When a slot frees, start_next() dequeues the next request and spawns a QProcess.

Live output streaming occurs through Qt signals:

• readyReadStandardOutput and readyReadStandardError trigger the drain_lines() lambda
• This splits output on \n boundaries and fires optional StreamCallback lambdas per line
• Analytics nodes use this to update progress bars or log windows while indicators calculate

When the process exits, the finish handler flushes remaining buffers, removes any spilled temporary files, and invokes extract_json() to parse the first { or [ block from stdout before delivering the final PythonResult to the caller's callback.

Error Handling and Result Extraction

Process failures trigger the errorOccurred signal handler, which cleans up temporary files and invokes the callback with a failure status. Successful executions flow through extract_json() at the bottom of PythonRunner.cpp, which scans stdout for JSON blocks. The runner also checks for Python-side error patterns like { "success": false, "error": "..." } that analytics scripts emit to indicate calculation failures despite clean process exits.

Summary

• PythonRunner is a singleton C++ wrapper in fincept-qt/src/python/ that manages Python execution for FinceptTerminal
• It discovers and routes to appropriate virtual environments (venv-numpy1 or venv-numpy2) using select_venv_for_script()
• Scripts are found via find_scripts_dir() and executed with prepared PYTHONPATH and environment variables
• The run() method accepts script names and argument lists, while run_code() handles temporary files for inline Python
• Argument spilling handles payloads >8KB by writing to temporary files referenced with @ syntax
• Concurrent execution is limited to 3 processes by default with queuing managed in start_next()
• Live output streams via drain_lines() enable real-time UI updates during script execution
• Results are extracted via extract_json() and returned through asynchronous callbacks to keep the Qt event loop responsive

Frequently Asked Questions

How does PythonRunner handle different Python virtual environments?

PythonRunner uses select_venv_for_script() in PythonRunner.cpp to route requests to either a NumPy 1 or NumPy 2 virtual environment based on known script compatibility requirements. The PythonSetupManager class provides the actual filesystem paths to these environments, and the runner overrides the base interpreter path when spawning the QProcess for specific scripts.

What happens when analytics scripts return large datasets?

When arguments exceed 8KB, PythonRunner automatically spills them to temporary JSON files in start_next(). These files are referenced using @/path/to/tempfile syntax on the command line, which the Python analytics scripts recognize and expand. This prevents command-line length errors while maintaining the same functional interface for the C++ calling code.

Can PythonRunner execute Python code from strings rather than files?

Yes, the run_code() method accepts a QString containing Python source code. It creates a temporary .py file, executes it through the same concurrency and environment preparation pipeline as regular scripts, and deletes the temporary file upon completion. This method always routes to the NumPy 2 virtual environment for maximum compatibility with modern analytics libraries.

How does the C++ side receive results from Python scripts?

Execution is asynchronous via callbacks. The run() method accepts a std::function<void(const PythonResult&)> callback that receives a struct containing success, output, error, and exit_code. The extract_json() function parses stdout to find JSON blocks, allowing C++ analytics nodes in AnalyticsNodes.cpp to deserialize results directly into QJsonDocument objects without manual string parsing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}