How External Function Callbacks Work in Monty: A Complete Technical Guide

External function callbacks in Monty enable sandboxed Python code to securely invoke host Python functions through a three-phase round-trip involving registration, VM pause/yield, and host dispatch with result resumption.

Monty is a secure Python sandbox that isolates untrusted code execution from the host interpreter. While this isolation protects the host environment, legitimate use cases require external function callbacks to let sandboxed code trigger controlled actions in the host Python process. This article examines the complete callback architecture implemented in the pydantic/monty repository, from registration through asynchronous resolution.

The Three-Phase Callback Architecture

External function callbacks operate through a well-defined protocol that bridges the Monty VM (Rust) and the host Python environment. The flow consists of three core phases: registration, pause/yield, and host dispatch.

Phase 1: Registration of External Functions

Before execution begins, the host must declare which Python functions the sandboxed code is permitted to call. When creating a MontyRun instance via MontyRun::new() in crates/monty/src/run.rs, the caller supplies a list of external function names through the external_functions parameter.

The Executor stores these names in the Interns table, allowing the bytecode to reference them as ExtFunctionId identifiers. This registration creates a whitelist that the VM checks during execution to distinguish between internal functions and external callbacks.

Phase 2: Pause and Yield to Host

During execution, when the VM encounters a call to a name that is not a built-in, user-defined, or module function, it treats it as an external call. Rather than executing the function directly, the VM returns RunProgress::FunctionCall defined in crates/monty/src/run.rs.

This variant contains:

• function_name: The interned identifier of the external function
• args and kwargs: Positional and keyword arguments as Monty objects
• call_id: A unique identifier for this specific invocation
• state: A Snapshot that captures the entire interpreter state, including the heap, namespaces, and execution frames

The VM effectively freezes execution at this point, yielding control back to the host.

Phase 3: Host Dispatch and Resume

The host receives the FunctionCall notification and dispatches it through the ExternalFunctionRegistry implemented in crates/monty-python/src/external.rs. This registry maintains a mapping from function names to Python callables.

The dispatch process follows these steps:

1. Argument conversion: The host converts Monty objects to Python objects using monty_to_py
2. Invocation: The registry invokes the Python callable with the converted arguments
3. Result conversion: The return value is converted back to a Monty object using py_to_monty
4. Error handling: Python exceptions are caught and converted to Monty exceptions via exc_py_to_monty

The result is wrapped in an ExternalResult and pushed back into the VM. The host calls snapshot.run(result, writer) for synchronous resolution or snapshot.run_pending() for asynchronous deferral. The VM restores the saved state, pops the external call frame, pushes the result onto the stack, and continues execution.

Implementing External Function Callbacks in Practice

Rust Implementation

When embedding Monty in a Rust application, you handle the callback loop manually:

use monty::{MontyRun, MontyObject, RunProgress};

let code = "def greet(name):\n    return external_hello(name)\n\ngreet('Alice')";
let runner = MontyRun::new(
    code.to_owned(),
    "example.py",
    vec![],               
    vec!["external_hello".into()], 
).unwrap();

let mut printer = monty::io::PrintWriter::Stdout;
let mut progress = runner.start(vec![], monty::resource::NoLimitTracker, &mut printer).unwrap();

loop {
    match progress {
        RunProgress::FunctionCall { function_name, args, kwargs, call_id, state } => {
            let result = host_dispatch(&function_name, &args, &kwargs);
            progress = state.run(result, &mut printer).unwrap();
        }
        RunProgress::Complete(value) => {
            println!("Finished with {value:?}");
            break;
        }
        _ => unreachable!(),
    }
}

Python Host Registry

On the Python side, you provide the callable implementations:

import pydantic_monty

def external_hello(name):
    return f"Hello, {name}!"

registry = {"external_hello": external_hello}

m = pydantic_monty.MontyRun(
    code="def greet(name): return external_hello(name)\nprint(greet('Bob'))",
    input_names=[],
    external_functions=["external_hello"],
)

result = m.run_with_external(registry)
print("Monty returned:", result)

The run_with_external method handles the RunProgress iteration internally, converting objects between Monty and Python representations using monty_to_py and py_to_monty.

Asynchronous External Function Callbacks

Monty supports asynchronous external functions through the ExternalFuture mechanism. When the host cannot immediately provide a result (e.g., the external function performs I/O), it calls snapshot.run_pending() instead of snapshot.run().

This creates an ExternalFuture(CallId) value (defined in crates/monty/src/value.rs) that the VM pushes onto the stack. The VM records this as a pending future and later yields RunProgress::ResolveFutures, allowing the host to supply results for multiple pending calls via FutureSnapshot::resume.

match progress {
    RunProgress::FunctionCall { state, .. } => {
        progress = state.run_pending().unwrap();
    }
    RunProgress::ResolveFutures(future_snapshot) => {
        let results = vec![monty::MontyObject::Int(42)];
        progress = future_snapshot.resume(results, &mut printer).unwrap();
    }
    _ => {}
}

This pattern enables non-blocking I/O operations while maintaining the sandbox boundary between Monty and the host.

Key Implementation Files

The external function callback system spans several critical files in the pydantic/monty repository:

• crates/monty/src/run.rs – Contains MontyRun::new(), the RunProgress enum (including FunctionCall and ResolveFutures), and Snapshot::run()/run_pending() methods for resuming execution.

• crates/monty/src/value.rs – Defines Value::ExtFunction for external function references and Value::ExternalFuture for pending asynchronous calls.

• crates/monty-python/src/external.rs – Implements ExternalFunctionRegistry, the host-side dispatch mechanism, and conversion functions monty_to_py and py_to_monty.

• crates/monty/src/vm/call.rs – Bytecode interpreter logic that detects external calls and returns FrameExit::ExternalCall.

• crates/monty-python/src/lib.rs – Python bindings exposing MontyRun and the run_with_external convenience method.

Summary

• External function callbacks provide a secure bridge between Monty's sandboxed VM and the host Python interpreter.
• The system operates through three phases: registration of allowed functions during MontyRun creation, pause and yield via RunProgress::FunctionCall, and host dispatch through ExternalFunctionRegistry.
• Snapshot-based resumption allows the VM to freeze and restore state across the host boundary using snapshot.run() for synchronous results or snapshot.run_pending() for asynchronous futures.
• Type conversion between Monty and Python objects is handled automatically by monty_to_py and py_to_monty in the Python bindings.
• The implementation spans crates/monty/src/run.rs, crates/monty/src/value.rs, and crates/monty-python/src/external.rs.

Frequently Asked Questions

How does Monty verify which external functions are allowed?

Monty validates external functions during the registration phase in MontyRun::new(). The caller must explicitly provide a list of external function names, which the Executor stores in the Interns table as ExtFunctionId identifiers. When the VM encounters a call, it checks if the name exists in this whitelist; if not, it raises an error rather than yielding to the host.

What happens if an external function raises a Python exception?

When a Python exception occurs inside an external function, the ExternalFunctionRegistry catches it in crates/monty-python/src/external.rs and converts it to a Monty exception using exc_py_to_monty. The error is wrapped in ExternalResult::Error and returned to the VM through snapshot.run(). The VM then propagates this exception up the call stack within the sandbox, allowing standard Python exception handling to occur inside Monty.

Can external function callbacks work with async/await patterns?

Yes, Monty supports asynchronous external functions through the ExternalFuture mechanism. When the host calls snapshot.run_pending() instead of snapshot.run(), the VM creates a Value::ExternalFuture containing the call_id and yields RunProgress::ResolveFutures. The host can later resolve multiple pending futures simultaneously using FutureSnapshot::resume(), making this pattern ideal for I/O-bound operations without blocking the VM.

Where is the boundary between the Monty VM and host Python defined?

The boundary is defined in crates/monty-python/src/external.rs through the ExternalFunctionRegistry struct and the conversion functions monty_to_py and py_to_monty. On the VM side, crates/monty/src/run.rs defines the RunProgress::FunctionCall interface and Snapshot resumption logic. This clean separation ensures that object conversion and privilege checking occur entirely within the host layer before any data re-enters the sandbox.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}