Monty's Safety Mechanisms for Memory Management: Reference Counting and RAII in a Sandboxed Rust VM

Monty guarantees safe memory management through a deterministic reference-counting system hardened by RAII guards, explicit cleanup macros, and resource limits, eliminating garbage collection cycles while preventing leaks, double-free, and use-after-free vulnerabilities in untrusted Python code.

Monty is a sandboxed Python runtime developed by Pydantic that executes untrusted code inside a Rust-based virtual machine. Unlike CPython, which relies on cyclic garbage collection, Monty implements explicit safety mechanisms for memory management that combine manual reference counting with compile-time cleanup guarantees to ensure deterministic resource deallocation and complete isolation from host system resources.

1. Reference-Counted Heap Architecture

Monty stores every heap-allocated Python object in a dense vector of HeapEntry structures managed by the Heap type in crates/monty/src/heap.rs. Each entry tracks a per-object reference count and the actual Python value data, enabling immediate deallocation when objects become unreachable.

HeapValue Structure and Reference Tracking

The core HeapValue struct maintains the reference count alongside the object data:

struct HeapValue {
    refcount: usize,          // reference count
    data: Option<HeapData>,   // actual Python value
    hash_state: Option<u64>,  // cached hash (lazy)
}

When an object is allocated via heap.alloc(), it enters the heap with refcount = 1. The Heap::inc_ref method (lines 1075–1087) increments this counter when a new reference is cloned, while Heap::dec_ref (lines 1098–1119) handles decrementing and potential deallocation.

Recursive Cleanup on Deallocation

When dec_ref detects that the count has reached zero, it immediately frees the slot and pushes the ID onto a free-list for reuse. Crucially, it then recursively invokes dec_ref on all child object IDs contained within the freed value (see the loop starting at line 1117). This ensures that complex object graphs—such as nested lists or dictionaries—are deallocated deterministically without requiring a separate garbage collection cycle.

The dec_ref implementation includes defensive checks that panic on invalid or already-freed slots, catching double-free and use-after-free errors during development and in the "ref-count-panic" test mode.

2. RAII Guards for Guaranteed Cleanup

Manual calls to inc_ref and dec_ref are error-prone when control flow includes early returns, ? propagation, or panics. Monty solves this through the DropWithHeap trait and HeapGuard struct, both defined in crates/monty/src/heap.rs.

The DropWithHeap Trait

Any container that may own heap references implements DropWithHeap:

pub(crate) trait DropWithHeap {
    fn drop_with_heap<T: ResourceTracker>(self, heap: &mut Heap<T>);
}

This method consumes the value and explicitly decrements all contained heap references. Primitive Value variants, Option<V>, Vec<V>, and tuples all implement this trait, ensuring that any composite structure can be safely decomposed.

HeapGuard Implementation

HeapGuard (lines 1170–1205) wraps a ManuallyDrop<V> and a mutable borrow of the heap. Its Drop implementation simply calls value.drop_with_heap(heap). Because HeapGuard lives on the stack, Rust guarantees its destructor runs on every exit path—including unwinding from panics—preventing reference leaks regardless of control flow complexity.

3. Ergonomic Safety with defer_drop! Macros

While HeapGuard provides safety, manually constructing guards is verbose. Monty exposes the defer_drop! and defer_drop_mut! macros (starting at line 2059) to provide zero-cost ergonomic wrappers.

These macros perform three operations atomically:

1. Create a HeapGuard for the supplied value and heap.
2. Re-bind the original identifiers to borrowed references (&V and &mut Heap).
3. Ensure the guard's Drop implementation runs automatically when the surrounding block ends.

defer_drop!(my_list, heap);

After this invocation, my_list is available as &List and heap as &mut Heap, while the underlying guard ensures that if the function returns early or panics, my_list is properly deallocated via drop_with_heap.

For mutable access patterns, such as iterating and modifying a list:

defer_drop_mut!(my_list, heap);
// my_list is now &mut List
my_list.push(Value::Int(42), heap)?;

This macro system eliminates the risk of forgetting manual cleanup calls while maintaining the performance characteristics of manual memory management.

4. Resource Limits and Allocation Tracking

Even with perfect reference counting, a malicious script could exhaust host memory by allocating millions of objects. Monty attaches a ResourceTracker to every heap instance to enforce hard limits.

The tracker monitors:

• Allocation count – total number of live objects
• Memory usage – estimated bytes consumed
• Recursion depth – call stack depth to prevent stack overflow

On each allocation, ResourceTracker::on_alloc checks against configurable limits. If a script exceeds these limits, the operation returns ResourceError::Allocation, which propagates as a Python MemoryError (defined in crates/monty/src/resource.rs, lines 80–96).

This mechanism terminates runaway scripts deterministically before they can impact host stability, providing a second line of defense beyond reference counting.

5. Host-Side Sandboxing

While the reference-counting system manages Python object lifecycles, Monty prevents external memory leaks through host resource isolation. The sandbox explicitly blocks all OS-level operations that could allocate external resources.

In crates/monty/src/os.rs and io.rs, system calls are replaced with thin wrappers that immediately raise PermissionError. As noted in the source header (lines 4–9), these modules ensure that "the interpreter never calls arbitrary host code," preventing file descriptor leaks, network socket exhaustion, or subprocess memory consumption.

Combined with ResourceTracker, this creates a defense-in-depth strategy: even if a script attempts to exploit Python-level allocation patterns, it cannot access host-level resources to amplify the attack.

Summary

Monty's safety mechanisms for memory management create a deterministic, branch-safe model that eliminates traditional garbage collection while preventing common memory errors:

• Explicit reference counting via Heap::inc_ref and Heap::dec_ref in crates/monty/src/heap.rs ensures immediate deallocation when objects become unreachable, with recursive cleanup of child references.
• RAII guards implemented through the DropWithHeap trait and HeapGuard struct guarantee cleanup on all exit paths, including panics and early returns.
• Ergonomic macros defer_drop! and defer_drop_mut! provide zero-cost abstractions that prevent developers from forgetting manual cleanup calls.
• Resource limits enforced by ResourceTracker prevent denial-of-service attacks through excessive allocation, converting limit breaches into Python MemoryError exceptions.
• Host sandboxing in os.rs and io.rs blocks external resource access, ensuring memory leaks cannot propagate to the host operating system.

Together, these mechanisms enable secure execution of untrusted Python code without relying on a traditional garbage collector.

Frequently Asked Questions

How does Monty prevent memory leaks without a garbage collector?

Monty uses explicit reference counting where every heap object tracks a refcount field in crates/monty/src/heap.rs. When references are cloned, Heap::inc_ref increments the count; when they go out of scope, Heap::dec_ref decrements it. When the count reaches zero, the object is immediately freed and recursively decrements its children. This deterministic approach ensures memory is reclaimed the moment it becomes unreachable, unlike tracing garbage collectors that require pause cycles.

What prevents double-free or use-after-free errors in Monty's heap?

The Heap::dec_ref implementation includes defensive checks that panic on invalid or already-freed slots, catching double-free attempts during development. Additionally, the DropWithHeap trait and HeapGuard struct ensure that cleanup only occurs through controlled RAII patterns. Because HeapGuard uses ManuallyDrop and guarantees execution via Rust's drop semantics, it prevents use-after-free by ensuring references remain valid until the guard's scope ends, even across panic unwinding.

How does Monty handle resource exhaustion attacks?

Monty attaches a ResourceTracker to every heap instance that monitors allocation count, estimated memory usage, and recursion depth. On every allocation, ResourceTracker::on_alloc checks against configurable limits. If a script exceeds these limits, the operation returns ResourceError::Allocation, which propagates as a Python MemoryError. This mechanism terminates runaway scripts deterministically before they can exhaust host memory, providing a hard ceiling on resource consumption regardless of reference-counting correctness.

What role do the defer_drop! macros play in memory safety?

The defer_drop! and defer_drop_mut! macros provide zero-cost ergonomic wrappers around HeapGuard. They rebind values to borrowed references while creating an invisible guard that ensures drop_with_heap runs when the scope exits. This prevents developers from forgetting manual cleanup calls in complex control flow with early returns, ? propagation, or panics. By making RAII guards implicit and ergonomic, these macros eliminate a major source of reference leaks while maintaining the performance characteristics of manual memory management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}