# Error Handling Mechanisms for Stack Overflow and Stack Underflow in NanoCore

> Discover NanoCore's robust error handling for stack overflow and underflow. Learn how it protects your emulator with precise pointer checks and typed emulator errors before critical instructions.

- Repository: [Afaan Bilal/nanocore](https://github.com/afaanbilal/nanocore)
- Tags: deep-dive
- Published: 2026-02-23

---

**NanoCore detects stack violations by checking the stack pointer against fixed bounds (0xEA minimum and 0xFF maximum) before every PUSH, POP, CALL, CALLR, and RET instruction, returning typed `EmulatorError` variants that preserve the offending pointer value without modifying CPU state.**

NanoCore is a compact 8-bit emulator implemented in Rust that manages program execution within a strict 256-byte memory model. Understanding the error handling mechanisms for stack overflow and stack underflow in NanoCore is essential for writing robust assembly programs and debugging runtime failures. The system employs compile-time constants and runtime guards to ensure stack operations remain within the reserved memory region.

## How NanoCore Defines Stack Boundaries

The NanoCore CPU treats the stack as a fixed region within the 256-byte address space. In [`src/cpu.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/cpu.rs) (lines 54-56), the implementation defines two critical constants:

- **`STACK_MAX = 0xFF`** — The initial stack pointer value representing the top of memory.
- **`STACK_MIN = 0xEA`** (decimal 234) — The lowest valid address for stack data, reserving 22 bytes for stack operations.

The stack pointer (`SP`) grows downwards from `0xFF` toward `0xEA`. Any operation that would move `SP` below `STACK_MIN` or above `STACK_MAX` triggers an immediate error.

## Error Variants for Stack Violations

NanoCore uses Rust’s type system to provide explicit error variants defined in [`src/error.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/error.rs) (lines 31-34). When the emulator detects an invalid stack operation, it aborts the current cycle and returns one of two specific errors:

- **`EmulatorError::StackOverflow { sp }`** — Raised when a **PUSH**, **CALL**, or **CALLR** instruction attempts to decrement the stack pointer below `STACK_MIN`. The error carries the offending `sp` value for debugging.
- **`EmulatorError::StackUnderflow { sp }`** — Raised when a **POP** or **RET** instruction attempts to increment the stack pointer above `STACK_MAX`. This indicates an attempt to read from an empty stack.

These errors propagate through the public API methods `NanoCore::cycle()` and `NanoCore::run()`, allowing calling code to handle violations gracefully via pattern matching.

## Runtime Detection in Instruction Execution

The emulator performs boundary checks immediately before modifying the stack pointer or memory. Each stack-manipulating instruction contains a guard condition that validates the current `SP` value.

### PUSH and Stack Overflow

In [`src/nanocore.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/nanocore.rs) (lines 413-416), the **PUSH** instruction implementation checks:

```rust
if self.cpu.sp == CPU::STACK_MIN {
    return Err(EmulatorError::StackOverflow { sp: self.cpu.sp });
}

```

If the stack pointer has already reached the minimum bound, the operation aborts before writing to memory.

### POP and Stack Underflow

The **POP** instruction in [`src/nanocore.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/nanocore.rs) (lines 335-337) validates the upper bound:

```rust
if self.cpu.sp == CPU::STACK_MAX {
    return Err(EmulatorError::StackUnderflow { sp: self.cpu.sp });
}

```

This prevents reading from addresses above the stack region.

### CALL and CALLR Operations

Subroutine calls use the stack to store return addresses. Both **CALL** (lines 733-735) and **CALLR** (lines 755-757) in [`src/nanocore.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/nanocore.rs) perform the same overflow check as **PUSH**, ensuring sufficient space exists to save the program counter before jumping to the subroutine.

### RET Instruction

The **RET** instruction in [`src/nanocore.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/nanocore.rs) (lines 670-672) checks for underflow before attempting to pop the return address. If the stack is empty (SP at `0xFF`), the error returns immediately.

When any guard condition fails, the emulator **does not modify the program counter or registers**, preserving CPU state for inspection and recovery.

## Practical Examples

### Triggering Stack Overflow

The following Rust code demonstrates how excessive push operations trigger the overflow error:

```rust
use nanocore::{EmulatorError, nanocore::NanoCore};

fn overflow_demo() -> Result<(), EmulatorError> {
    // Create a program that pushes 10 values onto the stack.
    // The stack pointer starts at 0xFF; we set it to 0xEB (just above the minimum)
    // so the 10th PUSH will hit STACK_MIN (0xEA) and overflow.
    let mut prog = vec![];
    for _ in 0..10 {
        prog.push(0x07); // PUSH opcode
        prog.push(0x00); // operand: R0
    }
    prog.push(0x00); // HLT

    let mut nano = NanoCore::new();
    nano.load_program(&prog, 0)?;
    nano.cpu.sp = 0xEB; // one byte above STACK_MIN

    // Run until the overflow occurs.
    loop {
        match nano.cycle() {
            Ok(_) => continue,
            Err(EmulatorError::StackOverflow { sp }) => {
                println!("Stack overflow detected at SP={:#04X}", sp);
                return Ok(());
            }
            Err(e) => panic!("Unexpected error: {:?}", e),
        }
    }
}

```

Running `overflow_demo()` outputs:

```

Stack overflow detected at SP=0xEA

```

The test suite validates this behavior in [`tests/error_handling_tests.rs`](https://github.com/afaanbilal/nanocore/blob/main/tests/error_handling_tests.rs) (lines 31-38).

### Triggering Stack Underflow

This example shows how returning without a corresponding call creates an underflow:

```rust
use nanocore::{EmulatorError, nanocore::NanoCore};

fn underflow_demo() -> Result<(), EmulatorError> {
    // Assemble a single RET instruction (opcode 0x0F).
    // Without a preceding CALL the stack pointer stays at its initial value (0xFF),
    // so RET will try to pop from an empty stack.
    let prog = vec![0x0F, 0x00, 0x00]; // RET, followed by HLT for safety

    let mut nano = NanoCore::new();
    nano.load_program(&prog, 0)?;
    match nano.cycle() {
        Err(EmulatorError::StackUnderflow { sp }) => {
            println!("Stack underflow detected at SP={:#04X}", sp);
            Ok(())
        }
        other => panic!("Expected underflow, got: {:?}", other),
    }
}

```

Output:

```

Stack underflow detected at SP=0xFF

```

The corresponding unit test resides in [`tests/error_handling_tests.rs`](https://github.com/afaanbilal/nanocore/blob/main/tests/error_handling_tests.rs) (lines 47-58).

## Summary

- **NanoCore** implements a fixed 256-byte memory model with the stack occupying addresses `0xEA` to `0xFF`.
- **Stack overflow** occurs when `PUSH`, `CALL`, or `CALLR` instructions attempt to decrement the stack pointer below `STACK_MIN` (0xEA), triggering `EmulatorError::StackOverflow`.
- **Stack underflow** occurs when `POP` or `RET` instructions attempt to increment the stack pointer above `STACK_MAX` (0xFF), triggering `EmulatorError::StackUnderflow`.
- All boundary checks reside in [`src/nanocore.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/nanocore.rs) and prevent CPU state modification when violations are detected, enabling safe error recovery through Rust’s Result type.

## Frequently Asked Questions

### What is the maximum stack size in NanoCore?

The maximum stack size is **22 bytes**, occupying addresses `0xEA` through `0xFF` in the 256-byte address space. The lower bound is defined as `STACK_MIN = 0xEA` in [`src/cpu.rs`](https://github.com/afaanbilal/nanocore/blob/main/src/cpu.rs), while `STACK_MAX` remains at `0xFF`.

### How does NanoCore prevent stack corruption?

NanoCore prevents corruption by validating the stack pointer **before** executing any stack-modifying instruction. If a `PUSH` or `CALL` would move `SP` below `STACK_MIN`, or a `POP` or `RET` would move it above `STACK_MAX`, the emulator immediately returns an `EmulatorError` without writing to memory or updating registers.

### Can I catch stack errors programmatically?

Yes. Both `NanoCore::cycle()` and `NanoCore::run()` return `Result` types containing `EmulatorError` variants. You can pattern match on `EmulatorError::StackOverflow { sp }` or `EmulatorError::StackUnderflow { sp }` to implement recovery logic or graceful shutdowns in your host application.

### What happens to the CPU state when a stack error occurs?

When either overflow or underflow is detected, the emulator **aborts the current instruction without modifying the program counter or any registers**. The stack pointer remains at its original value, and memory contents are unchanged. This atomic failure mode allows developers to inspect the exact machine state at the moment of violation.