Optimize Memory Usage When Processing High-Resolution Images in mcp_3d_relief

The most effective way to reduce memory consumption when using mcp_3d_relief is to downsample images with Pillow before converting them to NumPy arrays, process depth maps in tiles rather than full resolution, and explicitly delete large objects immediately after use.

The mcp_3d_relief repository converts photographs into 3D depth maps and STL files, but its default pipeline loads entire high-resolution images into RAM as Pillow Image objects, converts them to NumPy arrays, and allocates additional arrays for depth calculations. When processing 4K or larger images, this approach can quickly exhaust available memory on modest hardware. By restructuring the data flow in relief.py and adopting block-wise processing techniques, you can reduce peak memory usage from hundreds of megabytes to a manageable footprint without sacrificing output quality.

Identify Memory Hotspots in the Pipeline

The current implementation in relief.py creates several full-resolution arrays in sequence. First, the image is loaded and converted to grayscale using np.array(input_image.convert("L")) at lines 18–21, creating a complete array even if the final output requires only a small fraction of that resolution. Next, cv2.resize processes this full array at lines 22–24 to match the desired detail level. Finally, generate_depth_map() constructs a separate depth map array and applies cv2.GaussianBlur to the entire image at line 47, temporarily doubling memory requirements during the blur operation. The STL generation phase then allocates a vertices matrix holding floating-point coordinates for every pixel, further increasing RAM pressure.

Downsample Before Converting to NumPy

The most impactful optimization is resizing the image while it remains in Pillow's native format, avoiding the allocation of a large intermediate array entirely. The current code converts the full image to a NumPy array first, then resizes it, which means the full-resolution array persists in memory even though only the downsampled version is needed.

Replace the logic at lines 18–24 in relief.py with the following approach:

import numpy as np
from PIL import Image

# Calculate target dimensions based on detail_level parameter

max_dim = int(320 * detail_level)

# Resize with Pillow while still in Image format

input_image.thumbnail((max_dim, max_dim), Image.BICUBIC)

# Now convert the already-small image to NumPy array

img = np.array(input_image, dtype=np.uint8)

This modification ensures that img enters memory at the final target resolution rather than the original camera resolution, typically reducing memory usage by 90% or more for 4K+ sources.

Minimize Intermediate Float Copies

The depth map generation currently promotes the entire array to 64-bit floating point during the gamma correction step at lines 40–42: np.power(gray / 255.0, 1.5) * 255. This operation creates a temporary float64 array that doubles memory consumption for the duration of the calculation.

Optimize this section in generate_depth_map() by using 32-bit precision and eliminating temporary copies:

# Replace: gray = np.power(gray / 255.0, 1.5) * 255

gray = (gray.astype(np.float32) / 255.0) ** 1.5
gray = (gray * 255).astype(np.uint8)

Using float32 instead of float64 cuts the temporary buffer size in half, and converting immediately back to uint8 prevents the floating-point array from persisting into subsequent operations.

Implement Tile-Based Processing

Instead of applying Gaussian blur to the entire depth map simultaneously, which requires holding the complete image and blur kernel in memory, process the image in smaller blocks. This reduces peak memory usage from O(width × height) to O(tile_size).

Add the following helper function to relief.py and modify line 47 to use it:

def tile_blur(img: np.ndarray, ksize: int = 5, sigma: float = 1.5, tile: int = 256):
    h, w = img.shape
    out = np.empty_like(img)
    for y in range(0, h, tile):
        for x in range(0, w, tile):
            y_end = min(y + tile, h)
            x_end = min(x + tile, w)
            out[y:y_end, x:x_end] = cv2.GaussianBlur(
                img[y:y_end, x:x_end], (ksize, ksize), sigma
            )
    return out

# Replace line 47:

depth_map = tile_blur(gray, ksize=5, sigma=1.5)

A tile size of 256 pixels provides an optimal balance between memory reduction and computational efficiency, as OpenCV's blur operations remain cache-friendly at this scale.

Explicitly Release Large Objects

Python's garbage collector does not immediately reclaim memory when variables go out of scope; they persist until reference counting drops to zero. After writing the STL file in relief(), the depth_map and vertices arrays consume significant RAM but remain allocated until the function returns.

Insert explicit cleanup immediately before the return statement at lines 48–51:

# After STL file has been written successfully

del depth_map, vertices
import gc
gc.collect()

This pattern forces immediate deallocation of the largest buffers, making RAM available for subsequent image processing tasks without waiting for the garbage collector's next cycle.

Stream Large Downloads

When fetching images from URLs, the FastAPI wrapper in server.py currently loads the entire payload into memory using await response.read() at lines 85–92. For high-resolution images, this creates an unnecessary copy of the data before Pillow even opens it.

Replace the download logic with streaming to a temporary file:

import aiofiles
from aiohttp import ClientSession

async with ClientSession() as session:
    async with session.get(input_image_path) as resp:
        if resp.status != 200:
            raise ValueError(f"Failed to download: {resp.status}")
        
        async with aiofiles.tempfile.NamedTemporaryFile(delete=False, suffix=".png") as tmp:
            async for chunk in resp.content.iter_chunked(1024 * 64):
                await tmp.write(chunk)
            tmp_path = tmp.name
        
        input_image = Image.open(tmp_path)

This approach writes data directly to disk in 64KB chunks, maintaining constant memory usage regardless of file size. Once Image.open() completes, the temporary file can be deleted to reclaim disk space.

Summary

• Downsample early: Use Pillow's thumbnail() or resize() before np.array() conversion to avoid allocating full-resolution arrays in relief.py lines 18–24.
• Reduce precision: Cast to float32 instead of float64 during gamma correction in generate_depth_map() to halve temporary memory usage.
• Tile processing: Replace the global cv2.GaussianBlur at line 47 with a tile_blur() helper that processes 256-pixel blocks, reducing peak memory from full-image size to tile size.
• Immediate cleanup: Insert del depth_map, vertices followed by gc.collect() after STL generation to force memory release before returning.
• Stream inputs: Modify server.py lines 85–92 to use aiofiles and iter_chunked instead of response.read() for constant-memory downloads.

Frequently Asked Questions

Why does mcp_3d_relief run out of memory on 4K images?

The pipeline creates three separate full-resolution arrays: the initial Pillow-to-NumPy conversion, the depth map buffer, and the vertices matrix for STL generation. At 4K resolution (3840×2160), a single grayscale array consumes approximately 8MB, while RGB images require 24MB or more. Combined with intermediate copies during blurring and floating-point conversions, peak memory can exceed 200MB per image, which exceeds limits on constrained environments like small VPS instances or shared hosting.

Is OpenCV or Pillow better for memory efficiency in this pipeline?

Pillow is superior for the initial loading and resizing stages because it operates on compressed image formats without requiring full decompression into numpy arrays first. However, OpenCV's cv2.IMREAD_GRAYSCALE flag can save memory when you know the input is already grayscale, as it skips RGB conversion entirely. For general use, stick with Pillow for resizing and OpenCV for tile-based filtering operations.

How much memory can tile-based processing save?

Tile-based processing reduces peak memory usage from O(width × height) to O(tile_size × tile_size). For a 4096×4096 image processed with 256×256 tiles, peak memory drops from approximately 64MB (full float32 array plus blur buffers) to roughly 1MB (single tile buffers). This 64x reduction enables processing of gigapixel images on machines with limited RAM.

Should I use Python's garbage collector or manual deletion for large arrays?

Explicit deletion using del followed by gc.collect() is recommended for immediate relief when processing sequential large files. While CPython's reference counting automatically frees memory when objects go out of scope, the gc.collect() call ensures that any circular references or delayed cleanup in third-party libraries (like NumPy or OpenCV) are resolved immediately, preventing memory accumulation during batch processing jobs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}