How the scale_to_fit Function in Chandra Prepares Images for Model Input

The scale_to_fit function normalizes arbitrary input images to meet the strict size constraints of Chandra's vision-language models by enforcing pixel count limits, aligning dimensions to a 28×28 patch grid, and preserving aspect ratios during intelligent resizing.

The scale_to_fit function serves as the canonical preprocessing layer in the datalab-to/chandra repository, ensuring that both VLLM and Hugging Face backends receive optimally sized inputs. Located in chandra/model/util.py, this pure utility abstracts model-specific size constraints away from higher-level inference pipelines, guaranteeing GPU-safe memory usage while maintaining visual fidelity.

Enforcing Pixel Count Boundaries

The function first validates the image against configurable minimum and maximum pixel thresholds to prevent out-of-memory errors and ensure sufficient visual detail.

Maximum Size Protection

If an image exceeds max_size (defaulting to 3072 × 2048 pixels, approximately 6 megapixels), scale_to_fit down-scales the image to fit within these bounds. This protects GPU memory from being exhausted by high-resolution inputs that exceed the model's training distribution.

Minimum Size Guarantee

Conversely, images smaller than min_size (defaulting to 1792 × 28 pixels, approximately 50 kilopixels) are up-scaled to ensure the vision transformer can extract meaningful visual features. This prevents the model from processing thumbnails too tiny to contain useful information.

Grid Alignment for Vision Transformers

Chandra's models operate on a fixed 28 × 28 pixel patch grid, standard for many vision transformers. After initial scaling, scale_to_fit rounds the image dimensions to the nearest multiple of grid_size (default 28) to prevent partial patches that would corrupt positional embeddings.

The calculation determines the number of patch blocks in each dimension:

w_blocks = max(1, round((width * scale) / grid_size))
h_blocks = max(1, round((height * scale) / grid_size))

The final output dimensions become w_blocks * grid_size and h_blocks * grid_size, ensuring perfect alignment with the model's patch processing requirements.

Aspect Ratio Preservation Algorithm

A refinement loop ensures the grid-aligned image does not exceed the max_pixels ceiling while minimizing distortion to the original aspect ratio. The algorithm iteratively removes one block from either width or height, selecting the direction that yields the smallest aspect ratio deviation:

ar_w_loss = abs(((w_blocks - 1) / h_blocks) - original_ar)
ar_h_loss = abs((w_blocks / (h_blocks - 1)) - original_ar)
if ar_w_loss < ar_h_loss:
    w_blocks -= 1
else:
    h_blocks -= 1

This loop terminates when the image fits within max_pixels or when dimensions can no longer shrink without violating minimum constraints. The process guarantees the output maintains the closest possible proportions to the source image while respecting hardware limits.

Resampling and No-Op Optimization

When resizing is required, scale_to_fit applies the high-quality Lanczos filter (Image.Resampling.LANCZOS) from Pillow to preserve edge clarity and reduce aliasing artifacts. If the computed dimensions match the original image exactly, the function returns the original PIL.Image object unchanged, avoiding unnecessary memory copies and computational overhead.

Integration with Chandra Inference Backends

Both inference pipelines in Chandra rely on scale_to_fit as the final preprocessing step before model ingestion, centralizing image normalization logic in a single utility.

VLLM Backend Usage

In chandra/model/vllm.py, the function prepares images before base-64 encoding for API transmission:

from chandra.model.util import scale_to_fit

image = scale_to_fit(item.image)
image_b64 = image_to_base64(image)

The resized image is embedded in the payload sent to the VLLM endpoint, ensuring the remote model receives a grid-aligned, memory-safe input.

Hugging Face Backend Usage

Similarly, chandra/model/hf.py invokes the utility when building conversation payloads for local Hugging Face processors:

from chandra.model.util import scale_to_fit

image = scale_to_fit(item.image)  # Guarantee max size

content.append({"type": "image", "image": image})

This unified approach means any future adjustments to patch sizes or memory constraints require modifications only in chandra/model/util.py, propagating automatically to both backends.

Practical Implementation Examples

Basic Image Normalization

Use scale_to_fit directly on any PIL Image to produce model-ready dimensions:

from PIL import Image
from chandra.model.util import scale_to_fit

raw = Image.open("document.png")
model_ready = scale_to_fit(raw)

# Verify grid alignment (multiples of 28)

assert model_ready.width % 28 == 0
assert model_ready.height % 28 == 0

Batch Inference with VLLM

The function integrates seamlessly into batch processing workflows:

from chandra.model.vllm import generate_vllm
from chandra.model.schema import BatchInputItem
from PIL import Image

batch = [
    BatchInputItem(
        image=Image.open("chart.png"),
        prompt="Describe this visualization.",
        prompt_type="default"
    )
]

results = generate_vllm(batch)  # scale_to_fit called internally

print(results[0].raw)

Local Hugging Face Pipeline

For local model execution, preprocessing occurs during item preparation:

from chandra.model.hf import generate_hf, load_model
from chandra.model.schema import BatchInputItem
from PIL import Image

model = load_model()
batch = [
    BatchInputItem(
        image=Image.open("screenshot.png"),
        prompt="What UI element is focused?",
        prompt_type="default"
    )
]

outputs = generate_hf(batch, model)  # Internal scale_to_fit ensures compliance

print(outputs[0].raw)

Summary

• The scale_to_fit function in chandra/model/util.py enforces configurable pixel count limits (default max ~6 MP, min ~50 kpx) to balance GPU memory safety against visual information preservation.
• It aligns all outputs to a 28 × 28 patch grid through block-based rounding, preventing partial patch errors in vision transformer positional embeddings.
• An aspect-ratio-preserving refinement loop intelligently reduces dimensions when necessary, prioritizing minimal distortion over strict pixel limits.
• The utility is backend-agnostic, serving both VLLM and Hugging Face pipelines from a single implementation point to ensure consistent preprocessing across the entire datalab-to/chandra codebase.
• High-quality Lanczos resampling maintains image fidelity, while a no-op optimization eliminates unnecessary processing for already-conforming inputs.

Frequently Asked Questions

What is the default grid size used by scale_to_fit?

The default grid_size is 28 pixels, corresponding to the standard patch dimensions used by Chandra's underlying vision transformers. This value ensures compatibility with the 28 × 28 patch grid expected by the model's positional embedding layers.

Does scale_to_fit mutate the original image?

No, scale_to_fit is implemented as a pure function that does not mutate the input PIL.Image object. It either returns the original image unchanged (if dimensions already conform) or a new resampled copy, leaving the source image intact for downstream reuse.

Which resampling filter does scale_to_fit use?

When resizing is required, the function applies Pillow's Lanczos filter (Image.Resampling.LANCZOS). This high-quality algorithm minimizes aliasing and preserves edge sharpness better than bilinear or bicubic alternatives, maintaining visual fidelity critical for document and chart understanding tasks.

Where is scale_to_fit called in the Chandra pipeline?

According to the source code in datalab-to/chandra, scale_to_fit is invoked in chandra/model/vllm.py before base-64 encoding for remote inference, and in chandra/model/hf.py when constructing input batches for local Hugging Face processors. This centralized usage ensures all model inputs pass through identical normalization logic regardless of backend choice.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}