# How the scale_to_fit Function in Chandra Prepares Images for Model Input

> Understand how Chandra's scale_to_fit function prepares images for model input by normalizing size constraints, aligning to a 28x28 grid, and preserving aspect ratios for vision-language models.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: how-to-guide
- Published: 2026-03-27

---

**The `scale_to_fit` function normalizes arbitrary input images to meet the strict size constraints of Chandra's vision-language models by enforcing pixel count limits, aligning dimensions to a 28×28 patch grid, and preserving aspect ratios during intelligent resizing.**

The `scale_to_fit` function serves as the canonical preprocessing layer in the `datalab-to/chandra` repository, ensuring that both VLLM and Hugging Face backends receive optimally sized inputs. Located in [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py), this pure utility abstracts model-specific size constraints away from higher-level inference pipelines, guaranteeing GPU-safe memory usage while maintaining visual fidelity.

## Enforcing Pixel Count Boundaries

The function first validates the image against configurable **minimum** and **maximum** pixel thresholds to prevent out-of-memory errors and ensure sufficient visual detail.

### Maximum Size Protection

If an image exceeds `max_size` (defaulting to 3072 × 2048 pixels, approximately 6 megapixels), `scale_to_fit` down-scales the image to fit within these bounds. This protects GPU memory from being exhausted by high-resolution inputs that exceed the model's training distribution.

### Minimum Size Guarantee

Conversely, images smaller than `min_size` (defaulting to 1792 × 28 pixels, approximately 50 kilopixels) are up-scaled to ensure the vision transformer can extract meaningful visual features. This prevents the model from processing thumbnails too tiny to contain useful information.

## Grid Alignment for Vision Transformers

Chandra's models operate on a fixed **28 × 28 pixel patch grid**, standard for many vision transformers. After initial scaling, `scale_to_fit` rounds the image dimensions to the nearest multiple of `grid_size` (default 28) to prevent partial patches that would corrupt positional embeddings.

The calculation determines the number of patch blocks in each dimension:

```python
w_blocks = max(1, round((width * scale) / grid_size))
h_blocks = max(1, round((height * scale) / grid_size))

```

The final output dimensions become `w_blocks * grid_size` and `h_blocks * grid_size`, ensuring perfect alignment with the model's patch processing requirements.

## Aspect Ratio Preservation Algorithm

A refinement loop ensures the grid-aligned image does not exceed the `max_pixels` ceiling while minimizing distortion to the original aspect ratio. The algorithm iteratively removes one block from either width or height, selecting the direction that yields the smallest aspect ratio deviation:

```python
ar_w_loss = abs(((w_blocks - 1) / h_blocks) - original_ar)
ar_h_loss = abs((w_blocks / (h_blocks - 1)) - original_ar)
if ar_w_loss < ar_h_loss:
    w_blocks -= 1
else:
    h_blocks -= 1

```

This loop terminates when the image fits within `max_pixels` or when dimensions can no longer shrink without violating minimum constraints. The process guarantees the output maintains the closest possible proportions to the source image while respecting hardware limits.

## Resampling and No-Op Optimization

When resizing is required, `scale_to_fit` applies the high-quality **Lanczos filter** (`Image.Resampling.LANCZOS`) from Pillow to preserve edge clarity and reduce aliasing artifacts. If the computed dimensions match the original image exactly, the function returns the original `PIL.Image` object unchanged, avoiding unnecessary memory copies and computational overhead.

## Integration with Chandra Inference Backends

Both inference pipelines in Chandra rely on `scale_to_fit` as the final preprocessing step before model ingestion, centralizing image normalization logic in a single utility.

### VLLM Backend Usage

In [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py), the function prepares images before base-64 encoding for API transmission:

```python
from chandra.model.util import scale_to_fit

image = scale_to_fit(item.image)
image_b64 = image_to_base64(image)

```

The resized image is embedded in the payload sent to the VLLM endpoint, ensuring the remote model receives a grid-aligned, memory-safe input.

### Hugging Face Backend Usage

Similarly, [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py) invokes the utility when building conversation payloads for local Hugging Face processors:

```python
from chandra.model.util import scale_to_fit

image = scale_to_fit(item.image)  # Guarantee max size

content.append({"type": "image", "image": image})

```

This unified approach means any future adjustments to patch sizes or memory constraints require modifications only in [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py), propagating automatically to both backends.

## Practical Implementation Examples

### Basic Image Normalization

Use `scale_to_fit` directly on any PIL Image to produce model-ready dimensions:

```python
from PIL import Image
from chandra.model.util import scale_to_fit

raw = Image.open("document.png")
model_ready = scale_to_fit(raw)

# Verify grid alignment (multiples of 28)

assert model_ready.width % 28 == 0
assert model_ready.height % 28 == 0

```

### Batch Inference with VLLM

The function integrates seamlessly into batch processing workflows:

```python
from chandra.model.vllm import generate_vllm
from chandra.model.schema import BatchInputItem
from PIL import Image

batch = [
    BatchInputItem(
        image=Image.open("chart.png"),
        prompt="Describe this visualization.",
        prompt_type="default"
    )
]

results = generate_vllm(batch)  # scale_to_fit called internally

print(results[0].raw)

```

### Local Hugging Face Pipeline

For local model execution, preprocessing occurs during item preparation:

```python
from chandra.model.hf import generate_hf, load_model
from chandra.model.schema import BatchInputItem
from PIL import Image

model = load_model()
batch = [
    BatchInputItem(
        image=Image.open("screenshot.png"),
        prompt="What UI element is focused?",
        prompt_type="default"
    )
]

outputs = generate_hf(batch, model)  # Internal scale_to_fit ensures compliance

print(outputs[0].raw)

```

## Summary

- **The `scale_to_fit` function in [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py) enforces configurable pixel count limits** (default max ~6 MP, min ~50 kpx) to balance GPU memory safety against visual information preservation.
- **It aligns all outputs to a 28 × 28 patch grid** through block-based rounding, preventing partial patch errors in vision transformer positional embeddings.
- **An aspect-ratio-preserving refinement loop** intelligently reduces dimensions when necessary, prioritizing minimal distortion over strict pixel limits.
- **The utility is backend-agnostic**, serving both VLLM and Hugging Face pipelines from a single implementation point to ensure consistent preprocessing across the entire `datalab-to/chandra` codebase.
- **High-quality Lanczos resampling** maintains image fidelity, while a no-op optimization eliminates unnecessary processing for already-conforming inputs.

## Frequently Asked Questions

### What is the default grid size used by scale_to_fit?

The default `grid_size` is **28 pixels**, corresponding to the standard patch dimensions used by Chandra's underlying vision transformers. This value ensures compatibility with the 28 × 28 patch grid expected by the model's positional embedding layers.

### Does scale_to_fit mutate the original image?

No, `scale_to_fit` is implemented as a **pure function** that does not mutate the input `PIL.Image` object. It either returns the original image unchanged (if dimensions already conform) or a new resampled copy, leaving the source image intact for downstream reuse.

### Which resampling filter does scale_to_fit use?

When resizing is required, the function applies **Pillow's Lanczos filter** (`Image.Resampling.LANCZOS`). This high-quality algorithm minimizes aliasing and preserves edge sharpness better than bilinear or bicubic alternatives, maintaining visual fidelity critical for document and chart understanding tasks.

### Where is scale_to_fit called in the Chandra pipeline?

According to the source code in `datalab-to/chandra`, `scale_to_fit` is invoked in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) before base-64 encoding for remote inference, and in [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py) when constructing input batches for local Hugging Face processors. This centralized usage ensures all model inputs pass through identical normalization logic regardless of backend choice.