Creating a Custom Crop Tool for Detailed Image Region Analysis in Claude Vision

Implement a crop_image tool that accepts normalized bounding-box coordinates (0–1) and returns a base64-encoded PNG sub-region, enabling Claude to iteronomously zoom in on fine details through an agentic tool-use loop.

Claude's vision capabilities excel at analyzing entire images, but small text or fine chart details often require magnification. By creating a custom crop tool for detailed image region analysis in Claude vision, you enable the model to autonomously request close-up views of specific areas, transforming static image understanding into an interactive, multi-step reasoning process.

The Crop Tool Pattern

Claude processes images passed as image blocks in the conversation, but fine-grained analysis requires controlled zooming. The crop tool pattern exposes a callable function that Claude decides when to invoke, creating an agentic loop where the model iteratively examines regions until it extracts sufficient detail.

According to the reference implementation in multimodal/crop_tool.ipynb, this pattern requires three core components:

1. Tool schema – JSON definition describing the crop_image function and its parameters
2. Coordinate system – Normalized values (0.0 to 1.0) that scale to any image dimensions
3. Image transport – Base64-encoded PNG returned as a tool result content block

Defining the Tool Schema

The tool definition uses Anthropic's standard function-calling format. In multimodal/crop_tool.ipynb (lines 56–88), the schema specifies four normalized coordinates with strict boundary constraints:

{
  "name": "crop_image",
  "description": "Crop an image by specifying a bounding box.",
  "input_schema": {
    "type": "object",
    "properties": {
      "x1": {"type": "number", "minimum": 0, "maximum": 1, "description": "Left edge"},
      "y1": {"type": "number", "minimum": 0, "maximum": 1, "description": "Top edge"},
      "x2": {"type": "number", "minimum": 0, "maximum": 1, "description": "Right edge"},
      "y2": {"type": "number", "minimum": 0, "maximum": 1, "description": "Bottom edge"}
    },
    "required": ["x1", "y1", "x2", "y2"]
  }
}

This schema ensures Claude provides valid relative coordinates regardless of the image's absolute pixel dimensions.

Implementing the Crop Handler

The handle_crop function (lines 92–108 in multimodal/crop_tool.ipynb) executes the actual image manipulation and formats the result for Claude's vision model:

def handle_crop(image: PILImage.Image, x1: float, y1: float, x2: float, y2: float) -> list:
    """Execute the crop and return the result for Claude."""
    if not all(0 <= c <= 1 for c in [x1, y1, x2, y2]):
        return [{"type": "text", "text": "Error: Coordinates must be between 0 and 1"}]
    if x1 >= x2 or y1 >= y2:
        return [{"type": "text", "text": "Error: Invalid bounding box (need x1 < x2 and y1 < y2)"}]

    w, h = image.size
    cropped = image.crop((int(x1 * w), int(y1 * h), int(x2 * w), int(y2 * h)))

    return [
        {"type": "text",
         "text": f"Cropped to ({x1:.2f},{y1:.2f})-({x2:.2f},{y2:.2f}): {cropped.width}x{cropped.height}px"},
        {"type": "image",
         "source": {"type": "base64", "media_type": "image/png", "data": pil_to_base64(cropped)}}
    ]

The function validates inputs, converts normalized coordinates to absolute pixels, and returns both a text confirmation and the image block containing the base64-encoded crop.

Building the Agentic Loop

The agentic loop enables Claude to issue multiple crop requests until analysis is complete. The implementation (lines 94–119 in multimodal/crop_tool.ipynb) follows this pattern:

while True:
    response = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        tools=[CROP_TOOL],
        messages=messages,
    )
    for block in response.content:
        if block.type == "tool_use":
            result = handle_crop(image, **block.input)
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": block.id, "content": result}
            ]})
        else:
            print("[Assistant]", block.text)
    if response.stop_reason != "tool_use":
        break

This loop continues until Claude stops issuing tool_use requests, indicating it has gathered sufficient detail to answer the original question.

Using the Claude Agent SDK

For production implementations, the Claude Agent SDK automates much of the boilerplate. The same crop tool implemented with the @tool decorator (lines 73–110 in multimodal/crop_tool.ipynb) appears as follows:

@tool(
    "crop_image",
    "Crop an image by specifying a bounding box. Loads the image from a relative filepath.",
    {
        "type": "object",
        "properties": {
            "image_path": {"type": "string", "description": "Relative path to the image file"},
            "x1": {"type": "number", "minimum": 0, "maximum": 1},
            "y1": {"type": "number", "minimum": 0, "maximum": 1},
            "x2": {"type": "number", "minimum": 0, "maximum": 1},
            "y2": {"type": "number", "minimum": 0, "maximum": 1}
        },
        "required": ["image_path", "x1", "y1", "x2", "y2"]
    },
)
async def crop_image_tool(args: dict):
    path = Path(args["image_path"])
    img = PILImage.open(path)
    w, h = img.size
    cropped = img.crop((
        int(args["x1"] * w), int(args["y1"] * h),
        int(args["x2"] * w), int(args["y2"] * h)
    ))
    return {
        "content": [
            {"type": "text", "text": f"Cropped {args['image_path']}"},
            {"type": "image", "data": pil_to_base64(cropped), "mimeType": "image/png"}
        ]
    }

The SDK automatically registers this tool with an MCP server via create_sdk_mcp_server, handling the conversation state management without manual message plumbing.

Why Normalized Coordinates Matter

Claude never receives the original image dimensions in the prompt context, making absolute pixel coordinates unreliable. Using a 0–1 normalized coordinate system provides three critical advantages:

• Model agnostic – The same tool works across any image size without dimension-specific training
• Relative reasoning – Claude can specify "the top-right quadrant" or "center region" using intuitive fractions
• Validation simplicity – Bounds checking (0 ≤ x ≤ 1) prevents out-of-range errors before image processing

This approach aligns with how vision models internally represent spatial relationships, reducing coordinate hallucination errors.

Complete Implementation Example

Here is a minimal end-to-end script using the plain Anthropic API:

import base64
from io import BytesIO
from PIL import Image as PILImage
from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-4-6"

# Load image

image = PILImage.open("chart.png")

# Tool definition

CROP_TOOL = {
    "name": "crop_image",
    "description": "Crop an image by specifying a bounding box.",
    "input_schema": {
        "type": "object",
        "properties": {
            "x1": {"type": "number", "minimum": 0, "maximum": 1},
            "y1": {"type": "number", "minimum": 0, "maximum": 1},
            "x2": {"type": "number", "minimum": 0, "maximum": 1},
            "y2": {"type": "number", "minimum": 0, "maximum": 1},
        },
        "required": ["x1", "y1", "x2", "y2"],
    },
}

def pil_to_base64(img):
    buf = BytesIO()
    img.save(buf, format="PNG")
    return base64.b64encode(buf.getvalue()).decode()

def handle_crop(image, x1, y1, x2, y2):
    w, h = image.size
    cropped = image.crop((int(x1*w), int(y1*h), int(x2*w), int(y2*h)))
    return [
        {"type": "text", "text": f"Cropped ({x1:.2f},{y1:.2f})-({x2:.2f},{y2:.2f})"},
        {"type": "image", "source": {
            "type": "base64", "media_type": "image/png", 
            "data": pil_to_base64(cropped)
        }}
    ]

# Initialize conversation

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "Analyze this chart and compare the bar heights."},
        {"type": "image", "source": {
            "type": "base64", "media_type": "image/png", 
            "data": pil_to_base64(image)
        }},
        {"type": "text", "text": "Use the crop_image tool if you need to zoom in on specific bars."},
    ],
}]

# Agentic loop

while True:
    resp = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        tools=[CROP_TOOL],
        messages=messages,
    )
    for block in resp.content:
        if hasattr(block, "text"):
            print("[Assistant]", block.text)
        elif block.type == "tool_use":
            print("[Tool] crop_image", block.input)
            result = handle_crop(image, **block.input)
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": block.id, "content": result}
            ]})
    if resp.stop_reason != "tool_use":
        break

Key Reference Files

The anthropics/claude-cookbooks repository contains essential resources for extending this pattern:

• multimodal/crop_tool.ipynb – Complete reference implementation showing both plain API and Agent SDK approaches, including dataset loading and helper functions (lines 56–120)
• tool_use/utils/visualize.py – Utility functions for displaying base64 images in Jupyter notebooks when debugging crop regions
• CLAUDE.md – Repository quick-start guide covering dependency installation and API key configuration
• scripts/validate_notebooks.py – CI validation script to ensure your crop tool notebooks execute without errors
• tool_use/memory_tool.py – Advanced pattern showing how to combine cropping with persistent memory for multi-step visual reasoning

Summary

Creating a custom crop tool for detailed image region analysis in Claude vision unlocks agentic capabilities for fine-grained visual inspection:

• Normalized coordinates (0–1) allow Claude to specify regions without knowing absolute pixel dimensions
• Base64 transport returns cropped sub-images directly in tool result blocks
• Agentic loops enable iterative zooming until the model extracts sufficient detail
• Dual implementation paths support both direct API integration (lines 94–119 in multimodal/crop_tool.ipynb) and the higher-level Agent SDK (lines 73–110)

This pattern generalizes to any vision task requiring magnification, from reading small text in scanned documents to comparing adjacent elements in data visualizations.

Frequently Asked Questions

How do normalized coordinates work in the crop tool?

Normalized coordinates represent relative positions where 0,0 is the top-left corner and 1,1 is the bottom-right corner of the image. When Claude specifies x1: 0.25, y1: 0.25, x2: 0.75, y2: 0.75, the handler multiplies these values by the actual width and height (as implemented in lines 92–108 of multimodal/crop_tool.ipynb) to calculate the absolute pixel crop box. This ensures the tool works consistently across images of any resolution.

Can Claude crop the same image multiple times?

Yes. The agentic loop pattern (shown in lines 94–119 of multimodal/crop_tool.ipynb) allows Claude to issue sequential crop_image tool calls. After receiving each cropped region, Claude can analyze the result and request additional crops of different areas—or even nested crops of previously cropped regions—until it completes the analysis task.

What is the difference between the plain API and Agent SDK approaches?

The plain Anthropic API requires manual construction of the message history and tool result blocks, giving you explicit control over the conversation state (lines 94–119). The Claude Agent SDK (lines 73–110) abstracts this into a decorator-based system where @tool functions automatically integrate with MCP servers and the SDK manages the request/response cycle. Use the plain API for custom workflows or the SDK for rapid prototyping and standardized tool ecosystems.

Which image formats does the crop tool support?

The reference implementation uses PIL (Pillow) to load images, inheriting support for PNG, JPEG, TIFF, BMP, and other common formats. The tool specifically returns cropped regions as base64-encoded PNG (as seen in the handle_crop return value), ensuring lossless quality for detailed analysis regardless of the input format.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}