# How to Debug Vision Textualizer Failures When Generated Text Representations Are Invalid

> Fix Vision Textualizer invalid text generation. Debug prompt misconfigurations, missing code blocks, or image encoding issues by examining raw responses and extraction logic.

- Repository: [Junyi Ye/textflow](https://github.com/junyiye/textflow)
- Tags: how-to-guide
- Published: 2026-03-05

---

**The Vision Textualizer fails to generate valid text representations when prompts are misconfigured, model responses lack fenced code blocks, or image encoding exceeds model limits; debug by inspecting raw responses in [`src/textualizer.py`](https://github.com/junyiye/textflow/blob/main/src/textualizer.py) and validating extraction logic in [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py).**

When working with the `junyiye/textflow` repository, the Vision Textualizer converts flowchart images into structured text formats like Mermaid, Graphviz, or PlantUML. If you encounter invalid or empty text representations, the issue typically stems from one of four pipeline stages. This guide walks you through diagnosing and fixing these failures using the actual source code implementation.

## Understanding the Vision Textualizer Pipeline

The textualization process follows a strict four-stage pipeline defined in [`src/textualizer.py`](https://github.com/junyiye/textflow/blob/main/src/textualizer.py). Failures can occur at any transition point:

1. **Prompt Construction**: `load_textualizer_prompt(output_type)` builds the instruction sent to the visual language model (VLM)
2. **Model Invocation**: `ModelWrapper(textualizer).generate_response(prompt, image_path=image_path)` transmits the image and prompt
3. **Representation Extraction**: `extract_representation(response)` parses fenced code blocks from the raw model output
4. **Output Serialization**: Results are written to JSON files in `output/{dataset}/{output_type}/{model}.json`

## Stage-by-Stage Debugging Guide

### Prompt Construction Issues

Problems here typically involve typos in the `output_type` parameter or missing prompt files.

**Common failure modes:**
- Wrong `output_type` value (e.g., `mermiad` instead of `mermaid`)
- `load_textualizer_prompt` raising exceptions due to missing prompt templates

**Diagnostic steps:**

Verify the exact prompt string being sent to the model:

```python
from src.prompts import load_textualizer_prompt

output_type = "mermaid"
prompt = load_textualizer_prompt(output_type)
print("Prompt sent to model:\n", prompt)

```

Check that `output_type` matches one of the supported values defined in [`src/prompts/prompts.py`](https://github.com/junyiye/textflow/blob/main/src/prompts/prompts.py) (lines 5-74).

### Model Invocation Failures

This stage involves the `ModelWrapper` class communicating with the VLM API. Failures here often relate to image encoding or API limits.

**Common failure modes:**
- Image dimensions exceed model limits (e.g., Claude 3.5 Sonnet requires ≤ 8000px)
- API authentication or rate-limit errors
- Model returns conversational text instead of code

**Diagnostic steps:**

Enable debug logging to capture the raw response before extraction:

```python
import logging
from src.models import ModelWrapper

# Force debug level

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

model = ModelWrapper("Qwen2-VL-7B")
response = model.generate_response(
    prompt, 
    image_path="data/flowlearn/images/9361.jpeg"
)
print("Raw response:\n", response)

```

For image encoding issues, inspect [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py) where `encode_image_anthropic` automatically rescales images > 8000px. Verify your image format is supported by the specific model wrapper being used.

### Representation Extraction Errors

The `extract_representation` function (lines 86-94 in [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py)) uses regex to parse fenced code blocks. This is the most common failure point.

**Common failure modes:**
- Model omits the fence or uses incorrect language tags (e.g., ` ``` ` instead of ` ```mermaid `)
- Extra whitespace breaks the extraction regex
- Multiple code blocks present and the wrong one is selected

**Diagnostic steps:**

Test the extraction logic manually on problematic responses:

```python
from src.utils import extract_representation

# Test with missing language tag

bad_response = "Sure, here's the diagram:\n```\nflowchart TD\nA-->B;\n```"
print("Extracted:", extract_representation(bad_response))

# Test with correct format

good_response = "```mermaid\nflowchart TD\nA-->B;\n```"
print("Extracted:", extract_representation(good_response))

```

If the model consistently omits fences, modify the prompt in [`src/prompts/prompts.py`](https://github.com/junyiye/textflow/blob/main/src/prompts/prompts.py) to explicitly demand them:

```python
def load_textualizer_prompt(output_type):
    base = "Generate the {0} code for the provided flowchart. **Wrap the code in a fenced block** using ```{0}``` as the language tag."
    # ... existing return strings ...

```

### Output Serialization Validation

The final stage writes results to JSON. Empty or malformed strings here indicate upstream failures.

**Diagnostic steps:**

Verify the output file contains valid data:

```bash

# Check the generated JSON file

cat output/flowlearn/mermaid/Qwen2-VL-7B.json | head -20

```

Validate that each entry is a non-empty string before downstream processing.

## Practical Debugging Workflow

Follow this systematic approach to isolate Vision Textualizer failures:

1. **Run with verbose logging** using a single test image:

   ```bash
   python -m src.textualizer \
       --dataset flowlearn \
       --textualizer Qwen2-VL-7B \
       --output_type mermaid \
       --verbose
   ```

2. **Inspect the log file** located under `config["logging"]["log_dir"]/flowlearn/…`. Look for these specific log patterns:

   ```

   INFO: Starting the Vision Textualizer program...
   INFO: Prompt: Generate the Mermaid code for the provided flowchart....
   INFO: Raw model response: <the full text returned by the VLM>
   INFO: Extracted representation: flowchart TD ...
   ```

3. **Validate extraction manually** in a Python REPL using `extract_representation` on the logged raw response.

4. **Check image encoding** if the model returns errors about image format. Verify `utils.encode_image` selects the correct encoder for your model (e.g., `encode_image_anthropic` rescales images > 8000px).

5. **Confirm output integrity** by opening [`output/flowlearn/mermaid/Qwen2-VL-7B.json`](https://github.com/junyiye/textflow/blob/main/output/flowlearn/mermaid/Qwen2-VL-7B.json) and verifying non-empty string values.

## Quick Fix Checklist

Use this checklist to rapidly resolve common Vision Textualizer failures:

- [ ] Verify `--output_type` is one of `mermaid`, `graphviz`, or `plantuml` (check [`src/prompts/prompts.py`](https://github.com/junyiye/textflow/blob/main/src/prompts/prompts.py) lines 5-74)
- [ ] Add `logger.setLevel(logging.DEBUG)` after logger creation in [`src/textualizer.py`](https://github.com/junyiye/textflow/blob/main/src/textualizer.py) to capture raw model responses
- [ ] Ensure prompt strings in `load_textualizer_prompt` contain explicit fence instructions (e.g., "Wrap the code in a fenced block using ```mermaid```")
- [ ] Inspect `extract_representation` logic in [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py) (lines 86-94) if code blocks aren't parsing correctly
- [ ] Confirm image dimensions are within model limits (`encode_image_anthropic` rescales > 8000px automatically)

## Key Files and Functions

Understanding these source files is essential for debugging Vision Textualizer failures:

| File | Role | Relevant Lines |
|------|------|----------------|
| [`src/textualizer.py`](https://github.com/junyiye/textflow/blob/main/src/textualizer.py) | Orchestrates dataset loading, model calls, and result serialization. | `main()` definition (lines 16-86) – sets up logger, builds prompts, calls `ModelWrapper`, writes JSON. |
| [`src/prompts/prompts.py`](https://github.com/junyiye/textflow/blob/main/src/prompts/prompts.py) | Supplies the textualizer prompt for each output format. | `load_textualizer_prompt` (lines 5-74). |
| [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py) | Encodes images and extracts the code block from model replies. | `extract_representation` (lines 86-94) and format-specific extractors (lines 47-84). |
| [`src/models/__init__.py`](https://github.com/junyiye/textflow/blob/main/src/models/__init__.py) | Wraps the VLM API (`generate_response`). | Look for `ModelWrapper` implementation – it decides how images are sent to the model. |
| [`src/config.py`](https://github.com/junyiye/textflow/blob/main/src/config.py) | Holds paths and logging configuration referenced throughout. | `config["logging"]["log_dir"]` and `config["file_paths"]` (used in [`textualizer.py`](https://github.com/junyiye/textflow/blob/main/textualizer.py)). |

## Summary

Debugging Vision Textualizer failures requires systematic inspection of four pipeline stages:

- **Prompt Construction**: Verify `output_type` values and inspect prompts in [`src/prompts/prompts.py`](https://github.com/junyiye/textflow/blob/main/src/prompts/prompts.py)
- **Model Invocation**: Enable debug logging to capture raw VLM responses before extraction
- **Representation Extraction**: Test `extract_representation` in [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py) (lines 86-94) against problematic model outputs
- **Output Validation**: Confirm JSON files contain non-empty strings and image encoding respects model dimension limits

By tracing failures through [`src/textualizer.py`](https://github.com/junyiye/textflow/blob/main/src/textualizer.py), [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py), and the model wrapper, you can identify whether issues stem from prompt engineering, API constraints, or regex extraction logic.

## Frequently Asked Questions

### Why does the Vision Textualizer return empty strings instead of Mermaid code?

Empty strings typically indicate that `extract_representation` in [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py) failed to locate a fenced code block in the model's response. This happens when the VLM omits the triple backticks or uses incorrect language tags. Enable debug logging to inspect the raw response, then manually test the extraction regex against that output.

### How do I fix "image too large" errors when using Claude models?

The `encode_image_anthropic` function in [`src/utils.py`](https://github.com/junyiye/textflow/blob/main/src/utils.py) automatically rescales images exceeding 8000 pixels, but you should verify your input images meet model-specific constraints. For Claude 3.5 Sonnet, ensure dimensions stay below 8000px on any side. If you encounter persistent encoding errors, check that `ModelWrapper` selects the correct encoder for your specific model name.

### What should I check if the model generates code but the JSON output is malformed?

Malformed JSON outputs usually stem from serialization issues in [`src/textualizer.py`](https://github.com/junyiye/textflow/blob/main/src/textualizer.py) (lines 16-86) where `json.dump` writes the results. Verify that `extract_representation` returns valid strings rather than None or exception objects. Check the output file path—typically `output/{dataset}/{output_type}/{model}.json`—and ensure the directory exists and has write permissions before the textualizer runs.