# How the Pipeline Middleware Pattern Handles Code Generation Requests in Screenshot-to-Code

> Discover how the pipeline middleware pattern efficiently handles code generation requests in screenshot to code through a chained series of handlers, ensuring robust error management and separation of concerns.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**The pipeline middleware pattern in screenshot-to-code processes code generation requests through a chained series of middleware objects, each receiving a mutable `PipelineContext` to perform work before invoking the next handler, enabling clean separation of concerns and robust error handling over WebSocket connections.**

The `abi/screenshot-to-code` repository implements a sophisticated backend for converting visual inputs into functional code. Rather than processing requests through a monolithic handler, the system uses a **pipeline middleware pattern** to manage the WebSocket endpoint at `/generate-code`. Each middleware receives a shared context object, performs its specialized transformation, and explicitly delegates control to the next stage in the chain.

## Pipeline Architecture and Execution Chain

The core orchestration logic resides in [`backend/routes/generate_code.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/routes/generate_code.py), where the `Pipeline` class (lines 13–25) maintains an ordered list of middleware objects. When you instantiate the pipeline with `pipeline = Pipeline()` and add middleware via `pipeline.use(...)`, the system defers execution until `await pipeline.execute()` is called.

The `execute` method (lines 26–38) builds the runtime chain by wrapping each middleware with `_wrap_middleware`. The implementation reverses the middleware list during chain construction so that the first middleware added runs first, creating a nested closure structure where each layer invokes the next. This design ensures that cleanup operations—such as the `finally` block in `WebSocketSetupMiddleware`—execute in the correct order after the pipeline completes or fails.

## The Six-Stage Request Processing Flow

The code generation pipeline chains six specialized middleware components to transform a screenshot and prompt into executable code.

### 1. WebSocket Setup and Lifecycle Management

The `WebSocketSetupMiddleware` (lines 149–166) accepts the incoming socket connection and instantiates a `WebSocketCommunicator`. A critical `try...finally` block guarantees that `websocket.close()` is called after all downstream middleware finish, ensuring graceful resource cleanup even if intermediate stages raise exceptions.

### 2. Parameter Extraction and Validation

`ParameterExtractionMiddleware` (lines 166–188) receives the JSON payload from the client and validates it through `ParameterExtractionStage`. The extracted parameters—including the input image, text prompt, and history—are stored in `context.extracted_params`, making them available to all subsequent middleware without re-parsing the raw payload.

### 3. Status Broadcasting

Before expensive LLM operations begin, the `StatusBroadcastMiddleware` (lines 190–207) calculates the number of generation variants (parallel model runs) and sends an initial `"variantCount"` message to the client. It also broadcasts a `"status"` message for each variant, establishing the communication protocol for streaming results.

### 4. Prompt Construction

The `PromptCreationMiddleware` (lines 209–226) calls `PromptCreationStage.build_prompt_messages`, which internally delegates to `prompts.pipeline.build_prompt_messages`. This stage assembles the final list of `ChatCompletionMessageParam` objects—including system prompts, user messages, and image data—required by the LLM APIs.

### 5. Model Selection and Concurrent Generation

`CodeGenerationMiddleware` (lines 228–295) represents the core processing stage. It first selects appropriate LLMs via `ModelSelectionStage`, then instantiates `AgenticGenerationStage` (implemented in [`backend/agent/runner.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/runner.py)). The stage executes variants concurrently using `asyncio.gather`, streaming intermediate `"variantComplete"` and `"variantError"` messages back to the client as results arrive from OpenAI, Anthropic, or Gemini APIs.

### 6. Post-Processing and Audit Logging

After all variants complete, the `PostProcessingMiddleware` (lines 297–311) writes the first successful HTML output to a log file via [`backend/fs_logging/core.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/fs_logging/core.py). This stage performs final cleanup operations and prepares the context for graceful socket closure by the `WebSocketSetupMiddleware` finally block.

## Error Handling and Flow Abort

Every middleware receives `context.throw_error`, a callable that sends a structured `"error"` message to the client and closes the WebSocket. If any stage encounters a validation or runtime error, it awaits `context.throw_error(message)` instead of calling `next_func()`. When an error is thrown, the pipeline **skips all remaining middleware**, ensuring the client receives a single coherent error response rather than partial outputs or cascading failures.

## Extending the Pipeline with Custom Middleware

The architecture supports extensibility through subclassing. To inject custom logic—such as analytics logging—you implement the `Middleware` interface and register your component with `pipeline.use()`.

```python

# Defining a custom analytics middleware

from backend.routes.generate_code import Middleware, PipelineContext
from typing import Callable, Awaitable
import time

class AnalyticsMiddleware(Middleware):
    async def process(self, context: PipelineContext, next_func: Callable[[], Awaitable[None]]) -> None:
        context.metadata["start"] = time.time()
        await next_func()  # Execute downstream middleware

        duration = time.time() - context.metadata["start"]
        print(f"Code-gen request took {duration:.2f}s")

```

Insert your middleware into the pipeline in the route handler:

```python

# Adding to the execution chain

pipeline.use(AnalyticsMiddleware())

```

The complete request flow assembles like this:

```python
async def stream_code(websocket: WebSocket):
    pipeline = Pipeline()
    pipeline.use(WebSocketSetupMiddleware())        # Accept socket

    pipeline.use(ParameterExtractionMiddleware())   # Parse payload

    pipeline.use(StatusBroadcastMiddleware())       # Initialize client

    pipeline.use(PromptCreationMiddleware())        # Build LLM messages

    pipeline.use(CodeGenerationMiddleware())        # Run LLMs

    pipeline.use(PostProcessingMiddleware())        # Log results

    await pipeline.execute(websocket)

```

## Summary

- **Pipeline orchestration**: The `Pipeline` class in [`backend/routes/generate_code.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/routes/generate_code.py) manages ordered middleware execution via a reversed chain wrapper.
- **Shared state**: All middleware access a mutable `PipelineContext` object, enabling data flow without global variables.
- **Concurrent generation**: The `CodeGenerationMiddleware` uses `asyncio.gather` to run multiple LLM variants simultaneously.
- **Graceful degradation**: The `throw_error` mechanism ensures that validation or runtime errors abort the pipeline cleanly, sending a single error response to the client.
- **Extensibility**: New processing stages implement the `process` method and register via `pipeline.use()`, automatically gaining access to context and error handling.

## Frequently Asked Questions

### What is the pipeline middleware pattern used for in screenshot-to-code?

The pattern decouples the code generation workflow into discrete, composable stages. Each middleware handles a specific concern—such as WebSocket lifecycle management, parameter validation, or LLM interaction—while the `Pipeline` class manages execution order and error propagation. This design prevents the monolithic handler complexity common in WebSocket applications.

### How does the pipeline handle failures during code generation?

If any middleware encounters an error, it calls `await context.throw_error(message)` instead of invoking `next_func()`. This aborts the pipeline immediately, skipping all remaining middleware and triggering the `finally` blocks of upstream middleware (particularly `WebSocketSetupMiddleware`) to close resources. The client receives a single JSON error message with the failure details.

### Can I modify the pipeline to add preprocessing or caching?

Yes. You can subclass `Middleware`, implement the `async def process(self, context, next_func)` method, and insert it into the chain using `pipeline.use()`. Because the `PipelineContext` is shared, you can attach data like `context.cache_key = ...` in early middleware and reference it in later stages such as `CodeGenerationMiddleware`.

### Where are the core pipeline and LLM integration files located?

The main pipeline implementation resides in [`backend/routes/generate_code.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/routes/generate_code.py) (lines 13–311). Supporting files include [`backend/prompts/pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py) for message construction, [`backend/agent/runner.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/runner.py) for LLM API interaction, `backend/prompts/create/*` for prompt templates, and [`backend/fs_logging/core.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/fs_logging/core.py) for output persistence.