# Debugging Voicebox Generation Failures and Recovery Steps

> Troubleshoot Voicebox generation failures with error recording and recovery steps. Retry synthesis or regenerate with new seeds via REST endpoints.

- Repository: [Jamie Pine/voicebox](https://github.com/jamiepine/voicebox)
- Tags: how-to-guide
- Published: 2026-04-14

---

**Voicebox catches generation errors in the `run_generation` service, records the failure with the exception message in the database, and exposes REST endpoints to either retry the exact synthesis or regenerate with a new random seed.**

Voicebox is an open-source text-to-speech platform developed by Jamie Pine that orchestrates multi-stage speech synthesis pipelines. When **debugging Voicebox generation failures and recovery steps**, understanding the internal error handling in [`backend/services/generation.py`](https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py) and the state management endpoints is essential for maintaining reliable production workflows.

## Understanding the Voicebox Generation Pipeline

A speech synthesis request flows through four distinct stages orchestrated by the FastAPI backend:

1. **API Request** – The client calls `POST /generate`, which creates a database record with status **"generating"**.
2. **Task Queue** – The request is handed to the background task manager for asynchronous processing.
3. **`run_generation`** – The core routine loads the TTS model, creates voice prompts, streams text through the selected backend, applies optional audio normalization and trimming, and persists the result.
4. **Status Updates** – Throughout execution, the database record is updated via `history.update_generation_status` to reflect progress or completion.

If any step raises an exception, the routine catches it, logs the error, and transitions the status to **"failed"**.

## Where Failures Are Recorded

Error handling occurs in the orchestration layer at [`backend/services/generation.py`](https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py). The `run_generation` function wraps its logic in a broad exception handler that updates the `generations` table and logs the stack trace:

```python

# backend/services/generation.py – error handling

except Exception as e:
    await history.update_generation_status(
        generation_id,
        "failed",
        bg_db,
        error=str(e),
    )
    logger.exception("Generation failed")

```

The status is stored in the `status` column of the `generations` table, while the error message is written to the `error` column. You can inspect these values via the `GET /generate/{generation_id}/status` endpoint.

## Recovery Endpoints and Strategies

Voicebox provides two distinct recovery mechanisms depending on whether you want identical output or a variation.

### Retrying Failed Generations (Same Seed)

Use the retry endpoint to re-run the exact same parameters with the original seed. This is useful for transient failures such as temporary GPU memory pressure or network timeouts.

**`POST /generate/{generation_id}/retry`**

The implementation in [`backend/routes/generations.py`](https://github.com/jamiepine/voicebox/blob/main/backend/routes/generations.py) validates that the generation status is **"failed"**, resets it to **"generating"**, and re-enqueues the task with `mode="retry"` and the preserved `seed` value:

```python

# backend/routes/generations.py – retry endpoint

@router.post("/generate/{generation_id}/retry", response_model=models.GenerationResponse)
async def retry_generation(generation_id: str, db: Session = Depends(get_db)):
    gen = db.query(DBGeneration).filter_by(id=generation_id).first()
    if not gen:
        raise HTTPException(status_code=404, detail="Generation not found")
    if (gen.status or "completed") != "failed":
        raise HTTPException(status_code=400, detail="Only failed generations can be retried")
    # Reset status and enqueue a new run

    gen.status = "generating"
    db.commit()
    await history.update_generation_status(generation_id, "generating", db)
    enqueue_generation(
        run_generation(
            generation_id=generation_id,
            profile_id=gen.profile_id,
            text=gen.text,
            language=gen.language,
            engine=gen.engine,
            model_size=gen.model_size,
            seed=gen.seed,        # keep original seed

            normalize=gen.normalize,
            instruct=gen.instruct,
            mode="retry",
            max_chunk_chars=gen.max_chunk_chars,
            crossfade_ms=gen.crossfade_ms,
        )
    )
    return models.GenerationResponse.from_orm(gen)

```

### Regenerating Variations (New Seed)

Use the regenerate endpoint to create a new synthesis based on a completed generation but with a random seed, producing a different vocal inflection.

**`POST /generate/{generation_id}/regenerate`**

This endpoint in [`backend/routes/generations.py`](https://github.com/jamiepine/voicebox/blob/main/backend/routes/generations.py) requires the original generation to be **"completed"**. It creates a new database row with `seed=None` and enqueues the task with `mode="regenerate"`:

```python

# backend/routes/generations.py – regenerate endpoint

@router.post("/generate/{generation_id}/regenerate", response_model=models.GenerationResponse)
async def regenerate_generation(generation_id: str, db: Session = Depends(get_db)):
    gen = db.query(DBGeneration).filter_by(id=generation_id).first()
    if not gen:
        raise HTTPException(status_code=404, detail="Generation not found")
    if (gen.status or "completed") != "completed":
        raise HTTPException(status_code=400, detail="Generation must be completed to regenerate")
    # Create a new DB entry for the variation

    new_id = str(uuid.uuid4())
    new_gen = DBGeneration(
        id=new_id,
        profile_id=gen.profile_id,
        text=gen.text,
        language=gen.language,
        status="generating",
        engine=gen.engine,
        model_size=gen.model_size,
        normalize=gen.normalize,
        instruct=gen.instruct,
        seed=None,               # no seed → random variation

        # … other fields omitted for brevity …

    )
    db.add(new_gen)
    db.commit()
    # Enqueue generation with mode="regenerate"

    enqueue_generation(
        run_generation(
            generation_id=new_id,
            profile_id=gen.profile_id,
            text=gen.text,
            language=gen.language,
            engine=gen.engine,
            model_size=gen.model_size,
            seed=None,
            normalize=gen.normalize,
            instruct=gen.instruct,
            mode="regenerate",
            max_chunk_chars=gen.max_chunk_chars,
            crossfade_ms=gen.crossfade_ms,
        )
    )
    return models.GenerationResponse.from_orm(new_gen)

```

### Handling Stale "Generating" Records

If the server crashes during synthesis, records may remain stuck in the **"generating"** state. The startup routine in [`backend/app.py`](https://github.com/jamiepine/voicebox/blob/main/backend/app.py) automatically marks these as failed with a SQL update:

```sql
UPDATE generations SET status = 'failed' WHERE status = 'generating' AND updated_at < now() - interval '5 minutes';

```

Once marked failed, you can manually trigger a retry via the endpoints above.

## Step-by-Step Debugging Workflow

Follow this sequence to diagnose and recover from generation failures:

1. **Check current status** using the status endpoint:
   
   ```bash
   curl -s "http://localhost:8000/generate/<generation_id>/status"
   ```

2. **Inspect the error field** in the JSON response. Common causes include:
   - Model loading failures (engine initialization errors)
   - Missing reference audio for voice prompt creation
   - Runtime exceptions within the TTS backend library

3. **Retry the exact generation** if the status is failed:
   
   ```bash
   curl -X POST "http://localhost:8000/generate/<generation_id>/retry"
   ```

4. **Create a variation** if you want different output instead of identical retry:
   
   ```bash
   curl -X POST "http://localhost:8000/generate/<generation_id>/regenerate"
   ```

5. **Review server logs** for stack traces emitted by `logger.exception` in `run_generation`. Access these via Docker logs (`docker logs <container>`) or systemd journal.

6. **Verify model availability** at `GET /models/status`. The server returns HTTP 400 if a request targets a model that has not been downloaded.

## Practical Code Examples

### Python Client with Polling and Auto-Retry

```python
import requests
import time

BASE = "http://localhost:8000"

# Submit a generation request

payload = {
    "profile_id": "123e4567-e89b-12d3-a456-426614174000",
    "text": "Hello, world!",
    "language": "en",
    "engine": "qwen",
}
resp = requests.post(f"{BASE}/generate", json=payload)
gen_id = resp.json()["generation_id"]
print("Created:", gen_id)

# Poll for completion with automatic retry logic

while True:
    status = requests.get(f"{BASE}/generate/{gen_id}/status").json()
    
    if status["status"] == "completed":
        print("✅ Completed")
        break
        
    if status["status"] == "failed":
        print("❌ Failed –", status.get("error"))
        # Attempt a retry

        retry_resp = requests.post(f"{BASE}/generate/{gen_id}/retry")
        print("Retried, status code:", retry_resp.status_code)
        
    time.sleep(2)

```

### Curl Commands for Manual Recovery

```bash

# Submit a new generation

curl -X POST -H "Content-Type: application/json" \
  -d '{"profile_id":"<uuid>","text":"Debugging test","language":"en","engine":"qwen"}' \
  http://localhost:8000/generate

# Check status

curl http://localhost:8000/generate/<generation_id>/status

# Retry failed generation

curl -X POST http://localhost:8000/generate/<generation_id>/retry

# Regenerate with new seed (variation)

curl -X POST http://localhost:8000/generate/<generation_id>/regenerate

```

## Key Source Files for Debugging

| File | Role | Direct Link |
|------|------|-------------|
| [`backend/routes/generations.py`](https://github.com/jamiepine/voicebox/blob/main/backend/routes/generations.py) | FastAPI endpoints for generate, retry, regenerate, and status | https://github.com/jamiepine/voicebox/blob/main/backend/routes/generations.py |
| [`backend/services/generation.py`](https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py) | Core orchestration (`run_generation`) and exception handling | https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py |
| [`backend/services/history.py`](https://github.com/jamiepine/voicebox/blob/main/backend/services/history.py) | Helper functions to persist status updates and error messages | https://github.com/jamiepine/voicebox/blob/main/backend/services/history.py |
| [`backend/app.py`](https://github.com/jamiepine/voicebox/blob/main/backend/app.py) | Application startup logic including stale-generation cleanup | https://github.com/jamiepine/voicebox/blob/main/backend/app.py |
| [`backend/routes/models.py`](https://github.com/jamiepine/voicebox/blob/main/backend/routes/models.py) | Model download and availability status endpoints | https://github.com/jamiepine/voicebox/blob/main/backend/routes/models.py |

## Summary

- **Voicebox generation failures** are captured in [`backend/services/generation.py`](https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py), logged to the `generations` table with status **"failed"**, and expose the error message via the status endpoint.
- **Retry** (`/generate/{id}/retry`) re-queues the exact same parameters and seed, useful for transient hardware or network issues.
- **Regenerate** (`/generate/{id}/regenerate`) creates a new database entry with `seed=None` to produce a variation, requiring the original to be **"completed"**.
- **Stale records** from server crashes are automatically cleaned up on startup in [`backend/app.py`](https://github.com/jamiepine/voicebox/blob/main/backend/app.py) by marking records older than five minutes as failed.
- **Model availability** should be verified at `/models/status` when generations fail immediately upon submission.

## Frequently Asked Questions

### How do I check the status of a Voicebox generation?

Send a `GET` request to `/generate/{generation_id}/status`. The endpoint returns a JSON object containing the current `status` (**generating**, **completed**, or **failed**) and an `error` field populated only when the status is failed. This endpoint is defined in [`backend/routes/generations.py`](https://github.com/jamiepine/voicebox/blob/main/backend/routes/generations.py) and queries the `generations` table directly.

### What is the difference between retry and regenerate in Voicebox?

**Retry** (`POST /generate/{id}/retry`) is for failed generations only and re-runs the synthesis with the exact same `seed` value to produce identical output, making it ideal for recovering from transient errors. **Regenerate** (`POST /generate/{id}/regenerate`) requires a completed generation and creates a new database record with `seed=None`, triggering a fresh synthesis that yields a different vocal variation while using the same text and voice profile.

### Why does my Voicebox generation get stuck in "generating" status?

If the server process crashes or is killed during synthesis, the database record remains in **"generating"** status. The startup routine in [`backend/app.py`](https://github.com/jamiepine/voicebox/blob/main/backend/app.py) automatically runs a SQL update to mark any **"generating"** rows older than five minutes as **"failed"**. Once marked failed, you can manually trigger a recovery using the retry endpoint.

### Where are Voicebox generation errors logged?

Errors are logged in two places: the `error` column of the `generations` table (accessible via the status endpoint) and the server logs via `logger.exception` calls in [`backend/services/generation.py`](https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py). Check your process manager output (e.g., `docker logs` or systemd journal) for full Python stack traces when troubleshooting persistent failures.