Debugging Voicebox Generation Failures and Recovery Steps

Voicebox catches generation errors in the run_generation service, records the failure with the exception message in the database, and exposes REST endpoints to either retry the exact synthesis or regenerate with a new random seed.

Voicebox is an open-source text-to-speech platform developed by Jamie Pine that orchestrates multi-stage speech synthesis pipelines. When debugging Voicebox generation failures and recovery steps, understanding the internal error handling in backend/services/generation.py and the state management endpoints is essential for maintaining reliable production workflows.

Understanding the Voicebox Generation Pipeline

A speech synthesis request flows through four distinct stages orchestrated by the FastAPI backend:

1. API Request – The client calls POST /generate, which creates a database record with status "generating".
2. Task Queue – The request is handed to the background task manager for asynchronous processing.
3. run_generation – The core routine loads the TTS model, creates voice prompts, streams text through the selected backend, applies optional audio normalization and trimming, and persists the result.
4. Status Updates – Throughout execution, the database record is updated via history.update_generation_status to reflect progress or completion.

If any step raises an exception, the routine catches it, logs the error, and transitions the status to "failed".

Where Failures Are Recorded

Error handling occurs in the orchestration layer at backend/services/generation.py. The run_generation function wraps its logic in a broad exception handler that updates the generations table and logs the stack trace:

# backend/services/generation.py – error handling

except Exception as e:
    await history.update_generation_status(
        generation_id,
        "failed",
        bg_db,
        error=str(e),
    )
    logger.exception("Generation failed")

The status is stored in the status column of the generations table, while the error message is written to the error column. You can inspect these values via the GET /generate/{generation_id}/status endpoint.

Recovery Endpoints and Strategies

Voicebox provides two distinct recovery mechanisms depending on whether you want identical output or a variation.

Retrying Failed Generations (Same Seed)

Use the retry endpoint to re-run the exact same parameters with the original seed. This is useful for transient failures such as temporary GPU memory pressure or network timeouts.

POST /generate/{generation_id}/retry

The implementation in backend/routes/generations.py validates that the generation status is "failed", resets it to "generating", and re-enqueues the task with mode="retry" and the preserved seed value:

# backend/routes/generations.py – retry endpoint

@router.post("/generate/{generation_id}/retry", response_model=models.GenerationResponse)
async def retry_generation(generation_id: str, db: Session = Depends(get_db)):
    gen = db.query(DBGeneration).filter_by(id=generation_id).first()
    if not gen:
        raise HTTPException(status_code=404, detail="Generation not found")
    if (gen.status or "completed") != "failed":
        raise HTTPException(status_code=400, detail="Only failed generations can be retried")
    # Reset status and enqueue a new run

    gen.status = "generating"
    db.commit()
    await history.update_generation_status(generation_id, "generating", db)
    enqueue_generation(
        run_generation(
            generation_id=generation_id,
            profile_id=gen.profile_id,
            text=gen.text,
            language=gen.language,
            engine=gen.engine,
            model_size=gen.model_size,
            seed=gen.seed,        # keep original seed

            normalize=gen.normalize,
            instruct=gen.instruct,
            mode="retry",
            max_chunk_chars=gen.max_chunk_chars,
            crossfade_ms=gen.crossfade_ms,
        )
    )
    return models.GenerationResponse.from_orm(gen)

Regenerating Variations (New Seed)

Use the regenerate endpoint to create a new synthesis based on a completed generation but with a random seed, producing a different vocal inflection.

POST /generate/{generation_id}/regenerate

This endpoint in backend/routes/generations.py requires the original generation to be "completed". It creates a new database row with seed=None and enqueues the task with mode="regenerate":

# backend/routes/generations.py – regenerate endpoint

@router.post("/generate/{generation_id}/regenerate", response_model=models.GenerationResponse)
async def regenerate_generation(generation_id: str, db: Session = Depends(get_db)):
    gen = db.query(DBGeneration).filter_by(id=generation_id).first()
    if not gen:
        raise HTTPException(status_code=404, detail="Generation not found")
    if (gen.status or "completed") != "completed":
        raise HTTPException(status_code=400, detail="Generation must be completed to regenerate")
    # Create a new DB entry for the variation

    new_id = str(uuid.uuid4())
    new_gen = DBGeneration(
        id=new_id,
        profile_id=gen.profile_id,
        text=gen.text,
        language=gen.language,
        status="generating",
        engine=gen.engine,
        model_size=gen.model_size,
        normalize=gen.normalize,
        instruct=gen.instruct,
        seed=None,               # no seed → random variation

        # … other fields omitted for brevity …

    )
    db.add(new_gen)
    db.commit()
    # Enqueue generation with mode="regenerate"

    enqueue_generation(
        run_generation(
            generation_id=new_id,
            profile_id=gen.profile_id,
            text=gen.text,
            language=gen.language,
            engine=gen.engine,
            model_size=gen.model_size,
            seed=None,
            normalize=gen.normalize,
            instruct=gen.instruct,
            mode="regenerate",
            max_chunk_chars=gen.max_chunk_chars,
            crossfade_ms=gen.crossfade_ms,
        )
    )
    return models.GenerationResponse.from_orm(new_gen)

Handling Stale "Generating" Records

If the server crashes during synthesis, records may remain stuck in the "generating" state. The startup routine in backend/app.py automatically marks these as failed with a SQL update:

UPDATE generations SET status = 'failed' WHERE status = 'generating' AND updated_at < now() - interval '5 minutes';

Once marked failed, you can manually trigger a retry via the endpoints above.

Step-by-Step Debugging Workflow

Follow this sequence to diagnose and recover from generation failures:

1. Check current status using the status endpoint:

   curl -s "http://localhost:8000/generate/<generation_id>/status"

2. Inspect the error field in the JSON response. Common causes include:

   • Model loading failures (engine initialization errors)
   • Missing reference audio for voice prompt creation
   • Runtime exceptions within the TTS backend library

3. Retry the exact generation if the status is failed:

   curl -X POST "http://localhost:8000/generate/<generation_id>/retry"

4. Create a variation if you want different output instead of identical retry:

   curl -X POST "http://localhost:8000/generate/<generation_id>/regenerate"

5. Review server logs for stack traces emitted by logger.exception in run_generation. Access these via Docker logs (docker logs <container>) or systemd journal.

6. Verify model availability at GET /models/status. The server returns HTTP 400 if a request targets a model that has not been downloaded.

Practical Code Examples

Python Client with Polling and Auto-Retry

import requests
import time

BASE = "http://localhost:8000"

# Submit a generation request

payload = {
    "profile_id": "123e4567-e89b-12d3-a456-426614174000",
    "text": "Hello, world!",
    "language": "en",
    "engine": "qwen",
}
resp = requests.post(f"{BASE}/generate", json=payload)
gen_id = resp.json()["generation_id"]
print("Created:", gen_id)

# Poll for completion with automatic retry logic

while True:
    status = requests.get(f"{BASE}/generate/{gen_id}/status").json()
    
    if status["status"] == "completed":
        print("✅ Completed")
        break
        
    if status["status"] == "failed":
        print("❌ Failed –", status.get("error"))
        # Attempt a retry

        retry_resp = requests.post(f"{BASE}/generate/{gen_id}/retry")
        print("Retried, status code:", retry_resp.status_code)
        
    time.sleep(2)

Curl Commands for Manual Recovery

# Submit a new generation

curl -X POST -H "Content-Type: application/json" \
  -d '{"profile_id":"<uuid>","text":"Debugging test","language":"en","engine":"qwen"}' \
  http://localhost:8000/generate

# Check status

curl http://localhost:8000/generate/<generation_id>/status

# Retry failed generation

curl -X POST http://localhost:8000/generate/<generation_id>/retry

# Regenerate with new seed (variation)

curl -X POST http://localhost:8000/generate/<generation_id>/regenerate

Key Source Files for Debugging

File	Role	Direct Link
backend/routes/generations.py	FastAPI endpoints for generate, retry, regenerate, and status	https://github.com/jamiepine/voicebox/blob/main/backend/routes/generations.py
backend/services/generation.py	Core orchestration (run_generation) and exception handling	https://github.com/jamiepine/voicebox/blob/main/backend/services/generation.py
backend/services/history.py	Helper functions to persist status updates and error messages	https://github.com/jamiepine/voicebox/blob/main/backend/services/history.py
backend/app.py	Application startup logic including stale-generation cleanup	https://github.com/jamiepine/voicebox/blob/main/backend/app.py
backend/routes/models.py	Model download and availability status endpoints	https://github.com/jamiepine/voicebox/blob/main/backend/routes/models.py

Summary

• Voicebox generation failures are captured in backend/services/generation.py, logged to the generations table with status "failed", and expose the error message via the status endpoint.
• Retry (/generate/{id}/retry) re-queues the exact same parameters and seed, useful for transient hardware or network issues.
• Regenerate (/generate/{id}/regenerate) creates a new database entry with seed=None to produce a variation, requiring the original to be "completed".
• Stale records from server crashes are automatically cleaned up on startup in backend/app.py by marking records older than five minutes as failed.
• Model availability should be verified at /models/status when generations fail immediately upon submission.

Frequently Asked Questions

How do I check the status of a Voicebox generation?

Send a GET request to /generate/{generation_id}/status. The endpoint returns a JSON object containing the current status (generating, completed, or failed) and an error field populated only when the status is failed. This endpoint is defined in backend/routes/generations.py and queries the generations table directly.

What is the difference between retry and regenerate in Voicebox?

Retry (POST /generate/{id}/retry) is for failed generations only and re-runs the synthesis with the exact same seed value to produce identical output, making it ideal for recovering from transient errors. Regenerate (POST /generate/{id}/regenerate) requires a completed generation and creates a new database record with seed=None, triggering a fresh synthesis that yields a different vocal variation while using the same text and voice profile.

Why does my Voicebox generation get stuck in "generating" status?

If the server process crashes or is killed during synthesis, the database record remains in "generating" status. The startup routine in backend/app.py automatically runs a SQL update to mark any "generating" rows older than five minutes as "failed". Once marked failed, you can manually trigger a recovery using the retry endpoint.

Where are Voicebox generation errors logged?

Errors are logged in two places: the error column of the generations table (accessible via the status endpoint) and the server logs via logger.exception calls in backend/services/generation.py. Check your process manager output (e.g., docker logs or systemd journal) for full Python stack traces when troubleshooting persistent failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}