# How to Debug Request Failures Using Trace and Request Logs in AxonHub

> Debug request failures in AxonHub with AH Trace and Request IDs. Learn how to use these headers and logs for efficient end-to-end troubleshooting in your repository.

- Repository: [Loop/axonhub](https://github.com/looplj/axonhub)
- Tags: how-to-guide
- Published: 2026-03-06

---

**AxonHub automatically injects `AH-Trace-Id` and `AH-Request-Id` headers into every HTTP request, propagating these identifiers through the context to enrich all log entries and database records for end-to-end debugging.**

When troubleshooting failed requests in the `looplj/axonhub` repository, understanding how to leverage trace and request logs is essential. The platform implements a comprehensive tracing middleware chain that captures correlation IDs at the edge and persists them through every layer of the stack. This guide explains how to extract these identifiers from failed responses and use them to pinpoint root causes in application logs and the database.

## Understanding Trace and Request ID Injection

AxonHub uses two distinct identifiers to track requests: a **trace ID** (`AH-Trace-Id`) that represents a logical transaction across multiple requests, and a **request ID** (`AH-Request-Id`) that uniquely identifies a single HTTP request. These IDs are generated and injected by middleware before your handler executes.

### How IDs Are Generated in the Middleware Chain

The injection happens in [`internal/server/middleware/logging.go`](https://github.com/looplj/axonhub/blob/main/internal/server/middleware/logging.go) within the `WithLoggingTracing` function. This middleware checks for an existing trace header from the client or generates a new one using `tracing.GenerateTraceID()`, then creates a fresh request ID via `tracing.GenerateRequestID()`. Both values are stored in the request context using `tracing.WithTraceID()` and `tracing.WithRequestID()`.

The [`internal/server/middleware/trace.go`](https://github.com/looplj/axonhub/blob/main/internal/server/middleware/trace.go) file contains the `WithTrace` middleware, which extracts the trace ID from the context and either retrieves or creates a corresponding `Trace` entity in the database. This middleware enriches the context further using `contexts.WithTrace()` so that downstream components can access the full trace object.

### Context Propagation and Log Enrichment

Once the IDs are in the context, the custom logging helpers in `internal/log/*` automatically extract them. When you call `log.Error(ctx, "message", ...)` or `log.Warn(ctx, "message", ...)`, the logger retrieves the trace and request IDs from the context and includes them in the structured output. This ensures every log line emitted during request processing is automatically correlated.

## Locating Request Failures in Logs

When a request fails, the response headers contain the exact identifiers you need to filter the log stream. The server writes the `AH-Request-Id` back to the response headers, while the `AH-Trace-Id` is either client-provided or server-generated.

### Capturing Response Headers

Make your request and inspect the response headers to capture the IDs:

```bash
curl -i http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'

# Look for these headers in the response:

# AH-Request-Id: ar-<uuid>

# AH-Trace-Id: at-<uuid>

```

### Filtering Logs by Request and Trace IDs

Once you have the IDs, search your log aggregation system or log files:

```bash

# Find all logs for a specific request

grep "ar-1234abcd-5678-90ef-1234-567890abcdef" /var/log/axonhub/server.log

# Find all logs for the entire trace (may span multiple requests)

grep "at-5678efgh-1234-90ab-cdef-1234567890ab" /var/log/axonhub/server.log

```

The trace ID is particularly useful when debugging issues that involve multiple internal requests or retries, as it links all related operations across the distributed trace.

## Database-Level Trace Investigation

The trace ID corresponds to a persistent record in the `traces` table. When the `WithTrace` middleware processes a request, it calls `traceService.GetOrCreateTrace()`, which either retrieves an existing trace by ID or creates a new row.

To investigate a failure at the database level:

```sql
-- Query the trace record using the ID from your logs
SELECT * FROM axonhub.traces 
WHERE trace_id = 'at-5678efgh-1234-90ab-cdef-1234567890ab';

```

This table typically contains metadata about the trace origin, creation time, and associated provider or model information, giving you additional context about the request lineage.

## Common Debugging Pitfalls and Solutions

| Symptom | Root Cause | Solution |
|---------|------------|----------|
| **No `AH-Trace-Id` in logs** | The request body was consumed before `WithLoggingTracing` middleware executed, preventing ID generation. | Ensure `WithLoggingTracing` is registered as the **first** middleware in the chain in [`internal/server/server.go`](https://github.com/looplj/axonhub/blob/main/internal/server/server.go). |
| **Duplicate request IDs** | The client manually sets `AH-Request-Id` header. The middleware respects this for the response header but generates a separate internal ID for logging. | Avoid manually setting `AH-Request-Id` unless propagating from upstream services; let the server generate unique IDs. |
| **Trace not persisted in database** | `WithTrace` middleware cannot extract a trace ID from headers, body fields, or AI provider-specific extraction (Claude/Codex) is disabled. | Enable appropriate extraction flags in configuration: `trace.claude_code_trace_enabled`, `trace.codex_trace_enabled`, `trace.extra_trace_headers`, or `trace.extra_trace_body_fields`. |

## Practical Code Example

When implementing custom handlers, you can access the trace and request IDs directly from the context to add custom debugging information:

```go
func chatHandler(c *gin.Context) {
    ctx := c.Request.Context()

    // Retrieve IDs for custom logging or error reporting
    traceID, hasTrace := tracing.GetTraceID(ctx)
    reqID, hasReq := tracing.GetRequestID(ctx)

    // Example: Custom validation failure logging
    if c.Query("model") == "" {
        log.Error(ctx,
            "Chat request missing required 'model' parameter",
            log.String("trace_id", traceID),
            log.String("request_id", reqID))
        
        c.JSON(http.StatusBadRequest, gin.H{
            "error": "model is required",
            "request_id": reqID,
        })
        return
    }

    // Normal processing continues...
}

```

This pattern mirrors the internal implementation where `log.Error`, `log.Warn`, and `log.Debug` automatically extract these values from the context, ensuring consistent correlation across your entire application stack.

## Summary

- **AxonHub** automatically injects `AH-Trace-Id` and `AH-Request-Id` headers into every HTTP request via the `WithLoggingTracing` middleware in [`internal/server/middleware/logging.go`](https://github.com/looplj/axonhub/blob/main/internal/server/middleware/logging.go).
- These identifiers propagate through the request context, allowing the logging helpers in `internal/log/*` to automatically attach them to every log entry.
- When debugging failures, capture the `AH-Request-Id` from response headers and search logs for this ID or the associated `AH-Trace-Id` to find all related log lines.
- The trace ID corresponds to a persistent record in the database `traces` table, accessible via `traceService.GetOrCreateTrace()` in [`internal/server/middleware/trace.go`](https://github.com/looplj/axonhub/blob/main/internal/server/middleware/trace.go).
- Ensure `WithLoggingTracing` is registered first in the middleware chain to avoid missing trace IDs when request bodies are consumed early.

## Frequently Asked Questions

### How do I find the trace ID for a failed request if I only have the request ID?

Search your logs for the request ID first. Every log line includes both the request ID and trace ID, so once you locate any log entry for that request, you can extract the associated trace ID. Alternatively, query the database `traces` table using the request ID if your application logs it there, though typically you would query by trace ID directly.

### Why are my logs missing the AH-Trace-Id even though the middleware is enabled?

This usually occurs when the request body is consumed by a different middleware or handler before `WithLoggingTracing` executes. Because `WithLoggingTracing` in [`internal/server/middleware/logging.go`](https://github.com/looplj/axonhub/blob/main/internal/server/middleware/logging.go) must read headers and potentially the body to extract or generate IDs, it must be registered as the first middleware in the chain in [`internal/server/server.go`](https://github.com/looplj/axonhub/blob/main/internal/server/server.go). Check your server initialization code to ensure the registration order is correct.

### Can I provide my own trace ID instead of letting AxonHub generate one?

Yes, you can send the `AH-Trace-Id` header in your client request, and the `WithLoggingTracing` middleware will respect and propagate this value rather than generating a new one. However, avoid manually setting the `AH-Request-Id` header unless you are propagating IDs from upstream services, as this can lead to duplicate ID conflicts in your logging system. The middleware generates unique request IDs automatically to ensure log correlation integrity.