How to Debug Request Failures Using Trace and Request Logs in AxonHub

AxonHub automatically injects AH-Trace-Id and AH-Request-Id headers into every HTTP request, propagating these identifiers through the context to enrich all log entries and database records for end-to-end debugging.

When troubleshooting failed requests in the looplj/axonhub repository, understanding how to leverage trace and request logs is essential. The platform implements a comprehensive tracing middleware chain that captures correlation IDs at the edge and persists them through every layer of the stack. This guide explains how to extract these identifiers from failed responses and use them to pinpoint root causes in application logs and the database.

Understanding Trace and Request ID Injection

AxonHub uses two distinct identifiers to track requests: a trace ID (AH-Trace-Id) that represents a logical transaction across multiple requests, and a request ID (AH-Request-Id) that uniquely identifies a single HTTP request. These IDs are generated and injected by middleware before your handler executes.

How IDs Are Generated in the Middleware Chain

The injection happens in internal/server/middleware/logging.go within the WithLoggingTracing function. This middleware checks for an existing trace header from the client or generates a new one using tracing.GenerateTraceID(), then creates a fresh request ID via tracing.GenerateRequestID(). Both values are stored in the request context using tracing.WithTraceID() and tracing.WithRequestID().

The internal/server/middleware/trace.go file contains the WithTrace middleware, which extracts the trace ID from the context and either retrieves or creates a corresponding Trace entity in the database. This middleware enriches the context further using contexts.WithTrace() so that downstream components can access the full trace object.

Context Propagation and Log Enrichment

Once the IDs are in the context, the custom logging helpers in internal/log/* automatically extract them. When you call log.Error(ctx, "message", ...) or log.Warn(ctx, "message", ...), the logger retrieves the trace and request IDs from the context and includes them in the structured output. This ensures every log line emitted during request processing is automatically correlated.

Locating Request Failures in Logs

When a request fails, the response headers contain the exact identifiers you need to filter the log stream. The server writes the AH-Request-Id back to the response headers, while the AH-Trace-Id is either client-provided or server-generated.

Capturing Response Headers

Make your request and inspect the response headers to capture the IDs:

curl -i http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'

# Look for these headers in the response:

# AH-Request-Id: ar-<uuid>

# AH-Trace-Id: at-<uuid>

Filtering Logs by Request and Trace IDs

Once you have the IDs, search your log aggregation system or log files:

# Find all logs for a specific request

grep "ar-1234abcd-5678-90ef-1234-567890abcdef" /var/log/axonhub/server.log

# Find all logs for the entire trace (may span multiple requests)

grep "at-5678efgh-1234-90ab-cdef-1234567890ab" /var/log/axonhub/server.log

The trace ID is particularly useful when debugging issues that involve multiple internal requests or retries, as it links all related operations across the distributed trace.

Database-Level Trace Investigation

The trace ID corresponds to a persistent record in the traces table. When the WithTrace middleware processes a request, it calls traceService.GetOrCreateTrace(), which either retrieves an existing trace by ID or creates a new row.

To investigate a failure at the database level:

-- Query the trace record using the ID from your logs
SELECT * FROM axonhub.traces 
WHERE trace_id = 'at-5678efgh-1234-90ab-cdef-1234567890ab';

This table typically contains metadata about the trace origin, creation time, and associated provider or model information, giving you additional context about the request lineage.

Common Debugging Pitfalls and Solutions

Symptom	Root Cause	Solution
No AH-Trace-Id in logs	The request body was consumed before WithLoggingTracing middleware executed, preventing ID generation.	Ensure WithLoggingTracing is registered as the first middleware in the chain in internal/server/server.go.
Duplicate request IDs	The client manually sets AH-Request-Id header. The middleware respects this for the response header but generates a separate internal ID for logging.	Avoid manually setting AH-Request-Id unless propagating from upstream services; let the server generate unique IDs.
Trace not persisted in database	WithTrace middleware cannot extract a trace ID from headers, body fields, or AI provider-specific extraction (Claude/Codex) is disabled.	Enable appropriate extraction flags in configuration: trace.claude_code_trace_enabled, trace.codex_trace_enabled, trace.extra_trace_headers, or trace.extra_trace_body_fields.

Practical Code Example

When implementing custom handlers, you can access the trace and request IDs directly from the context to add custom debugging information:

func chatHandler(c *gin.Context) {
    ctx := c.Request.Context()

    // Retrieve IDs for custom logging or error reporting
    traceID, hasTrace := tracing.GetTraceID(ctx)
    reqID, hasReq := tracing.GetRequestID(ctx)

    // Example: Custom validation failure logging
    if c.Query("model") == "" {
        log.Error(ctx,
            "Chat request missing required 'model' parameter",
            log.String("trace_id", traceID),
            log.String("request_id", reqID))
        
        c.JSON(http.StatusBadRequest, gin.H{
            "error": "model is required",
            "request_id": reqID,
        })
        return
    }

    // Normal processing continues...
}

This pattern mirrors the internal implementation where log.Error, log.Warn, and log.Debug automatically extract these values from the context, ensuring consistent correlation across your entire application stack.

Summary

• AxonHub automatically injects AH-Trace-Id and AH-Request-Id headers into every HTTP request via the WithLoggingTracing middleware in internal/server/middleware/logging.go.
• These identifiers propagate through the request context, allowing the logging helpers in internal/log/* to automatically attach them to every log entry.
• When debugging failures, capture the AH-Request-Id from response headers and search logs for this ID or the associated AH-Trace-Id to find all related log lines.
• The trace ID corresponds to a persistent record in the database traces table, accessible via traceService.GetOrCreateTrace() in internal/server/middleware/trace.go.
• Ensure WithLoggingTracing is registered first in the middleware chain to avoid missing trace IDs when request bodies are consumed early.

Frequently Asked Questions

How do I find the trace ID for a failed request if I only have the request ID?

Search your logs for the request ID first. Every log line includes both the request ID and trace ID, so once you locate any log entry for that request, you can extract the associated trace ID. Alternatively, query the database traces table using the request ID if your application logs it there, though typically you would query by trace ID directly.

Why are my logs missing the AH-Trace-Id even though the middleware is enabled?

This usually occurs when the request body is consumed by a different middleware or handler before WithLoggingTracing executes. Because WithLoggingTracing in internal/server/middleware/logging.go must read headers and potentially the body to extract or generate IDs, it must be registered as the first middleware in the chain in internal/server/server.go. Check your server initialization code to ensure the registration order is correct.

Can I provide my own trace ID instead of letting AxonHub generate one?

Yes, you can send the AH-Trace-Id header in your client request, and the WithLoggingTracing middleware will respect and propagate this value rather than generating a new one. However, avoid manually setting the AH-Request-Id header unless you are propagating IDs from upstream services, as this can lead to duplicate ID conflicts in your logging system. The middleware generates unique request IDs automatically to ensure log correlation integrity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}