How the LangSmith Observability Integration Works in Omi: Runtime Tracing and Prompt Management

The LangSmith observability integration in Omi enables runtime tracing, prompt versioning, and user feedback collection through environment-driven configuration and lazy-loaded Python utilities in the backend/utils/observability/ directory.

Omi leverages LangSmith to provide comprehensive observability for its AI-driven conversations. This integration allows developers to trace request execution, version system prompts, and collect user feedback without modifying application code. The implementation resides in two core utility modules that handle runtime tracing and prompt management respectively.

Architecture of the LangSmith Observability Integration

The integration splits responsibilities between runtime tracing and prompt versioning:

• Runtime tracing (backend/utils/observability/langsmith.py): Detects global tracing configuration, logs startup status, creates per-request tracer callbacks, and submits user feedback.
• Prompt versioning (backend/utils/observability/langsmith_prompts.py): Fetches system-prompt templates from LangSmith, caches them with TTL, provides fallback hard-coded prompts, and exposes metadata for debugging.

Enabling and Configuring LangSmith Tracing

Configuration relies on environment variables. Omi honors both the modern LANGSMITH_* prefix and the legacy LANGCHAIN_* prefix for backward compatibility.

Required variables:

• LANGSMITH_TRACING or LANGCHAIN_TRACING_V2: Set to "true" to enable global tracing.
• LANGSMITH_API_KEY or LANGCHAIN_API_KEY: Required for any LangSmith interaction.
• LANGSMITH_PROJECT or LANGCHAIN_PROJECT: Specifies the project name for run organization.
• LANGSMITH_ENDPOINT or LANGCHAIN_ENDPOINT: Optional custom API endpoint (defaults to https://api.smith.langchain.com).

The module provides helper functions to check these configurations:

• is_langsmith_enabled(): Returns True if tracing is globally enabled.
• has_langsmith_api_key(): Verifies API key presence.
• get_langsmith_project(): Retrieves the configured project name.
• get_langsmith_endpoint(): Returns the API endpoint URL.

Runtime Tracing Implementation

Startup Status Logging

When the backend initializes, log_langsmith_status() (lines 73-96 in langsmith.py) executes to report the observability configuration:

• GLOBAL tracing ENABLED: Both tracing flag and API key are present.
• Per-request tracing: API key exists but global tracing is disabled (allows selective tracing).
• DISABLED: No API key configured.

Per-Request Tracer Callbacks

For endpoints requiring tracing without global enablement, get_chat_tracer_callbacks() (lines 99-138) creates tracer instances:

from utils.observability.langsmith import get_chat_tracer_callbacks

def handle_chat_request(session_id: str, user_id: str, messages: list):
    # Generate tracer callbacks for this specific run

    tracer_cbs = get_chat_tracer_callbacks(
        run_id=session_id,
        run_name="chat.agentic.stream",
        tags=["chat", "agentic"],
        metadata={"user_id": user_id}
    )
    
    # Pass callbacks to the LangChain runnable

    response = chat_chain.invoke(
        {"messages": messages},
        callbacks=tracer_cbs,
    )
    return response

This function:

1. Lazily imports LangChainTracer from langchain_core.tracers to avoid startup overhead when tracing is disabled.
2. Returns a list containing the tracer instance if the API key is present; otherwise returns an empty list.
3. Accepts run_id, run_name, tags, and metadata for granular run identification.

User Feedback Collection

After conversation completion, submit_langsmith_feedback() (lines 140-185) records user ratings:

from utils.observability.langsmith import submit_langsmith_feedback

def process_user_rating(run_id: str, is_positive: bool, comment: str = None):
    score = 1.0 if is_positive else 0.0
    success = submit_langsmith_feedback(
        run_id=run_id,
        score=score,
        comment=comment,
        feedback_key="user_feedback"
    )
    return success

The function instantiates langsmith.Client, calls client.create_feedback() with the run identifier, a numeric score (0.0–1.0), and optional comment, then logs the result.

Prompt Versioning and Caching

Cache Configuration and TTL

The prompt management module (backend/utils/observability/langsmith_prompts.py) implements a time-based cache to reduce API calls. Configuration options:

• Cache TTL: Controlled by OMI_LANGSMITH_PROMPT_CACHE_TTL_SECONDS environment variable; defaults to 300 seconds (5 minutes) via DEFAULT_CACHE_TTL_SECONDS.
• Cache key format: "agentic:{prompt_name}" for agentic system prompts.

The _get_cache_ttl() function (lines 39-45) reads the environment variable, while clear_prompt_cache() (lines 19-23) provides a utility to empty the cache for testing.

Fetching and Fallback Mechanism

The get_agentic_system_prompt_template() function (lines 42-78) orchestrates prompt retrieval:

1. Cache check: Looks for cached entry using key "agentic:{prompt_name}".
2. Fetch: If cache miss, calls _fetch_prompt_from_langsmith() (lines 58-140), which uses Client().pull_prompt() to retrieve the template.
3. Template extraction: Handles multiple prompt types (ChatPromptTemplate, PromptTemplate, plain string) to extract the system message.
4. Metadata: Stores prompt_commit (commit hash from LangSmith) or timestamp-based placeholder; records source="langsmith".
5. Fallback: On API failure or missing key, uses _get_fallback_agentic_prompt_template() (lines 25-67) and sets source="fallback".

The get_prompt_metadata() function (lines 8-16) exposes prompt_name, prompt_commit, and source for debugging.

Practical Implementation Examples

Checking LangSmith Status at Startup

# backend/main.py

from utils.observability.langsmith import log_langsmith_status

# Log configuration on application boot

log_langsmith_status()   # prints GLOBAL ENABLED, per-request, or DISABLED

Attaching Tracing to LangChain Chains

from utils.observability.langsmith import get_chat_tracer_callbacks

def handle_chat_request(session_id: str, user_id: str, messages: list):
    # Generate tracer callbacks for this specific run

    tracer_cbs = get_chat_tracer_callbacks(
        run_id=session_id,
        run_name="chat.agentic.stream",
        tags=["chat", "agentic"],
        metadata={"user_id": user_id}
    )
    
    # Pass callbacks to the LangChain runnable

    response = chat_chain.invoke(
        {"messages": messages},
        callbacks=tracer_cbs,
    )
    return response

Recording User Feedback

from utils.observability.langsmith import submit_langsmith_feedback

def process_user_rating(run_id: str, is_positive: bool, comment: str = None):
    score = 1.0 if is_positive else 0.0
    success = submit_langsmith_feedback(
        run_id=run_id,
        score=score,
        comment=comment,
        feedback_key="user_feedback"
    )
    return success

Fetching Versioned System Prompts

from utils.observability.langsmith_prompts import get_agentic_system_prompt_template, get_prompt_metadata

def build_system_prompt():
    # Fetch cached or fresh prompt from LangSmith

    prompt_data = get_agentic_system_prompt_template()
    
    # Access metadata for debugging

    meta = get_prompt_metadata()
    print(f"Using prompt: {meta['prompt_name']} (commit: {meta['prompt_commit']})")
    
    return prompt_data.template_text

Clearing the Prompt Cache

from utils.observability.langsmith_prompts import clear_prompt_cache

# Empty the in-memory cache (useful in tests or after prompt updates)

clear_prompt_cache()

Summary

• Environment-driven configuration: Omi uses LANGSMITH_* and LANGCHAIN_* environment variables to control tracing, project assignment, and API authentication without code changes.
• Lazy loading: The integration imports LangSmith components only when needed, ensuring zero overhead when observability is disabled.
• Dual-mode tracing: Supports both global tracing (via LANGSMITH_TRACING=true) and per-request tracing (via get_chat_tracer_callbacks()) for flexible observability coverage.
• Feedback loop: The submit_langsmith_feedback() function enables explicit user ratings (0.0–1.0 scores) to be attached to specific run IDs for sentiment analysis.
• Prompt lifecycle management: System prompts are versioned in LangSmith, cached locally with a 5-minute TTL, and fallback to hard-coded templates when the service is unavailable.

Frequently Asked Questions

What environment variables are required to enable LangSmith tracing in Omi?

You need at minimum LANGSMITH_API_KEY (or the legacy LANGCHAIN_API_KEY) to authenticate with the LangSmith API. To enable global tracing, set LANGSMITH_TRACING (or LANGCHAIN_TRACING_V2) to "true". Optional variables include LANGSMITH_PROJECT for run organization and LANGSMITH_ENDPOINT for custom API endpoints.

How does Omi handle LangSmith outages or missing API keys?

When the API key is missing or the LangSmith service is unreachable, Omi degrades gracefully. The get_chat_tracer_callbacks() function returns an empty list, allowing LangChain chains to execute without tracing. For system prompts, the code falls back to _get_fallback_agentic_prompt_template(), a hard-coded template stored in langsmith_prompts.py, ensuring the application remains functional.

Can I use per-request tracing without enabling global tracing?

Yes. Omi supports selective tracing via get_chat_tracer_callbacks(). Even when LANGSMITH_TRACING is not set to "true", if you have configured LANGSMITH_API_KEY, this function will create a LangChainTracer instance for specific runs. This allows you to trace critical chat interactions without the overhead of tracing every LLM call in the application.

How long are system prompts cached, and can I clear the cache?

By default, system prompts fetched from LangSmith are cached for 300 seconds (5 minutes). You can customize this duration by setting the OMI_LANGSMITH_PROMPT_CACHE_TTL_SECONDS environment variable. To clear the cache manually—for example, during testing or when deploying new prompt versions—call clear_prompt_cache() from backend/utils/observability/langsmith_prompts.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}