# How the LangSmith Observability Integration Works in Omi: Runtime Tracing and Prompt Management

> Explore the LangSmith observability integration in Omi, featuring runtime tracing, prompt management, and user feedback. Learn how Omi streamlines LLM application development.

- Repository: [omi/omi](https://github.com/basedhardware/omi)
- Tags: deep-dive
- Published: 2026-02-26

---

**The LangSmith observability integration in Omi enables runtime tracing, prompt versioning, and user feedback collection through environment-driven configuration and lazy-loaded Python utilities in the `backend/utils/observability/` directory.**

Omi leverages LangSmith to provide comprehensive observability for its AI-driven conversations. This integration allows developers to trace request execution, version system prompts, and collect user feedback without modifying application code. The implementation resides in two core utility modules that handle runtime tracing and prompt management respectively.

## Architecture of the LangSmith Observability Integration

The integration splits responsibilities between runtime tracing and prompt versioning:

- **Runtime tracing** ([`backend/utils/observability/langsmith.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/observability/langsmith.py)): Detects global tracing configuration, logs startup status, creates per-request tracer callbacks, and submits user feedback.
- **Prompt versioning** ([`backend/utils/observability/langsmith_prompts.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/observability/langsmith_prompts.py)): Fetches system-prompt templates from LangSmith, caches them with TTL, provides fallback hard-coded prompts, and exposes metadata for debugging.

## Enabling and Configuring LangSmith Tracing

Configuration relies on environment variables. Omi honors both the modern `LANGSMITH_*` prefix and the legacy `LANGCHAIN_*` prefix for backward compatibility.

Required variables:

- `LANGSMITH_TRACING` or `LANGCHAIN_TRACING_V2`: Set to `"true"` to enable **global** tracing.
- `LANGSMITH_API_KEY` or `LANGCHAIN_API_KEY`: Required for any LangSmith interaction.
- `LANGSMITH_PROJECT` or `LANGCHAIN_PROJECT`: Specifies the project name for run organization.
- `LANGSMITH_ENDPOINT` or `LANGCHAIN_ENDPOINT`: Optional custom API endpoint (defaults to `https://api.smith.langchain.com`).

The module provides helper functions to check these configurations:

- `is_langsmith_enabled()`: Returns `True` if tracing is globally enabled.
- `has_langsmith_api_key()`: Verifies API key presence.
- `get_langsmith_project()`: Retrieves the configured project name.
- `get_langsmith_endpoint()`: Returns the API endpoint URL.

## Runtime Tracing Implementation

### Startup Status Logging

When the backend initializes, `log_langsmith_status()` (lines 73-96 in [`langsmith.py`](https://github.com/basedhardware/omi/blob/main/langsmith.py)) executes to report the observability configuration:

- **GLOBAL tracing ENABLED**: Both tracing flag and API key are present.
- **Per-request tracing**: API key exists but global tracing is disabled (allows selective tracing).
- **DISABLED**: No API key configured.

### Per-Request Tracer Callbacks

For endpoints requiring tracing without global enablement, `get_chat_tracer_callbacks()` (lines 99-138) creates tracer instances:

```python
from utils.observability.langsmith import get_chat_tracer_callbacks

def handle_chat_request(session_id: str, user_id: str, messages: list):
    # Generate tracer callbacks for this specific run

    tracer_cbs = get_chat_tracer_callbacks(
        run_id=session_id,
        run_name="chat.agentic.stream",
        tags=["chat", "agentic"],
        metadata={"user_id": user_id}
    )
    
    # Pass callbacks to the LangChain runnable

    response = chat_chain.invoke(
        {"messages": messages},
        callbacks=tracer_cbs,
    )
    return response

```

This function:

1. Lazily imports `LangChainTracer` from `langchain_core.tracers` to avoid startup overhead when tracing is disabled.
2. Returns a list containing the tracer instance if the API key is present; otherwise returns an empty list.
3. Accepts `run_id`, `run_name`, `tags`, and `metadata` for granular run identification.

### User Feedback Collection

After conversation completion, `submit_langsmith_feedback()` (lines 140-185) records user ratings:

```python
from utils.observability.langsmith import submit_langsmith_feedback

def process_user_rating(run_id: str, is_positive: bool, comment: str = None):
    score = 1.0 if is_positive else 0.0
    success = submit_langsmith_feedback(
        run_id=run_id,
        score=score,
        comment=comment,
        feedback_key="user_feedback"
    )
    return success

```

The function instantiates `langsmith.Client`, calls `client.create_feedback()` with the run identifier, a numeric score (0.0–1.0), and optional comment, then logs the result.

## Prompt Versioning and Caching

### Cache Configuration and TTL

The prompt management module ([`backend/utils/observability/langsmith_prompts.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/observability/langsmith_prompts.py)) implements a time-based cache to reduce API calls. Configuration options:

- **Cache TTL**: Controlled by `OMI_LANGSMITH_PROMPT_CACHE_TTL_SECONDS` environment variable; defaults to 300 seconds (5 minutes) via `DEFAULT_CACHE_TTL_SECONDS`.
- **Cache key format**: `"agentic:{prompt_name}"` for agentic system prompts.

The `_get_cache_ttl()` function (lines 39-45) reads the environment variable, while `clear_prompt_cache()` (lines 19-23) provides a utility to empty the cache for testing.

### Fetching and Fallback Mechanism

The `get_agentic_system_prompt_template()` function (lines 42-78) orchestrates prompt retrieval:

1. **Cache check**: Looks for cached entry using key `"agentic:{prompt_name}"`.
2. **Fetch**: If cache miss, calls `_fetch_prompt_from_langsmith()` (lines 58-140), which uses `Client().pull_prompt()` to retrieve the template.
3. **Template extraction**: Handles multiple prompt types (ChatPromptTemplate, PromptTemplate, plain string) to extract the system message.
4. **Metadata**: Stores `prompt_commit` (commit hash from LangSmith) or timestamp-based placeholder; records `source="langsmith"`.
5. **Fallback**: On API failure or missing key, uses `_get_fallback_agentic_prompt_template()` (lines 25-67) and sets `source="fallback"`.

The `get_prompt_metadata()` function (lines 8-16) exposes `prompt_name`, `prompt_commit`, and `source` for debugging.

## Practical Implementation Examples

### Checking LangSmith Status at Startup

```python

# backend/main.py

from utils.observability.langsmith import log_langsmith_status

# Log configuration on application boot

log_langsmith_status()   # prints GLOBAL ENABLED, per-request, or DISABLED

```

### Attaching Tracing to LangChain Chains

```python
from utils.observability.langsmith import get_chat_tracer_callbacks

def handle_chat_request(session_id: str, user_id: str, messages: list):
    # Generate tracer callbacks for this specific run

    tracer_cbs = get_chat_tracer_callbacks(
        run_id=session_id,
        run_name="chat.agentic.stream",
        tags=["chat", "agentic"],
        metadata={"user_id": user_id}
    )
    
    # Pass callbacks to the LangChain runnable

    response = chat_chain.invoke(
        {"messages": messages},
        callbacks=tracer_cbs,
    )
    return response

```

### Recording User Feedback

```python
from utils.observability.langsmith import submit_langsmith_feedback

def process_user_rating(run_id: str, is_positive: bool, comment: str = None):
    score = 1.0 if is_positive else 0.0
    success = submit_langsmith_feedback(
        run_id=run_id,
        score=score,
        comment=comment,
        feedback_key="user_feedback"
    )
    return success

```

### Fetching Versioned System Prompts

```python
from utils.observability.langsmith_prompts import get_agentic_system_prompt_template, get_prompt_metadata

def build_system_prompt():
    # Fetch cached or fresh prompt from LangSmith

    prompt_data = get_agentic_system_prompt_template()
    
    # Access metadata for debugging

    meta = get_prompt_metadata()
    print(f"Using prompt: {meta['prompt_name']} (commit: {meta['prompt_commit']})")
    
    return prompt_data.template_text

```

### Clearing the Prompt Cache

```python
from utils.observability.langsmith_prompts import clear_prompt_cache

# Empty the in-memory cache (useful in tests or after prompt updates)

clear_prompt_cache()

```

## Summary

- **Environment-driven configuration**: Omi uses `LANGSMITH_*` and `LANGCHAIN_*` environment variables to control tracing, project assignment, and API authentication without code changes.
- **Lazy loading**: The integration imports LangSmith components only when needed, ensuring zero overhead when observability is disabled.
- **Dual-mode tracing**: Supports both global tracing (via `LANGSMITH_TRACING=true`) and per-request tracing (via `get_chat_tracer_callbacks()`) for flexible observability coverage.
- **Feedback loop**: The `submit_langsmith_feedback()` function enables explicit user ratings (0.0–1.0 scores) to be attached to specific run IDs for sentiment analysis.
- **Prompt lifecycle management**: System prompts are versioned in LangSmith, cached locally with a 5-minute TTL, and fallback to hard-coded templates when the service is unavailable.

## Frequently Asked Questions

### What environment variables are required to enable LangSmith tracing in Omi?

You need at minimum `LANGSMITH_API_KEY` (or the legacy `LANGCHAIN_API_KEY`) to authenticate with the LangSmith API. To enable global tracing, set `LANGSMITH_TRACING` (or `LANGCHAIN_TRACING_V2`) to `"true"`. Optional variables include `LANGSMITH_PROJECT` for run organization and `LANGSMITH_ENDPOINT` for custom API endpoints.

### How does Omi handle LangSmith outages or missing API keys?

When the API key is missing or the LangSmith service is unreachable, Omi degrades gracefully. The `get_chat_tracer_callbacks()` function returns an empty list, allowing LangChain chains to execute without tracing. For system prompts, the code falls back to `_get_fallback_agentic_prompt_template()`, a hard-coded template stored in [`langsmith_prompts.py`](https://github.com/basedhardware/omi/blob/main/langsmith_prompts.py), ensuring the application remains functional.

### Can I use per-request tracing without enabling global tracing?

Yes. Omi supports selective tracing via `get_chat_tracer_callbacks()`. Even when `LANGSMITH_TRACING` is not set to `"true"`, if you have configured `LANGSMITH_API_KEY`, this function will create a `LangChainTracer` instance for specific runs. This allows you to trace critical chat interactions without the overhead of tracing every LLM call in the application.

### How long are system prompts cached, and can I clear the cache?

By default, system prompts fetched from LangSmith are cached for 300 seconds (5 minutes). You can customize this duration by setting the `OMI_LANGSMITH_PROMPT_CACHE_TTL_SECONDS` environment variable. To clear the cache manually—for example, during testing or when deploying new prompt versions—call `clear_prompt_cache()` from [`backend/utils/observability/langsmith_prompts.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/observability/langsmith_prompts.py).