# Optimal Redis Configuration Options for Caching and Session Management in LMForge

> Discover optimal Redis configuration for LMForge caching and session management. Secure your setup with SSL, segment databases, and tune connection pools for high throughput.

- Repository: [Haohao/lmforge-end-to-end-llmops-platform-for-multi-model-agents](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents)
- Tags: performance
- Published: 2026-03-03

---

**The optimal Redis configuration for the LMForge LLMOps platform requires enabling SSL encryption for network security, separating logical databases for distributed locks and Celery task queues, and tuning connection pools with authentication to support high-throughput multi-model agent operations.**

The LMForge end-to-end LLMOps platform for multi-model agents relies on Redis as a lightweight, in-memory store for critical infrastructure including distributed locking, short-term caching, and Celery message brokering. Understanding the optimal Redis configuration options for caching and session management ensures stable, secure, and performant operation of the platform's concurrent document processing and asynchronous task execution.

## Redis Architecture and Use Cases in LMForge

The platform utilizes Redis across three distinct architectural patterns according to the source code in `haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents`:

- **Distributed locking and short-term cache**: Implemented in [`api/internal/entity/cache_entity.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/entity/cache_entity.py) using lock keys such as `LOCK_DOCUMENT_UPDATE_ENABLED` with a default TTL of **600 seconds** to protect concurrent updates of documents, keywords, and segments.
- **Celery broker and result backend**: Configured in [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py) within the `CELERY` dictionary to manage task queues and asynchronous processing results.
- **General Redis client**: Centralized in [`api/internal/extension/redis_extension.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/extension/redis_extension.py) as `redis_client`, injected into services like `DocumentService` and `KeywordTableService`.

Because the platform depends on Redis for **fast, atomic operations** (locks) and **message passing** (Celery), the configuration must prioritize stability, security, and workload-specific tuning.

## Environment Configuration and Connection Pool Setup

The platform loads Redis settings from environment variables with sensible defaults defined in [`api/config/default_config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/default_config.py):

```python

# api/config/default_config.py

DEFAULT_CONFIG = {
    "REDIS_HOST": "localhost",
    "REDIS_PORT": 6379,
    "REDIS_USERNAME": "",
    "REDIS_PASSWORD": "",
    "REDIS_DB": 0,
    "REDIS_USE_SSL": "False",
}

```

At runtime, [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py) reads these variables and exposes them as configuration attributes:

```python

# api/config/config.py

self.REDIS_HOST = _get_env("REDIS_HOST")
self.REDIS_PORT = _get_env("REDIS_PORT")
self.REDIS_USERNAME = _get_env("REDIS_USERNAME")
self.REDIS_PASSWORD = _get_env("REDIS_PASSWORD")
self.REDIS_DB = _get_env("REDIS_DB")
self.REDIS_USE_SSL = _get_bool_env("REDIS_USE_SSL")

```

The **Redis extension** creates a connection pool in [`api/internal/extension/redis_extension.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/extension/redis_extension.py) by selecting either `Connection` or `SSLConnection` based on the `REDIS_USE_SSL` flag:

```python

# api/internal/extension/redis_extension.py

import redis
from redis.connection import Connection, SSLConnection

redis_client = redis.Redis()

def init_app(app: Flask):
    connection_class = Connection
    if app.config.get("REDIS_USE_SSL", False):
        connection_class = SSLConnection

    redis_client.connection_pool = redis.ConnectionPool(**{
        "host": app.config.get("REDIS_HOST", "localhost"),
        "port": app.config.get("REDIS_PORT", 6379),
        "username": app.config.get("REDIS_USERNAME", None),
        "password": app.config.get("REDIS_PASSWORD", None),
        "db": app.config.get("REDIS_DB", 0),
        "encoding": "utf-8",
        "decode_responses": False,
    }, connection_class=connection_class)

    app.extensions["redis"] = redis_client

```

The client is then bound to the Flask-Injector module in [`api/app/http/module.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/app/http/module.py) for dependency injection:

```python

# api/app/http/module.py

binder.bind(Redis, to=redis_client)

```

## Production-Ready Redis Configuration Options

For optimal caching and session management in production environments, configure the following settings:

**`REDIS_HOST` and `REDIS_PORT`**
Point to your dedicated Redis instance or managed service endpoint (e.g., `redis.production.internal` on port `6379`).

**`REDIS_DB` (Logical Database Separation)**
Maintain **DB 0** for the platform's lock cache and allocate **DB 1** for Celery operations (`CELERY_BROKER_DB` / `CELERY_RESULT_BACKEND_DB`). This separation prevents lock entries from being evicted by Celery's TTL policies.

**Authentication (`REDIS_USERNAME` and `REDIS_PASSWORD`)**
Enforce authentication via environment variables, especially in multi-tenant environments. Never hardcode credentials in [`default_config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/default_config.py).

**`REDIS_USE_SSL`**
Enable SSL encryption (`True`) when accessing Redis over untrusted networks or cloud VPCs. The extension automatically switches to `SSLConnection` when this flag is active.

**Connection Pool Sizing**
High-throughput request handling can exhaust the default pool size. Add `"max_connections": 100` (or higher based on traffic) to the `ConnectionPool` constructor in [`redis_extension.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/redis_extension.py).

**`decode_responses`**
Set to `True` to receive Python `str` objects instead of `bytes` when storing plain strings or JSON values, unless deliberately handling binary data.

**Socket Keepalive and Retry Logic**
Extend the connection dictionary with `"socket_keepalive": True` and `"retry_on_timeout": True` to improve resilience against network blips and `ConnectionError` scenarios.

**Lock TTL (`LOCK_EXPIRE_TIME`)**
The default **600 seconds** defined in [`api/internal/entity/cache_entity.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/entity/cache_entity.py) prevents deadlocks if a process crashes while holding a lock. Adjust based on your longest expected critical section duration.

**Production Connection Pool Example:**

```python
redis_client.connection_pool = redis.ConnectionPool(**{
    "host": app.config.get("REDIS_HOST", "localhost"),
    "port": app.config.get("REDIS_PORT", 6379),
    "username": app.config.get("REDIS_USERNAME"),
    "password": app.config.get("REDIS_PASSWORD"),
    "db": app.config.get("REDIS_DB", 0),
    "encoding": "utf-8",
    "decode_responses": True,
    "max_connections": 100,
    "socket_keepalive": True,
    "retry_on_timeout": True,
}, connection_class=connection_class)

```

## Implementing Distributed Locks with Redis

The platform uses Redis for atomic locking during document updates. In [`api/internal/service/document_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/document_service.py), the service checks for existing locks before processing:

```python

# api/internal/service/document_service.py

cache_key = LOCK_DOCUMENT_UPDATE_ENABLED.format(document_id=document.id)
cache_result = self.redis_client.get(cache_key)
if cache_result is not None:
    raise FailException("当前文档正在修改启用状态，请稍后再次尝试")

# Perform update operations...

self.redis_client.setex(cache_key, LOCK_EXPIRE_TIME, 1)

```

**Key implementation details:**
- Lock keys follow a namespace pattern defined in [`api/internal/entity/cache_entity.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/entity/cache_entity.py) (e.g., `lock:document:update:enabled_{document_id}`)
- `setex` ensures automatic expiration, preventing permanent deadlocks
- All services follow this consistent pattern for cache management across the platform

## Celery Broker and Result Backend Configuration

Celery utilizes the same Redis infrastructure but targets separate logical databases to avoid interference with caching operations. The configuration is assembled in [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py):

```python

# api/config/config.py

self.CELERY = {
    "broker_url": f"redis://{self.REDIS_USERNAME}:{self.REDIS_PASSWORD}"
                  f"@{self.REDIS_HOST}:{self.REDIS_PORT}/{int(_get_env('CELERY_BROKER_DB'))}",
    "result_backend": f"redis://{self.REDIS_USERNAME}:{self.REDIS_PASSWORD}"
                      f"@{self.REDIS_HOST}:{self.REDIS_PORT}/{int(_get_env('CELERY_RESULT_BACKEND_DB'))}",
}

```

**Best practice**: Keep Celery databases distinct from the lock cache database (DB 0). The default configuration in [`default_config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/default_config.py) typically assigns DB 1 for these purposes, ensuring task queue operations do not affect distributed locking availability.

## Summary

- **Enable SSL encryption** via `REDIS_USE_SSL=True` for any production deployment accessing Redis over a network.
- **Separate logical databases** using DB 0 for distributed locks and DB 1 for Celery broker/result backend operations.
- **Configure connection pools** with increased `max_connections` (100+) and enable `socket_keepalive` and `retry_on_timeout` for high-availability scenarios.
- **Implement authentication** using `REDIS_USERNAME` and `REDIS_PASSWORD` environment variables without hardcoding credentials.
- **Maintain default lock TTL** at 600 seconds or adjust based on critical section duration requirements.
- **Use dependency injection** via [`api/app/http/module.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/app/http/module.py) rather than instantiating manual Redis clients throughout the codebase.

## Frequently Asked Questions

### How does LMForge handle Redis connection security in production?

LMForge handles Redis connection security through the `REDIS_USE_SSL` configuration flag in [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py). When set to `True`, the `init_app` function in [`api/internal/extension/redis_extension.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/extension/redis_extension.py) automatically switches from the standard `Connection` class to `SSLConnection`, encrypting all traffic between the application and the Redis server. Additionally, the platform supports username and password authentication via `REDIS_USERNAME` and `REDIS_PASSWORD` environment variables to prevent unauthorized access in multi-tenant environments.

### Why should I use separate Redis databases for locks and Celery tasks?

You should separate logical databases because the platform uses **DB 0** for distributed locking (with keys like `LOCK_DOCUMENT_UPDATE_ENABLED`) and recommends **DB 1** for Celery broker and result backend operations. This isolation prevents Celery's aggressive TTL policies and high write volumes from evicting or interfering with critical lock entries that protect document consistency during concurrent updates.

### What is the optimal connection pool size for high-traffic LMForge deployments?

For high-traffic deployments handling concurrent document updates and multi-model agent operations, increase the `max_connections` parameter in the `ConnectionPool` constructor within [`api/internal/extension/redis_extension.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/extension/redis_extension.py) from the default (typically 10) to **100 or higher**. This accommodates the simultaneous connections required by `DocumentService`, `KeywordTableService`, and Celery workers without exhausting the pool and causing connection timeouts.

### How does the platform prevent deadlocks when using Redis distributed locks?

The platform prevents deadlocks by setting an explicit expiration time on all lock keys using the `setex` command with a `LOCK_EXPIRE_TIME` of **600 seconds** (10 minutes) by default. This TTL ensures that if a process crashes or hangs while holding a lock in `DocumentService` or related services, the lock automatically expires after the configured duration, allowing other processes to acquire the lock and continue operations.