# How the Adaptive Cache Manager Handles Expired and Stale Data in TradingAgents-CN

> Discover how the adaptive cache manager in TradingAgents-CN prevents stale data. Learn about TTL validation, backend expiration, and cache cleanup for reliable trading pipelines.

- Repository: [hsliuping/TradingAgents-CN](https://github.com/hsliuping/tradingagents-cn)
- Tags: internals
- Published: 2026-02-16

---

**The adaptive cache manager in TradingAgents-CN prevents stale data from entering trading pipelines through TTL-based validation, backend-native expiration mechanisms, and periodic file cache cleanup.**

The **adaptive cache manager** is a critical component in the TradingAgents-CN repository that intelligently routes market data across multiple storage backends while ensuring expired entries never contaminate trading decisions. By combining timestamp validation with backend-specific expiration features, the system maintains data freshness across file system, Redis, and MongoDB deployments.

## TTL-Based Validation for All Backends

Every cache entry carries a `datetime` timestamp recording when it was written. The manager calculates a **time-to-live (TTL)** in seconds based on the symbol's market and data type using `_get_ttl_seconds`. When reading an entry, the `_is_cache_valid` method compares the stored timestamp plus TTL against the current time.

If the computed expiry time has passed, the entry is treated as stale and ignored. This validation runs uniformly across all three backends—file, Redis, and MongoDB—ensuring consistent expiration semantics regardless of storage medium.

```python

# From tradingagents/dataflows/cache/adaptive.py (lines 64-70)

def _is_cache_valid(self, cache_data: Dict) -> bool:
    """Check if cached data is still valid based on TTL."""
    timestamp = cache_data.get('timestamp')
    if not timestamp:
        return False
    
    ttl = cache_data.get('ttl', self.default_ttl)
    return datetime.now() - timestamp < timedelta(seconds=ttl)

```

## Backend-Native Expiration Mechanisms

While TTL validation catches expired entries during reads, the adaptive cache manager also leverages native expiration features in Redis and MongoDB to proactively remove stale data without manual intervention.

### Redis Automatic Key Expiration

When persisting to Redis, the manager uses the `setex` command in `_save_to_redis`, which atomically sets the key value and its TTL in seconds. Redis automatically deletes the key once the TTL elapses, preventing memory bloat from obsolete market data.

```python

# From tradingagents/dataflows/cache/adaptive.py (lines 124-127)

def _save_to_redis(self, key: str, data: Dict, ttl: int):
    """Save data to Redis with automatic expiration."""
    serialized = json.dumps(data, default=str)
    self.redis_client.setex(key, ttl, serialized)

```

### MongoDB TTL Indexes

For MongoDB storage, the `_save_to_mongodb` method injects an `expires_at` field calculated as `datetime.now() + timedelta(seconds=ttl)`. The MongoDB collection maintains a TTL index on this field, causing the database to automatically purge documents when their `expires_at` time passes.

```python

# From tradingagents/dataflows/cache/adaptive.py (lines 182-184)

def _save_to_mongodb(self, key: str, data: Dict, ttl: int):
    """Save data to MongoDB with expiration timestamp."""
    expires_at = datetime.now() + timedelta(seconds=ttl)
    document = {
        'key': key,
        'data': data,
        'expires_at': expires_at,
        'created_at': datetime.now()
    }
    self.mongo_collection.insert_one(document)

```

## Periodic Cleanup of File Cache

Unlike Redis and MongoDB, the file system backend lacks native expiration capabilities. The adaptive cache manager implements `clear_expired_cache` to handle stale file entries explicitly.

This method walks through the `data/cache` directory, deserializes each pickled file, validates freshness using `_is_cache_valid`, and deletes any file that has surpassed its TTL. This cleanup prevents the file cache from accumulating obsolete market data during long-running deployments.

```python

# From tradingagents/dataflows/cache/adaptive.py (lines 404-410)

def clear_expired_cache(self):
    """Remove expired entries from file cache."""
    cache_dir = Path("data/cache")
    if not cache_dir.exists():
        return
    
    removed_count = 0
    for cache_file in cache_dir.glob("*.pkl"):
        try:
            with open(cache_file, 'rb') as f:
                cache_data = pickle.load(f)
            
            if not self._is_cache_valid(cache_data):
                cache_file.unlink()
                removed_count += 1
        except Exception as e:
            self.logger.warning(f"Error processing {cache_file}: {e}")
    
    self.logger.info(f"Cleared {removed_count} expired cache files")

```

## Complete Data Flow: From Save to Validation

Understanding how the adaptive cache manager handles expiration requires following the data lifecycle:

1. **Write Path**: When `save_stock_data` is called, the manager generates a cache key via `_get_cache_key`, calculates the appropriate TTL using `_get_ttl_seconds`, and persists to the primary backend. Redis receives the TTL via `setex`, MongoDB receives an `expires_at` timestamp, and files receive an embedded timestamp metadata.

2. **Read Path**: During `load_stock_data`, the manager attempts to read from the primary backend. If successful, it immediately validates the payload through `_is_cache_valid`, comparing the stored timestamp plus TTL against current time. Stale entries return `None`, triggering a cache miss and subsequent refetch from the data source.

3. **Maintenance Path**: The `clear_expired_cache` method runs periodically (typically via scheduled job or manual invocation) to purge stale entries from the file system, ensuring disk space remains available for fresh market data.

## Practical Code Examples

### Basic Usage: Save, Load, and Check Expiration

```python
from tradingagents.dataflows.cache_manager import get_cache

# Initialize (or retrieve the global instance)

cache = get_cache()

# Save AAPL stock data (file fallback will be used if Redis is unavailable)

cache_key = cache.save_stock_data(
    symbol="AAPL",
    data="price data …",
    start_date="2024-01-01",
    end_date="2024-01-31",
    data_source="yfinance",
)

# Load the same entry later

data = cache.load_stock_data(cache_key)
if data is None:
    print("Cache miss or data expired – refetch from source")
else:
    print("Cache hit:", data)

```

### Force Cleanup of Stale File Cache

```python
from tradingagents.dataflows.cache_manager import get_cache

cache = get_cache()
cache.clear_expired_cache()          # removes any *.pkl that surpassed its TTL

```

### Inspect Cache Statistics

```python
from tradingagents.dataflows.cache_manager import get_cache

cache = get_cache()
stats = cache.get_cache_stats()
print("Cache backend:", stats["backend_info"]["primary_backend"])
print("File cache entries:", stats["backend_info"]["file_cache_count"])
print("Redis keys (if enabled):", stats["backend_info"].get("redis_keys"))

```

### Query TTL for Specific Symbols

```python
ttl = cache._get_ttl_seconds("AAPL", data_type="stock_data")
print(f"TTL for US stock data: {ttl} seconds")

```

## Key Implementation Files

| File | Role |
|------|------|
| [[`scripts/development/adaptive_cache_manager.py`](https://github.com/hsliuping/TradingAgents-CN/blob/main/scripts/development/adaptive_cache_manager.py)](https://github.com/hsliuping/TradingAgents-CN/blob/main/scripts/development/adaptive_cache_manager.py) | High-level wrapper used by tests and scripts; implements the same TTL, validation and fallback logic as the core cache system. |
| [[`tradingagents/dataflows/cache/adaptive.py`](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/adaptive.py)](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/adaptive.py) | Core implementation of the **AdaptiveCacheSystem** – defines `_is_cache_valid`, `_get_ttl_seconds`, backend-specific save/load, and `clear_expired_cache`. |
| [[`tradingagents/dataflows/cache/integrated.py`](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/integrated.py)](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/integrated.py) | Provides a thin façade (`get_cache_system`) that returns a singleton `AdaptiveCacheSystem` for the rest of the codebase. |
| [[`tests/test_cache_optimization.py`](https://github.com/hsliuping/TradingAgents-CN/blob/main/tests/test_cache_optimization.py)](https://github.com/hsliuping/TradingAgents-CN/blob/main/tests/test_cache_optimization.py) | Unit-tests that exercise saving, loading, TTL checks and clean-up, demonstrating the intended behaviour. |

## Summary

- **TTL-based validation** ensures every cache entry is checked for freshness on read via `_is_cache_valid`, comparing stored timestamps against calculated time-to-live values.
- **Backend-native expiration** leverages Redis `setex` and MongoDB TTL indexes to automatically purge stale data without manual intervention.
- **File cache cleanup** requires explicit maintenance through `clear_expired_cache`, which scans the `data/cache` directory and removes expired pickle files.
- **Multi-layer fallback** ensures that even if primary backends fail, the system validates file cache entries before returning potentially stale data to trading pipelines.

## Frequently Asked Questions

### How does the adaptive cache manager determine if data is expired?

The manager uses the `_is_cache_valid` method in [`tradingagents/dataflows/cache/adaptive.py`](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/adaptive.py) to compare the cached entry's stored timestamp plus its TTL against the current time. If the current time exceeds the calculated expiration, the data is considered stale and the method returns `None`, triggering a cache miss.

### What happens when Redis or MongoDB data expires?

Redis automatically removes keys when their TTL elapses because the manager uses the `setex` command during writes. MongoDB relies on a TTL index on the `expires_at` field, causing the database to automatically delete documents when their expiration time passes. Neither requires explicit cleanup code from the application.

### How often should I run clear_expired_cache for file-based caching?

You should invoke `clear_expired_cache` periodically based on your data volume and disk constraints. For high-frequency trading environments, running it daily or via a scheduled cron job prevents the `data/cache` directory from accumulating obsolete pickle files. The method is safe to run frequently as it only removes files that fail the `_is_cache_valid` check.

### Can I customize TTL values for different symbols or data types?

Yes. The `_get_ttl_seconds` method calculates TTL based on the symbol and data type parameters. You can extend this logic to assign longer TTLs for stable historical data or shorter TTLs for volatile intraday prices. The TTL is embedded in the cache metadata and enforced during both read validation and backend-native expiration.