How the Adaptive Cache Manager Handles Expired and Stale Data in TradingAgents-CN

The adaptive cache manager in TradingAgents-CN prevents stale data from entering trading pipelines through TTL-based validation, backend-native expiration mechanisms, and periodic file cache cleanup.

The adaptive cache manager is a critical component in the TradingAgents-CN repository that intelligently routes market data across multiple storage backends while ensuring expired entries never contaminate trading decisions. By combining timestamp validation with backend-specific expiration features, the system maintains data freshness across file system, Redis, and MongoDB deployments.

TTL-Based Validation for All Backends

Every cache entry carries a datetime timestamp recording when it was written. The manager calculates a time-to-live (TTL) in seconds based on the symbol's market and data type using _get_ttl_seconds. When reading an entry, the _is_cache_valid method compares the stored timestamp plus TTL against the current time.

If the computed expiry time has passed, the entry is treated as stale and ignored. This validation runs uniformly across all three backends—file, Redis, and MongoDB—ensuring consistent expiration semantics regardless of storage medium.

# From tradingagents/dataflows/cache/adaptive.py (lines 64-70)

def _is_cache_valid(self, cache_data: Dict) -> bool:
    """Check if cached data is still valid based on TTL."""
    timestamp = cache_data.get('timestamp')
    if not timestamp:
        return False
    
    ttl = cache_data.get('ttl', self.default_ttl)
    return datetime.now() - timestamp < timedelta(seconds=ttl)

Backend-Native Expiration Mechanisms

While TTL validation catches expired entries during reads, the adaptive cache manager also leverages native expiration features in Redis and MongoDB to proactively remove stale data without manual intervention.

Redis Automatic Key Expiration

When persisting to Redis, the manager uses the setex command in _save_to_redis, which atomically sets the key value and its TTL in seconds. Redis automatically deletes the key once the TTL elapses, preventing memory bloat from obsolete market data.

# From tradingagents/dataflows/cache/adaptive.py (lines 124-127)

def _save_to_redis(self, key: str, data: Dict, ttl: int):
    """Save data to Redis with automatic expiration."""
    serialized = json.dumps(data, default=str)
    self.redis_client.setex(key, ttl, serialized)

MongoDB TTL Indexes

For MongoDB storage, the _save_to_mongodb method injects an expires_at field calculated as datetime.now() + timedelta(seconds=ttl). The MongoDB collection maintains a TTL index on this field, causing the database to automatically purge documents when their expires_at time passes.

# From tradingagents/dataflows/cache/adaptive.py (lines 182-184)

def _save_to_mongodb(self, key: str, data: Dict, ttl: int):
    """Save data to MongoDB with expiration timestamp."""
    expires_at = datetime.now() + timedelta(seconds=ttl)
    document = {
        'key': key,
        'data': data,
        'expires_at': expires_at,
        'created_at': datetime.now()
    }
    self.mongo_collection.insert_one(document)

Periodic Cleanup of File Cache

Unlike Redis and MongoDB, the file system backend lacks native expiration capabilities. The adaptive cache manager implements clear_expired_cache to handle stale file entries explicitly.

This method walks through the data/cache directory, deserializes each pickled file, validates freshness using _is_cache_valid, and deletes any file that has surpassed its TTL. This cleanup prevents the file cache from accumulating obsolete market data during long-running deployments.

# From tradingagents/dataflows/cache/adaptive.py (lines 404-410)

def clear_expired_cache(self):
    """Remove expired entries from file cache."""
    cache_dir = Path("data/cache")
    if not cache_dir.exists():
        return
    
    removed_count = 0
    for cache_file in cache_dir.glob("*.pkl"):
        try:
            with open(cache_file, 'rb') as f:
                cache_data = pickle.load(f)
            
            if not self._is_cache_valid(cache_data):
                cache_file.unlink()
                removed_count += 1
        except Exception as e:
            self.logger.warning(f"Error processing {cache_file}: {e}")
    
    self.logger.info(f"Cleared {removed_count} expired cache files")

Complete Data Flow: From Save to Validation

Understanding how the adaptive cache manager handles expiration requires following the data lifecycle:

1. Write Path: When save_stock_data is called, the manager generates a cache key via _get_cache_key, calculates the appropriate TTL using _get_ttl_seconds, and persists to the primary backend. Redis receives the TTL via setex, MongoDB receives an expires_at timestamp, and files receive an embedded timestamp metadata.

2. Read Path: During load_stock_data, the manager attempts to read from the primary backend. If successful, it immediately validates the payload through _is_cache_valid, comparing the stored timestamp plus TTL against current time. Stale entries return None, triggering a cache miss and subsequent refetch from the data source.

3. Maintenance Path: The clear_expired_cache method runs periodically (typically via scheduled job or manual invocation) to purge stale entries from the file system, ensuring disk space remains available for fresh market data.

Practical Code Examples

Basic Usage: Save, Load, and Check Expiration

from tradingagents.dataflows.cache_manager import get_cache

# Initialize (or retrieve the global instance)

cache = get_cache()

# Save AAPL stock data (file fallback will be used if Redis is unavailable)

cache_key = cache.save_stock_data(
    symbol="AAPL",
    data="price data …",
    start_date="2024-01-01",
    end_date="2024-01-31",
    data_source="yfinance",
)

# Load the same entry later

data = cache.load_stock_data(cache_key)
if data is None:
    print("Cache miss or data expired – refetch from source")
else:
    print("Cache hit:", data)

Force Cleanup of Stale File Cache

from tradingagents.dataflows.cache_manager import get_cache

cache = get_cache()
cache.clear_expired_cache()          # removes any *.pkl that surpassed its TTL

Inspect Cache Statistics

from tradingagents.dataflows.cache_manager import get_cache

cache = get_cache()
stats = cache.get_cache_stats()
print("Cache backend:", stats["backend_info"]["primary_backend"])
print("File cache entries:", stats["backend_info"]["file_cache_count"])
print("Redis keys (if enabled):", stats["backend_info"].get("redis_keys"))

Query TTL for Specific Symbols

ttl = cache._get_ttl_seconds("AAPL", data_type="stock_data")
print(f"TTL for US stock data: {ttl} seconds")

Key Implementation Files

File	Role
[scripts/development/adaptive_cache_manager.py](https://github.com/hsliuping/TradingAgents-CN/blob/main/scripts/development/adaptive_cache_manager.py)	High-level wrapper used by tests and scripts; implements the same TTL, validation and fallback logic as the core cache system.
[tradingagents/dataflows/cache/adaptive.py](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/adaptive.py)	Core implementation of the AdaptiveCacheSystem – defines _is_cache_valid, _get_ttl_seconds, backend-specific save/load, and clear_expired_cache.
[tradingagents/dataflows/cache/integrated.py](https://github.com/hsliuping/TradingAgents-CN/blob/main/tradingagents/dataflows/cache/integrated.py)	Provides a thin façade (get_cache_system) that returns a singleton AdaptiveCacheSystem for the rest of the codebase.
[tests/test_cache_optimization.py](https://github.com/hsliuping/TradingAgents-CN/blob/main/tests/test_cache_optimization.py)	Unit-tests that exercise saving, loading, TTL checks and clean-up, demonstrating the intended behaviour.

Summary

• TTL-based validation ensures every cache entry is checked for freshness on read via _is_cache_valid, comparing stored timestamps against calculated time-to-live values.
• Backend-native expiration leverages Redis setex and MongoDB TTL indexes to automatically purge stale data without manual intervention.
• File cache cleanup requires explicit maintenance through clear_expired_cache, which scans the data/cache directory and removes expired pickle files.
• Multi-layer fallback ensures that even if primary backends fail, the system validates file cache entries before returning potentially stale data to trading pipelines.

Frequently Asked Questions

How does the adaptive cache manager determine if data is expired?

The manager uses the _is_cache_valid method in tradingagents/dataflows/cache/adaptive.py to compare the cached entry's stored timestamp plus its TTL against the current time. If the current time exceeds the calculated expiration, the data is considered stale and the method returns None, triggering a cache miss.

What happens when Redis or MongoDB data expires?

Redis automatically removes keys when their TTL elapses because the manager uses the setex command during writes. MongoDB relies on a TTL index on the expires_at field, causing the database to automatically delete documents when their expiration time passes. Neither requires explicit cleanup code from the application.

How often should I run clear_expired_cache for file-based caching?

You should invoke clear_expired_cache periodically based on your data volume and disk constraints. For high-frequency trading environments, running it daily or via a scheduled cron job prevents the data/cache directory from accumulating obsolete pickle files. The method is safe to run frequently as it only removes files that fail the _is_cache_valid check.

Can I customize TTL values for different symbols or data types?

Yes. The _get_ttl_seconds method calculates TTL based on the symbol and data type parameters. You can extend this logic to assign longer TTLs for stable historical data or shorter TTLs for volatile intraday prices. The TTL is embedded in the cache metadata and enforced during both read validation and backend-native expiration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}