Heterogeneous Storage Strategy in TencentDB Agent Memory: COS, SQLite, and Graceful Fallback

TencentDB Agent Memory implements a heterogeneous storage strategy that automatically cascades through COS, SQLite, filesystem, and in-memory backends to ensure the system never crashes while maintaining strict multi-node safety for production deployments.

The heterogeneous storage strategy in TencentDB Agent Memory (hosted in the TencentCloud/TencentDB-Agent-Memory repository) enables the MemoryProxy service to operate across diverse environments—from production multi-node clusters to lightweight local development—without code changes. This architecture, implemented primarily in MemoryProxy/src/storage/factory.ts, provides a configurable fallback chain that degrades gracefully when preferred backends fail, while enforcing strict failure modes for shared storage requirements.

How the Storage Factory Implements Backend Selection

The core of the heterogeneous approach resides in the storage factory, which dynamically instantiates storage backends based on runtime configuration and availability. The factory defines a preference-ordered chain via the orderFrom() function, creating a sequence where each backend serves as a fallback for the previous one.

When initialization begins, the factory attempts to instantiate the requested backend. If instantiation fails due to missing credentials, network issues, or submodule load failures, the system logs a !!! DEGRADED !!! warning and proceeds to the next backend in the chain. This ensures the agent remains operational even when infrastructure dependencies are unavailable.

// MemoryProxy/src/storage/factory.ts – builds the fallback chain
function orderFrom(preferred: ProxyStorageType): ProxyStorageType[] {
  const all: ProxyStorageType[] = ["cos", "sqlite", "fs", "memory"];
  const start = all.indexOf(preferred);
  if (start < 0) return all;
  return all.slice(start);
}

The function returns an ordered array starting from the preferred backend through to the final memory fallback, establishing the sequence: COS → SQLite → Filesystem → Memory.

Available Storage Backends and Their Characteristics

The heterogeneous strategy supports four distinct backends, each optimized for specific deployment scenarios regarding persistence, performance, and multi-node synchronization capabilities.

COS (Cloud Object Storage)

COS serves as the primary production backend for multi-node deployments. It provides durable, remote object storage via Tencent Cloud Object Storage, enabling all nodes in a cluster to read and write shared state. The COS backend requires successful initialization—if COS credentials are invalid or the network is unreachable, the process aborts immediately rather than degrading, ensuring no divergent state risks occur in distributed environments. The implementation dynamically imports the cost-guard sub-module only when COS is selected, supplying the kernel-STS client for authentication.

SQLite

SQLite offers a lightweight, file-based relational database suitable for single-node deployments or local development. It persists data durably on the host filesystem but maintains per-node copies, making it unsuitable for multi-node clusters. When selected, the factory automatically starts a TTL sweeper that cleans up objects prefixed with ttl/ to prevent unbounded growth.

FS (Filesystem)

FS provides simple directory-based key-value storage using raw files. Like SQLite, it offers host-durable persistence but lacks multi-node sharing capabilities. The filesystem backend serves as an intermediate fallback when SQLite is unavailable or misconfigured.

Memory

Memory acts as the final safety net—a non-persistent, in-process volatile map that loses all data on restart. This backend guarantees the agent process never crashes due to storage unavailability, though it offers no durability guarantees.

Backend	Persistence	Multi-Node Sharing	Use Case
COS	Durable (remote)	Yes (shared objects)	Production clusters
SQLite	Durable (local)	No (per-node)	Development, single-node
FS	Durable (local)	No (per-node)	Lightweight file storage
Memory	Non-persistent	No (volatile)	Emergency fallback, prototyping

Configuring the Fallback Chain

The heterogeneous storage strategy is configurable via the StorageConfig interface defined at the top of factory.ts. Developers specify their preferred backend through environment variables or direct configuration objects, with the factory handling the instantiation logic and fallback transitions automatically.

Production Configuration with COS

For production deployments requiring multi-node consistency, configure COS as the primary backend. The factory will abort startup if COS initialization fails, preventing unsafe operation without shared storage.

// .env configuration
PROXY_STORAGE_BACKEND=cos
PROXY_STORAGE_TTL_DAYS=30
PROXY_STORAGE_COS_ROOTPREFIX=team1/
PROXY_STORAGE_COS_SHARK_BASEURL=https://shark.example.com

// Application bootstrap
import { initProxyStorage } from "./src/storage/factory.js";
import { loadConfig } from "./config.js";

async function start() {
  const cfg = loadConfig(); // Maps env vars to StorageConfig
  const storage = await initProxyStorage(cfg);
  console.log("Storage ready:", storage.type); // "cos"
}
start().catch(console.error);

Local Development with SQLite Fallback

For local testing or offline development, SQLite provides durable storage without cloud dependencies. The factory automatically instantiates the SQLite backend and starts the TTL sweeper.

// .env configuration
PROXY_STORAGE_BACKEND=sqlite
PROXY_STORAGE_TTL_DAYS=7
PROXY_STORAGE_SQLITE_DBPATH=/tmp/proxy.db

import { getProxyStorage } from "./src/storage/factory.js";

const cfg = {
  backend: "sqlite",
  ttlDays: 7,
  sqlite: { dbPath: "/tmp/proxy.db" },
  cos: { rootPrefix: "", shark: { baseUrl: "" } },
  fs: { fsRoot: "" },
};
const storage = getProxyStorage(cfg);
console.log(`Effective backend: ${storage.type}`); // "sqlite"

Emergency In-Memory Operation

For quick prototypes or crash-guaranteed scenarios, force the memory backend to eliminate all external dependencies.

const cfg = {
  backend: "memory",
  ttlDays: 0,
  cos: { rootPrefix: "", shark: { baseUrl: "" } },
  sqlite: { dbPath: "" },
  fs: { fsRoot: "" }
};
const storage = getProxyStorage(cfg);
console.log(storage.type); // "memory"

Multi-Node Safety and Production Guarantees

The heterogeneous storage strategy enforces strict multi-node safety by treating COS differently from other backends. Since COS is the only backend that provides correct shared state across distributed nodes, the factory disables graceful degradation for COS initialization failures. If COS is requested but cannot be instantiated (missing credentials, network failure, or STS submodule error), the process exits immediately rather than falling back to SQLite or Memory, which would cause state divergence across the cluster.

For SQLite backends, the factory implements automatic TTL management, spawning a background sweeper that removes expired entries based on the ttlDays configuration parameter. This prevents local database bloat during extended operation periods.

Summary

• Heterogeneous storage strategy in TencentDB Agent Memory supports four backends: COS, SQLite, Filesystem, and Memory, configured via StorageConfig in MemoryProxy/src/storage/factory.ts.
• Graceful degradation follows the chain COS → SQLite → FS → Memory, logging !!! DEGRADED !!! when falling back to lesser backends.
• Multi-node safety requires COS for production clusters; COS initialization failures abort the process rather than degrading to prevent split-brain scenarios.
• SQLite includes TTL sweeper support for automatic cleanup of temporary data, activated when the sqlite backend is instantiated.
• Extensible architecture allows new backends by implementing the ProxyStorage interface and extending the orderFrom() function.

Frequently Asked Questions

What happens if COS initialization fails in a production deployment?

The process aborts immediately without fallback. According to the source code in MemoryProxy/src/storage/factory.ts, COS degradation is explicitly disabled because it is the only backend that safely supports multi-node shared state. A COS failure triggers a fatal error rather than risking divergent state across cluster nodes.

How does the TTL sweeper work in the SQLite backend?

When the factory instantiates the SQLite backend, it automatically initializes a background TTL sweeper that periodically scans for and deletes objects with the ttl/ prefix. This prevents the local database from growing unbounded as temporary data accumulates, using the ttlDays parameter from StorageConfig to determine expiration windows.

Can I use the Memory backend for production workloads?

No. The Memory backend is strictly for development, prototyping, or emergency fallback scenarios. It stores data in a volatile in-process map that is destroyed when the process restarts, offering zero durability guarantees and no multi-node sharing capabilities.

How do I add a new storage backend to the heterogeneous strategy?

Implement the ProxyStorage interface defined in MemoryProxy/src/storage/proxy-storage.ts, then extend the orderFrom() function in factory.ts to include your new backend type in the fallback chain. Update the ProxyStorageType union type and ensure your implementation handles the required CRUD operations and TTL management if applicable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}