Understanding the Heterogeneous Storage Strategy in TencentDB Agent Memory

TencentDB Agent Memory implements a heterogeneous storage strategy that automatically cascades through COS, SQLite, filesystem, and in-memory backends to ensure the system remains operational even when preferred storage layers fail.

The TencentDB-Agent-Memory repository provides a flexible storage abstraction designed to support both production multi-node deployments and lightweight local development. At its core, the heterogeneous storage strategy defined in MemoryProxy/src/storage/factory.ts enables dynamic backend selection with automatic degradation, ensuring data persistence across varying infrastructure constraints.

Architecture of the Heterogeneous Storage Strategy

The system supports four distinct storage backends arranged in a priority chain. Each backend offers different trade-offs between durability, multi-node consistency, and operational complexity.

• COS (Tencent Cloud Object Storage): The primary production backend providing durable, remote object storage with full multi-node sharing capabilities. This is the only backend safe for multi-node deployments where all agents must read and write identical state.
• SQLite: A lightweight file-based database offering local durability without network dependencies. Each node maintains an isolated copy, making this suitable for single-node development or testing.
• FS (Filesystem): A simple directory-based key-value store that persists raw files to the host filesystem. Like SQLite, this is node-local and non-shared.
• Memory: An in-process volatile map providing no persistence but guaranteed availability. This serves as the final safety net when all persistent options fail.

The Fallback Chain Logic

The factory implements a deterministic fallback mechanism through the orderFrom() function. When a preferred backend fails initialization, the system automatically attempts the next option in the sequence.

// MemoryProxy/src/storage/factory.ts
function orderFrom(preferred: ProxyStorageType): ProxyStorageType[] {
  const all: ProxyStorageType[] = ["cos", "sqlite", "fs", "memory"];
  const start = all.indexOf(preferred);
  if (start < 0) return all;
  return all.slice(start);
}

When instantiation fails for any backend except COS, the factory emits an error-level log containing !!! DEGRADED !!! and proceeds to the next backend in the chain. COS degradation is explicitly disabled—if COS initialization fails, the entire process aborts because COS represents the only safe shared storage for multi-node clusters.

Storage Backend Reference

COS (Tencent Cloud Object Storage)

COS serves as the production-grade backend for distributed deployments. The factory dynamically imports the cost-guard sub-module to supply the COS kernel-STS client only when this backend is selected. Configuration requires valid Tencent Cloud credentials and endpoint definitions.

Characteristics:

• Persistence: Durable remote storage
• Multi-node sharing: Supported (all nodes access identical objects)
• Degradation: Disabled—initialization failures abort startup

SQLite

SQLite provides a serverless, file-based relational database suitable for development environments or edge deployments without cloud connectivity. The factory automatically starts a TTL sweeper for SQLite backends that cleans up ttl/ prefixed objects according to configured retention policies.

Characteristics:

• Persistence: Durable on host filesystem
• Multi-node sharing: Not supported (each node has isolated storage)
• Degradation: Enabled—falls back to FS or Memory on failure

FS (Filesystem)

The filesystem backend offers a minimal storage layer that writes raw files to a designated directory structure. This option requires no external dependencies beyond write permissions to the host filesystem.

Characteristics:

• Persistence: Durable on host
• Multi-node sharing: Not supported
• Degradation: Enabled—falls back to Memory if directory operations fail

Memory

The memory backend implements a simple in-process Map structure. While this guarantees the process never crashes due to storage unavailability, all data is lost on process restart.

Characteristics:

• Persistence: Non-persistent (volatile)
• Multi-node sharing: Not supported
• Degradation: Final fallback—always available

Configuration and Usage Examples

Production COS Configuration

For multi-node production clusters, configure COS as the primary backend. The system expects environment variables or configuration objects matching the StorageConfig interface:

// .env configuration
PROXY_STORAGE_BACKEND=cos
PROXY_STORAGE_TTL_DAYS=30
PROXY_STORAGE_COS_ROOTPREFIX=team1/
PROXY_STORAGE_COS_SHARK_BASEURL=https://shark.example.com

// Application initialization
import { initProxyStorage } from "./src/storage/factory.js";
import { loadConfig } from "./config.js";

async function start() {
  const cfg = loadConfig();
  const storage = await initProxyStorage(cfg);
  console.log("Storage ready:", storage.type); // "cos"
}
start().catch(console.error);

Local Development with SQLite

For single-node development or when COS credentials are unavailable:

// .env configuration
PROXY_STORAGE_BACKEND=sqlite
PROXY_STORAGE_TTL_DAYS=7
PROXY_STORAGE_SQLITE_DBPATH=/tmp/proxy.db

// Programmatic configuration
import { getProxyStorage } from "./src/storage/factory.js";

const cfg = {
  backend: "sqlite",
  ttlDays: 7,
  sqlite: { dbPath: "/tmp/proxy.db" },
  cos: { rootPrefix: "", shark: { baseUrl: "" } },
  fs: { fsRoot: "" },
};
const storage = getProxyStorage(cfg);
console.log(`Effective backend: ${storage.type}`); // "sqlite"

Prototyping with In-Memory Storage

For rapid testing without persistence requirements:

const cfg = { 
  backend: "memory", 
  ttlDays: 0, 
  cos: { rootPrefix: "", shark: { baseUrl: "" } }, 
  sqlite: { dbPath: "" }, 
  fs: { fsRoot: "" } 
};
const storage = getProxyStorage(cfg);
console.log(storage.type); // "memory"

Core Implementation Files

The heterogeneous storage strategy is implemented across the following source files in the TencentDB-Agent-Memory repository:

• MemoryProxy/src/storage/factory.ts: Implements the storage factory, fallback chain logic, and backend initialization sequence.
• MemoryProxy/src/storage/cos-storage.ts: COS backend implementation handling object-store operations and STS authentication.
• MemoryProxy/src/storage/sqlite-storage.ts: SQLite wrapper including database operations and automatic TTL sweeper integration.
• MemoryProxy/src/storage/fs-storage.ts: Filesystem-based storage for simple directory-based persistence.
• MemoryProxy/src/storage/memory-storage.ts: In-memory map implementation serving as the final degradation target.
• MemoryProxy/src/storage/proxy-storage.ts: TypeScript interface definitions shared across all storage implementations.

Summary

• The heterogeneous storage strategy in TencentDB Agent Memory supports four backends: COS, SQLite, FS, and Memory, arranged in a priority chain from most to least durable.
• Automatic degradation allows the system to fallback from SQLite → FS → Memory when preferred options fail, logging !!! DEGRADED !!! warnings for operational visibility.
• COS is strictly required for multi-node deployments because it is the only backend supporting shared state across nodes; COS initialization failures abort startup rather than degrade.
• The orderFrom() function in factory.ts determines the fallback sequence dynamically based on the requested StorageConfig.backend value.
• SQLite backends automatically run a TTL sweeper to expire ttl/ prefixed objects, while Memory backends provide zero-persistence guarantees for emergency operation.

Frequently Asked Questions

What happens if COS initialization fails in a production deployment?

If COS initialization fails, the process aborts immediately without falling back to local storage. According to the source code in MemoryProxy/src/storage/factory.ts, degradation is disabled for COS because it is the only backend that maintains consistency across multiple nodes. Using SQLite or FS in a multi-node environment would result in divergent state across agents, so the system prefers to crash rather than risk data inconsistency.

Can I use SQLite for production multi-node deployments?

No. While SQLite provides durable storage, it operates as a local file-based database where each node maintains an isolated copy. The heterogeneous storage strategy explicitly marks SQLite as unsuitable for multi-node sharing. For production clusters requiring shared state, you must use COS (Tencent Cloud Object Storage) to ensure all nodes read and write the same objects.

How do I configure the storage backend for different environments?

Set the PROXY_STORAGE_BACKEND environment variable to cos, sqlite, fs, or memory, or pass the corresponding value in the StorageConfig object when calling initProxyStorage() or getProxyStorage(). The factory consumes this configuration to determine the starting point of the fallback chain via the orderFrom() function. Additional backend-specific settings (such as PROXY_STORAGE_COS_ROOTPREFIX or PROXY_STORAGE_SQLITE_DBPATH) customize the selected backend's behavior.

What is the TTL sweeper and when does it operate?

The TTL sweeper is a background maintenance process that automatically removes expired objects prefixed with ttl/ from the storage backend. According to the implementation in MemoryProxy/src/storage/sqlite-storage.ts, this sweeper initializes automatically when using SQLite backends. It ensures that temporary data with defined lifetimes does not consume disk space indefinitely, using the ttlDays value from the configuration to determine retention periods.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}