# Heterogeneous Storage Strategy in TencentDB Agent Memory: COS, SQLite, and Graceful Fallback

> Explore the heterogeneous storage strategy in TencentDB Agent Memory. Discover how COS, SQLite, and graceful fallback ensure system stability and multi-node safety for your production deployments.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: deep-dive
- Published: 2026-08-24

---

**TencentDB Agent Memory implements a heterogeneous storage strategy that automatically cascades through COS, SQLite, filesystem, and in-memory backends to ensure the system never crashes while maintaining strict multi-node safety for production deployments.**

The **heterogeneous storage strategy** in TencentDB Agent Memory (hosted in the `TencentCloud/TencentDB-Agent-Memory` repository) enables the `MemoryProxy` service to operate across diverse environments—from production multi-node clusters to lightweight local development—without code changes. This architecture, implemented primarily in [`MemoryProxy/src/storage/factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/storage/factory.ts), provides a configurable fallback chain that degrades gracefully when preferred backends fail, while enforcing strict failure modes for shared storage requirements.

## How the Storage Factory Implements Backend Selection

The core of the heterogeneous approach resides in the **storage factory**, which dynamically instantiates storage backends based on runtime configuration and availability. The factory defines a preference-ordered chain via the `orderFrom()` function, creating a sequence where each backend serves as a fallback for the previous one.

When initialization begins, the factory attempts to instantiate the requested backend. If instantiation fails due to missing credentials, network issues, or submodule load failures, the system logs a **`!!! DEGRADED !!!`** warning and proceeds to the next backend in the chain. This ensures the agent remains operational even when infrastructure dependencies are unavailable.

```typescript
// MemoryProxy/src/storage/factory.ts – builds the fallback chain
function orderFrom(preferred: ProxyStorageType): ProxyStorageType[] {
  const all: ProxyStorageType[] = ["cos", "sqlite", "fs", "memory"];
  const start = all.indexOf(preferred);
  if (start < 0) return all;
  return all.slice(start);
}

```

The function returns an ordered array starting from the preferred backend through to the final `memory` fallback, establishing the sequence: **COS → SQLite → Filesystem → Memory**.

## Available Storage Backends and Their Characteristics

The heterogeneous strategy supports four distinct backends, each optimized for specific deployment scenarios regarding persistence, performance, and multi-node synchronization capabilities.

### COS (Cloud Object Storage)

**COS** serves as the primary production backend for multi-node deployments. It provides durable, remote object storage via Tencent Cloud Object Storage, enabling all nodes in a cluster to read and write shared state. The COS backend requires successful initialization—if COS credentials are invalid or the network is unreachable, the process **aborts immediately** rather than degrading, ensuring no divergent state risks occur in distributed environments. The implementation dynamically imports the `cost-guard` sub-module only when COS is selected, supplying the kernel-STS client for authentication.

### SQLite

**SQLite** offers a lightweight, file-based relational database suitable for single-node deployments or local development. It persists data durably on the host filesystem but maintains per-node copies, making it unsuitable for multi-node clusters. When selected, the factory automatically starts a **TTL sweeper** that cleans up objects prefixed with `ttl/` to prevent unbounded growth.

### FS (Filesystem)

**FS** provides simple directory-based key-value storage using raw files. Like SQLite, it offers host-durable persistence but lacks multi-node sharing capabilities. The filesystem backend serves as an intermediate fallback when SQLite is unavailable or misconfigured.

### Memory

**Memory** acts as the final safety net—a non-persistent, in-process volatile map that loses all data on restart. This backend guarantees the agent process never crashes due to storage unavailability, though it offers no durability guarantees.

| Backend | Persistence | Multi-Node Sharing | Use Case |
|---------|-------------|-------------------|----------|
| **COS** | Durable (remote) | Yes (shared objects) | Production clusters |
| **SQLite** | Durable (local) | No (per-node) | Development, single-node |
| **FS** | Durable (local) | No (per-node) | Lightweight file storage |
| **Memory** | Non-persistent | No (volatile) | Emergency fallback, prototyping |

## Configuring the Fallback Chain

The heterogeneous storage strategy is configurable via the **`StorageConfig`** interface defined at the top of [`factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/factory.ts). Developers specify their preferred backend through environment variables or direct configuration objects, with the factory handling the instantiation logic and fallback transitions automatically.

### Production Configuration with COS

For production deployments requiring multi-node consistency, configure COS as the primary backend. The factory will abort startup if COS initialization fails, preventing unsafe operation without shared storage.

```typescript
// .env configuration
PROXY_STORAGE_BACKEND=cos
PROXY_STORAGE_TTL_DAYS=30
PROXY_STORAGE_COS_ROOTPREFIX=team1/
PROXY_STORAGE_COS_SHARK_BASEURL=https://shark.example.com

```

```typescript
// Application bootstrap
import { initProxyStorage } from "./src/storage/factory.js";
import { loadConfig } from "./config.js";

async function start() {
  const cfg = loadConfig(); // Maps env vars to StorageConfig
  const storage = await initProxyStorage(cfg);
  console.log("Storage ready:", storage.type); // "cos"
}
start().catch(console.error);

```

### Local Development with SQLite Fallback

For local testing or offline development, SQLite provides durable storage without cloud dependencies. The factory automatically instantiates the SQLite backend and starts the TTL sweeper.

```typescript
// .env configuration
PROXY_STORAGE_BACKEND=sqlite
PROXY_STORAGE_TTL_DAYS=7
PROXY_STORAGE_SQLITE_DBPATH=/tmp/proxy.db

```

```typescript
import { getProxyStorage } from "./src/storage/factory.js";

const cfg = {
  backend: "sqlite",
  ttlDays: 7,
  sqlite: { dbPath: "/tmp/proxy.db" },
  cos: { rootPrefix: "", shark: { baseUrl: "" } },
  fs: { fsRoot: "" },
};
const storage = getProxyStorage(cfg);
console.log(`Effective backend: ${storage.type}`); // "sqlite"

```

### Emergency In-Memory Operation

For quick prototypes or crash-guaranteed scenarios, force the memory backend to eliminate all external dependencies.

```typescript
const cfg = {
  backend: "memory",
  ttlDays: 0,
  cos: { rootPrefix: "", shark: { baseUrl: "" } },
  sqlite: { dbPath: "" },
  fs: { fsRoot: "" }
};
const storage = getProxyStorage(cfg);
console.log(storage.type); // "memory"

```

## Multi-Node Safety and Production Guarantees

The heterogeneous storage strategy enforces **strict multi-node safety** by treating COS differently from other backends. Since COS is the only backend that provides correct shared state across distributed nodes, the factory disables graceful degradation for COS initialization failures. If COS is requested but cannot be instantiated (missing credentials, network failure, or STS submodule error), the process exits immediately rather than falling back to SQLite or Memory, which would cause state divergence across the cluster.

For SQLite backends, the factory implements **automatic TTL management**, spawning a background sweeper that removes expired entries based on the `ttlDays` configuration parameter. This prevents local database bloat during extended operation periods.

## Summary

- **Heterogeneous storage strategy** in TencentDB Agent Memory supports four backends: COS, SQLite, Filesystem, and Memory, configured via `StorageConfig` in [`MemoryProxy/src/storage/factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/storage/factory.ts).
- **Graceful degradation** follows the chain COS → SQLite → FS → Memory, logging `!!! DEGRADED !!!` when falling back to lesser backends.
- **Multi-node safety** requires COS for production clusters; COS initialization failures abort the process rather than degrading to prevent split-brain scenarios.
- **SQLite includes TTL sweeper** support for automatic cleanup of temporary data, activated when the sqlite backend is instantiated.
- **Extensible architecture** allows new backends by implementing the `ProxyStorage` interface and extending the `orderFrom()` function.

## Frequently Asked Questions

### What happens if COS initialization fails in a production deployment?

The process aborts immediately without fallback. According to the source code in [`MemoryProxy/src/storage/factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/storage/factory.ts), COS degradation is explicitly disabled because it is the only backend that safely supports multi-node shared state. A COS failure triggers a fatal error rather than risking divergent state across cluster nodes.

### How does the TTL sweeper work in the SQLite backend?

When the factory instantiates the SQLite backend, it automatically initializes a background TTL sweeper that periodically scans for and deletes objects with the `ttl/` prefix. This prevents the local database from growing unbounded as temporary data accumulates, using the `ttlDays` parameter from `StorageConfig` to determine expiration windows.

### Can I use the Memory backend for production workloads?

No. The Memory backend is strictly for development, prototyping, or emergency fallback scenarios. It stores data in a volatile in-process map that is destroyed when the process restarts, offering zero durability guarantees and no multi-node sharing capabilities.

### How do I add a new storage backend to the heterogeneous strategy?

Implement the `ProxyStorage` interface defined in [`MemoryProxy/src/storage/proxy-storage.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/storage/proxy-storage.ts), then extend the `orderFrom()` function in [`factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/factory.ts) to include your new backend type in the fallback chain. Update the `ProxyStorageType` union type and ensure your implementation handles the required CRUD operations and TTL management if applicable.