# How to Implement Session-Based Memory Auto-Compression in OpenViking

> Learn to implement session-based memory auto-compression in OpenViking. Store and decompress query results efficiently using SessionManager for automatic cleanup. Read the guide now.

- Repository: [Volcengine/OpenViking](https://github.com/volcengine/OpenViking)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Store query results in a compressed form inside a `Session` and decompress on read, leveraging the existing `SessionManager` for automatic cleanup without modifying the idle-session logic.**

OpenViking's SQL-FS2 plugin implements a plan-9-style session model where large query results can consume significant heap memory until sessions expire. Implementing session-based memory auto-compression in OpenViking allows you to transparently compress JSON result payloads while preserving the existing virtual filesystem API and automatic session lifecycle management.

## Architectural Overview

OpenViking's SQL-FS2 plugin manages per-query state through a centralized session registry.

| Component | Role | Relevant Source |
|-----------|------|-----------------|
| `Session` | Holds per-query state including transaction, result payload, error string, and last access timestamp. | [`third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go) (lines 23-33) |
| `SessionManager` | Central registry assigning numeric session IDs and managing idle cleanup via background goroutine. | [`third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go) (lines 48-69) |
| `cleanupLoop` / `cleanupExpired` | Periodically removes sessions untouched for `session_timeout` duration. | [`third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go) (lines 71-84) |
| `Read` / `Write` | Virtual filesystem operations exposing sessions through paths like `…/query` and `…/result`. | [`third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go) (lines 563-620 for read, lines 428-511 for write) |

### Memory Growth Patterns

After executing a **SELECT** query, the plugin marshals rows into a JSON `[]byte` slice stored in `session.result`. For large tables, this slice can reach tens of megabytes and persists in the Go heap until the session expires or closes manually. Because `SessionManager` already handles automatic cleanup, the optimization target is compressing the `result` payload while it resides in memory.

## Design of Auto-Compression

1. **Compress on write** – When the `query` file receives a `SELECT` statement, marshal rows to JSON and compress the payload using `gzip`. Store compressed bytes in a new field `compressedResult []byte`.
2. **Decompress on read** – The `result` file handler checks for `compressedResult`, inflates the data on-the-fly, and returns plain JSON to the virtual filesystem client.
3. **Transparent fallback** – For non-`SELECT` statements or compression failures, retain the original uncompressed `result` field to maintain compatibility.
4. **Thread-safety** – The existing `session.mu` mutex already guards result field access; extend this protection to cover `compressedResult`.

This approach localizes changes entirely within [`sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/sqlfs2.go), leaving other plugins, filesystem layers, and backend code untouched.

## Step-by-Step Implementation

Add the following modifications to [`third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go):

### 1. Add Required Imports

```go
import (
    "bytes"
    "compress/gzip"
    "io"
    // ... existing imports
)

```

### 2. Extend the Session Struct

```go
type Session struct {
    id         int64
    dbName     string
    tableName  string
    tx         *sql.Tx
    result     []byte // retained for backward compatibility
    // NEW: compressed JSON result storage
    compressedResult []byte
    lastError  string
    lastAccess time.Time
    mu         sync.Mutex
}

```

### 3. Implement Compression Helpers

```go
func compress(data []byte) ([]byte, error) {
    var buf bytes.Buffer
    gz := gzip.NewWriter(&buf)
    if _, err := gz.Write(data); err != nil {
        return nil, err
    }
    if err := gz.Close(); err != nil {
        return nil, err
    }
    return buf.Bytes(), nil
}

func decompress(data []byte) ([]byte, error) {
    gz, err := gzip.NewReader(bytes.NewReader(data))
    if err != nil {
        return nil, err
    }
    defer gz.Close()
    return io.ReadAll(gz)
}

```

### 4. Store Compressed Results

Locate the query handling block (around line 610) and replace the result assignment logic:

```go
// Marshal query results to JSON
jsonData, err := json.MarshalIndent(results, "", "  ")
if err != nil {
    session.lastError = err.Error()
    session.result = nil
    session.compressedResult = nil
    return 0, fmt.Errorf("json marshal error: %w", err)
}

// Attempt compression; fall back to uncompressed on failure
if comp, err := compress(jsonData); err == nil {
    session.compressedResult = comp
    session.result = nil // clear uncompressed to save memory
} else {
    session.result = append(jsonData, '\n')
    session.compressedResult = nil
}
session.lastError = ""

```

### 5. Decompress on Read

Modify the `result` case in the read handler (around line 587):

```go
case "result":
    session.mu.Lock()
    var data []byte
    if session.compressedResult != nil {
        // Decompress on-the-fly
        if d, err := decompress(session.compressedResult); err == nil {
            data = d
        } else {
            // Fallback to raw result if decompression fails
            data = session.result
        }
    } else {
        data = session.result
    }
    session.mu.Unlock()

    if data == nil {
        return []byte{}, nil
    }
    return plugin.ApplyRangeRead(data, offset, size)

```

## Usage Example

Mount the SQL-FS2 plugin and test the compression workflow:

```bash

# 1. Mount the filesystem

./agfs-fuse -mount /sqlfs2 -config '{"backend":"sqlite","db_path":"demo.db","session_timeout":"5m"}' &

# 2. Create a new session

echo "" > /sqlfs2/demo_db/demo_table/ctl
SESSION_ID=$(cat /sqlfs2/demo_db/demo_table/ctl)
echo "Created session: $SESSION_ID"

# 3. Execute a SELECT query (triggers compression)

echo "SELECT * FROM demo_table LIMIT 1000" > /sqlfs2/demo_db/demo_table/${SESSION_ID}/query

# 4. Read results (transparently decompressed)

cat /sqlfs2/demo_db/demo_table/${SESSION_ID}/result | jq .

# 5. Clean up session

echo "close" > /sqlfs2/demo_db/demo_table/${SESSION_ID}/ctl

```

During steps 3-4, the JSON payload resides in memory as gzip-compressed bytes, typically reducing memory usage by 50-70% for repetitive or structured data.

## Key Files

| File | Purpose |
|------|---------|
| [`third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go) | Core plugin containing `Session` struct, `SessionManager`, and read/write handlers. All compression logic resides here. |
| [`third_party/agfs/agfs-server/pkg/config/config.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/config/config.go) | Defines `session_timeout` parameter controlling automatic session cleanup. |
| [`third_party/agfs/agfs-server/pkg/plugin/api/plugin_api.go`](https://github.com/volcengine/OpenViking/blob/main/third_party/agfs/agfs-server/pkg/plugin/api/plugin_api.go) | Defines the plugin interface that exposes `Read` and `Write` operations to the FUSE layer. |

## Benefits and Trade-offs

| Benefit | Trade-off |
|---------|-----------|
| **Memory efficiency** – gzip compression typically reduces JSON payload size by 2-3x, lowering per-session heap usage. | **CPU overhead** – Each SELECT incurs compression cost on write and decompression on read. For most workloads this is negligible compared to database I/O, but high-throughput scenarios require benchmarking. |
| **Transparent operation** – Virtual filesystem clients continue to read plain JSON from the `result` file without protocol changes. | **Implementation complexity** – Requires adding `compressedResult` field and error handling paths, though changes are isolated to [`sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/sqlfs2.go). |
| **Compatible with auto-cleanup** – Works seamlessly with existing `session_timeout` based session eviction. | **Decompression latency** – First read of `result` pays the inflation cost; subsequent reads are fast because data remains compressed in memory. |

## Summary

- OpenViking's SQL-FS2 plugin stores query results in `Session` objects managed by `SessionManager`, which automatically cleans up idle sessions based on `session_timeout`.
- Adding session-based memory auto-compression in OpenViking requires extending the `Session` struct with a `compressedResult` field and implementing `compress()` and `decompress()` helpers using standard `compress/gzip`.
- Modify the write path (around line 610 in [`sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/sqlfs2.go)) to gzip JSON results after marshaling, and update the read path (around line 587) to transparently decompress when serving the `result` file.
- This approach reduces per-session memory usage by 50-70% without changing the virtual filesystem interface or affecting the existing auto-cleanup mechanism.

## Frequently Asked Questions

### How does session-based memory auto-compression affect the existing session timeout behavior?

The compression layer operates independently of the cleanup logic. The `SessionManager` continues to track `lastAccess` timestamps and removes idle sessions after the configured `session_timeout` expires. Compressed payloads are evicted along with their parent `Session` objects, so no additional cleanup code is required.

### What compression algorithm does the implementation use?

The reference implementation uses `compress/gzip` from the Go standard library with default compression levels. This provides a balance between CPU usage and compression ratio (typically 2-3x for JSON). You can substitute `flate`, `snappy`, or `zstd` by replacing the `compress()` and `decompress()` helper functions in [`sqlfs2.go`](https://github.com/volcengine/OpenViking/blob/main/sqlfs2.go).

### Will clients see any difference in the data returned from the result file?

No. The compression is transparent to virtual filesystem clients. When reading the `result` file, the plugin decompresses the payload on-the-fly before returning data via `plugin.ApplyRangeRead()`. Clients continue to receive plain JSON identical to the uncompressed implementation.

### How can I verify that compression is actually reducing memory usage?

You can verify memory savings by running a large SELECT query and inspecting the Go heap using `pprof`. After implementing the changes, the `Session` objects should show `compressedResult` fields significantly smaller than the original JSON payloads. Unit tests can also assert that `len(session.compressedResult) < len(jsonData)` for typical result sets.