How to Implement Session-Based Memory Auto-Compression in OpenViking

Store query results in a compressed form inside a Session and decompress on read, leveraging the existing SessionManager for automatic cleanup without modifying the idle-session logic.

OpenViking's SQL-FS2 plugin implements a plan-9-style session model where large query results can consume significant heap memory until sessions expire. Implementing session-based memory auto-compression in OpenViking allows you to transparently compress JSON result payloads while preserving the existing virtual filesystem API and automatic session lifecycle management.

Architectural Overview

OpenViking's SQL-FS2 plugin manages per-query state through a centralized session registry.

Component	Role	Relevant Source
Session	Holds per-query state including transaction, result payload, error string, and last access timestamp.	third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go (lines 23-33)
SessionManager	Central registry assigning numeric session IDs and managing idle cleanup via background goroutine.	third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go (lines 48-69)
cleanupLoop / cleanupExpired	Periodically removes sessions untouched for session_timeout duration.	third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go (lines 71-84)
Read / Write	Virtual filesystem operations exposing sessions through paths like …/query and …/result.	third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go (lines 563-620 for read, lines 428-511 for write)

Memory Growth Patterns

After executing a SELECT query, the plugin marshals rows into a JSON []byte slice stored in session.result. For large tables, this slice can reach tens of megabytes and persists in the Go heap until the session expires or closes manually. Because SessionManager already handles automatic cleanup, the optimization target is compressing the result payload while it resides in memory.

Design of Auto-Compression

1. Compress on write – When the query file receives a SELECT statement, marshal rows to JSON and compress the payload using gzip. Store compressed bytes in a new field compressedResult []byte.
2. Decompress on read – The result file handler checks for compressedResult, inflates the data on-the-fly, and returns plain JSON to the virtual filesystem client.
3. Transparent fallback – For non-SELECT statements or compression failures, retain the original uncompressed result field to maintain compatibility.
4. Thread-safety – The existing session.mu mutex already guards result field access; extend this protection to cover compressedResult.

This approach localizes changes entirely within sqlfs2.go, leaving other plugins, filesystem layers, and backend code untouched.

Step-by-Step Implementation

Add the following modifications to third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go:

1. Add Required Imports

import (
    "bytes"
    "compress/gzip"
    "io"
    // ... existing imports
)

2. Extend the Session Struct

type Session struct {
    id         int64
    dbName     string
    tableName  string
    tx         *sql.Tx
    result     []byte // retained for backward compatibility
    // NEW: compressed JSON result storage
    compressedResult []byte
    lastError  string
    lastAccess time.Time
    mu         sync.Mutex
}

3. Implement Compression Helpers

func compress(data []byte) ([]byte, error) {
    var buf bytes.Buffer
    gz := gzip.NewWriter(&buf)
    if _, err := gz.Write(data); err != nil {
        return nil, err
    }
    if err := gz.Close(); err != nil {
        return nil, err
    }
    return buf.Bytes(), nil
}

func decompress(data []byte) ([]byte, error) {
    gz, err := gzip.NewReader(bytes.NewReader(data))
    if err != nil {
        return nil, err
    }
    defer gz.Close()
    return io.ReadAll(gz)
}

4. Store Compressed Results

Locate the query handling block (around line 610) and replace the result assignment logic:

// Marshal query results to JSON
jsonData, err := json.MarshalIndent(results, "", "  ")
if err != nil {
    session.lastError = err.Error()
    session.result = nil
    session.compressedResult = nil
    return 0, fmt.Errorf("json marshal error: %w", err)
}

// Attempt compression; fall back to uncompressed on failure
if comp, err := compress(jsonData); err == nil {
    session.compressedResult = comp
    session.result = nil // clear uncompressed to save memory
} else {
    session.result = append(jsonData, '\n')
    session.compressedResult = nil
}
session.lastError = ""

5. Decompress on Read

Modify the result case in the read handler (around line 587):

case "result":
    session.mu.Lock()
    var data []byte
    if session.compressedResult != nil {
        // Decompress on-the-fly
        if d, err := decompress(session.compressedResult); err == nil {
            data = d
        } else {
            // Fallback to raw result if decompression fails
            data = session.result
        }
    } else {
        data = session.result
    }
    session.mu.Unlock()

    if data == nil {
        return []byte{}, nil
    }
    return plugin.ApplyRangeRead(data, offset, size)

Usage Example

Mount the SQL-FS2 plugin and test the compression workflow:

# 1. Mount the filesystem

./agfs-fuse -mount /sqlfs2 -config '{"backend":"sqlite","db_path":"demo.db","session_timeout":"5m"}' &

# 2. Create a new session

echo "" > /sqlfs2/demo_db/demo_table/ctl
SESSION_ID=$(cat /sqlfs2/demo_db/demo_table/ctl)
echo "Created session: $SESSION_ID"

# 3. Execute a SELECT query (triggers compression)

echo "SELECT * FROM demo_table LIMIT 1000" > /sqlfs2/demo_db/demo_table/${SESSION_ID}/query

# 4. Read results (transparently decompressed)

cat /sqlfs2/demo_db/demo_table/${SESSION_ID}/result | jq .

# 5. Clean up session

echo "close" > /sqlfs2/demo_db/demo_table/${SESSION_ID}/ctl

During steps 3-4, the JSON payload resides in memory as gzip-compressed bytes, typically reducing memory usage by 50-70% for repetitive or structured data.

Key Files

File	Purpose
third_party/agfs/agfs-server/pkg/plugins/sqlfs2/sqlfs2.go	Core plugin containing Session struct, SessionManager, and read/write handlers. All compression logic resides here.
third_party/agfs/agfs-server/pkg/config/config.go	Defines session_timeout parameter controlling automatic session cleanup.
third_party/agfs/agfs-server/pkg/plugin/api/plugin_api.go	Defines the plugin interface that exposes Read and Write operations to the FUSE layer.

Benefits and Trade-offs

Benefit	Trade-off
Memory efficiency – gzip compression typically reduces JSON payload size by 2-3x, lowering per-session heap usage.	CPU overhead – Each SELECT incurs compression cost on write and decompression on read. For most workloads this is negligible compared to database I/O, but high-throughput scenarios require benchmarking.
Transparent operation – Virtual filesystem clients continue to read plain JSON from the result file without protocol changes.	Implementation complexity – Requires adding compressedResult field and error handling paths, though changes are isolated to sqlfs2.go.
Compatible with auto-cleanup – Works seamlessly with existing session_timeout based session eviction.	Decompression latency – First read of result pays the inflation cost; subsequent reads are fast because data remains compressed in memory.

Summary

• OpenViking's SQL-FS2 plugin stores query results in Session objects managed by SessionManager, which automatically cleans up idle sessions based on session_timeout.
• Adding session-based memory auto-compression in OpenViking requires extending the Session struct with a compressedResult field and implementing compress() and decompress() helpers using standard compress/gzip.
• Modify the write path (around line 610 in sqlfs2.go) to gzip JSON results after marshaling, and update the read path (around line 587) to transparently decompress when serving the result file.
• This approach reduces per-session memory usage by 50-70% without changing the virtual filesystem interface or affecting the existing auto-cleanup mechanism.

Frequently Asked Questions

How does session-based memory auto-compression affect the existing session timeout behavior?

The compression layer operates independently of the cleanup logic. The SessionManager continues to track lastAccess timestamps and removes idle sessions after the configured session_timeout expires. Compressed payloads are evicted along with their parent Session objects, so no additional cleanup code is required.

What compression algorithm does the implementation use?

The reference implementation uses compress/gzip from the Go standard library with default compression levels. This provides a balance between CPU usage and compression ratio (typically 2-3x for JSON). You can substitute flate, snappy, or zstd by replacing the compress() and decompress() helper functions in sqlfs2.go.

Will clients see any difference in the data returned from the result file?

No. The compression is transparent to virtual filesystem clients. When reading the result file, the plugin decompresses the payload on-the-fly before returning data via plugin.ApplyRangeRead(). Clients continue to receive plain JSON identical to the uncompressed implementation.

How can I verify that compression is actually reducing memory usage?

You can verify memory savings by running a large SELECT query and inspecting the Go heap using pprof. After implementing the changes, the Session objects should show compressedResult fields significantly smaller than the original JSON payloads. Unit tests can also assert that len(session.compressedResult) < len(jsonData) for typical result sets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}