Understanding the Framing Mechanism for Compressed Data in OpenFlux

OpenFlux implements a lightweight single-byte prefix framing mechanism in transport/compressor.go that uses marker 0x1F for LZ4-compressed payloads and 0x00 for uncompressed data.

The OpenFlux messaging library employs a custom framing protocol to distinguish between compressed and uncompressed network payloads. This deterministic approach, defined in the transport package, enables receivers to transparently handle both small messages and large compressed blocks without external metadata headers.

How the Framing Mechanism Works

OpenFlux uses a single-byte prefix at the start of every packet to indicate the compression state of the following bytes.

The Single-Byte Marker Protocol

The protocol defines two constant markers at the top of transport/compressor.go:

• 0x00: Indicates the remainder of the packet contains raw, uncompressed data.
• 0x1F: Indicates the remainder contains an LZ4-compressed block.

During reception, the decompress function examines the first byte to determine processing:

if data[0] == 0x00 {
    return data[1:], nil // raw payload
}
r := lz4.NewReader(bytes.NewReader(data[1:]))
return io.ReadAll(r)    // LZ4-decompressed payload

Compression Threshold Logic

The implementation includes intelligent fallback logic to avoid compression overhead on small payloads. The MinCompressSize constant (set to 200 bytes) defines the minimum payload size eligible for compression.

When compress processes data larger than 200 bytes, it attempts LZ4 compression. If the resulting compressed size exceeds the original payload size plus the one-byte marker overhead, OpenFlux automatically falls back to the uncompressed format using marker 0x00.

Implementation in transport/compressor.go

The core constants and logic reside in transport/compressor.go:

const (
    MinCompressSize   = 200
    CompressionMarker = 0x1F // <-- indicates LZ4-compressed payload
)

The compress function creates an LZ4 writer, writes the payload, and prefixes the result with CompressionMarker. The decompress function strips the marker and returns either the raw bytes or the inflated LZ4 stream.

Working with CompressedTransport

OpenFlux wraps existing transports with compression support through NewCompressedTransport.

Sending Compressed Data

Automatic compression occurs when wrapping any transport implementation:

import "github.com/p1neappleXpress/OpenFlux/main/transport"

func sendExample(t transport.Transport, payload []byte) error {
    // Wrap the inner transport with compression support
    comp := transport.NewCompressedTransport(t)

    // The transport will add the appropriate marker automatically
    return comp.Send(payload)
}

Receiving and Decompressing

The CompressedTransport.Receive method handles framing transparently:

func receiveExample(t transport.Transport) {
    comp := transport.NewCompressedTransport(t)

    comp.Receive(func(msg []byte) {
        // `msg` is already decompressed if it was sent with the 0x1F marker
        fmt.Printf("Received %d bytes: %s\n", len(msg), string(msg))
    })
}

Manual Frame Inspection

For debugging or protocol analysis, you can inspect the framing marker directly:

func inspectMarker(data []byte) string {
    if len(data) == 0 {
        return "empty"
    }
    switch data[0] {
    case 0x00:
        return "uncompressed"
    case 0x1F:
        return "lz4-compressed"
    default:
        return "unknown"
    }
}

Summary

• OpenFlux uses a single-byte prefix framing mechanism where 0x1F signals LZ4 compression and 0x00 signals raw data.
• The logic is implemented in transport/compressor.go with CompressionMarker and MinCompressSize constants.
• Payloads smaller than 200 bytes are transmitted uncompressed to avoid overhead.
• The system automatically falls back to uncompressed format when compression would not reduce total size.
• LZ4 is the compression algorithm used for all compressed payloads.
• NewCompressedTransport wraps any existing transport to add transparent compression support.

Frequently Asked Questions

What compression algorithm does OpenFlux use for framed data?

OpenFlux uses LZ4 compression for all payloads marked with 0x1F. The implementation creates an lz4.Writer during compression and an lz4.NewReader during decompression, providing high-speed compression with minimal CPU overhead according to the source code in transport/compressor.go.

How does OpenFlux handle small messages that don't benefit from compression?

Messages smaller than MinCompressSize (200 bytes) are automatically sent uncompressed with the 0x00 marker. Additionally, if compression would result in a larger payload than the original data plus marker overhead, the system falls back to the uncompressed format to optimize bandwidth.

Can I use the compressed transport with encrypted connections?

Yes. OpenFlux supports transport layering, allowing CompressedTransport to wrap encryption transports or vice versa. The transport/encrypted.go file demonstrates how compressed transports compose with additional transport layers for end-to-end encryption while maintaining the single-byte framing protocol.

Where is the framing logic implemented in the OpenFlux source code?

The framing mechanism is implemented in transport/compressor.go, which defines the marker constants, compression threshold logic, and the compress and decompress functions that handle the single-byte prefix protocol.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}