How Compression Is Implemented in OpenFlux Transports: A Deep Dive into LZ4 Optimization

OpenFlux implements transport compression using a CompressedTransport decorator pattern that wraps any concrete transport, applying LZ4 compression only when payloads exceed 200 bytes and actually reduce in size.

The p1neappleXpress/OpenFlux repository provides a lightweight, transparent compression layer for TCP transport implementations. By wrapping existing transports with a decorator, the system enables on-the-fly payload compression without modifying individual transport logic.

The CompressedTransport Decorator Pattern

OpenFlux uses the decorator design pattern to add compression capabilities to any transport implementation. The wrapper resides in transport/compressor.go and implements the same Transport interface defined in transport/transport.go, allowing seamless integration with existing code.

The decorator intercepts Send and Receive calls, automatically applying compression logic based on payload characteristics. This approach decouples compression concerns from transport-specific implementations like YandexDocs, OneMe, or Cups.online transports.

Compression Thresholds and Markers

The compression system uses specific constants to determine when and how to compress data.

Size-Based Compression Logic

OpenFlux only attempts compression when the payload meets the minimum threshold:

• MinCompressSize = 200 bytes – Packets smaller than this limit are transmitted uncompressed to avoid overhead
• Size comparison – Even for packets exceeding 200 bytes, the wrapper compares the compressed output against the original size; if compression does not reduce the payload, it falls back to uncompressed transmission

Protocol Markers for Transport Negotiation

The system prefixes all packets with marker bytes to allow the receiver to distinguish between compressed and uncompressed data:

• CompressionMarker = 0x1F – Indicates the following bytes contain LZ4-compressed data
• 0x00 – Indicates the remaining bytes are uncompressed raw payload

These single-byte headers enable stateless decompression on the receiving end without requiring out-of-band negotiation.

Send and Receive Flows

The CompressedTransport wrapper intercepts data flows through two primary paths.

Compression Flow (Send)

When Send(data) is called on a wrapped transport, the following logic executes according to the source in transport/compressor.go:

1. Check if len(data) <= MinCompressSize – if true, prefix with 0x00 and forward unchanged
2. For larger payloads, write the CompressionMarker (0x1F)
3. Create an LZ4 writer using github.com/pierrec/lz4/v4
4. Stream data into the LZ4 writer and close it
5. Compare compressed buffer size against original size
6. If compressed size is not smaller, discard and send 0x00 + original data instead

inner := yandex.NewYandexDocsTransport(docURL, cfg) // any Transport implementation
comp := transport.NewCompressedTransport(inner)

if err := comp.Start(); err != nil {
    log.Fatalf("failed to start transport: %v", err)
}
defer comp.Stop()

// Sending data – automatically compressed if large enough
payload := []byte("...large TCP payload...")
if err := comp.Send(payload); err != nil {
    log.Printf("send error: %v", err)
}

Decompression Flow (Receive)

The Receive callback in the wrapper inspects incoming packets via the decompress(data) function:

• If the first byte is 0x00, the payload is simply data[1:] (uncompressed)
• If the first byte is 0x1F, the remaining bytes feed into an LZ4 reader to restore the original data
• Any decompression error triggers a safety fallback, passing the raw data through unchanged

// Receiving data – automatically decompressed
comp.Receive(func(pkt []byte) {
    // pkt is the original uncompressed payload
    handlePacket(pkt)
})

Integration Points and Usage Examples

The compressed transport is instantiated at key entry points in the codebase. In main.go at line 111, the wrapper decorates the transport pipeline:

// main.go (line 111)
trans := transport.NewCompressedTransport(inner)

For iOS static library builds, the wrapper appears in export_ios.go between lines 170-173:

// export_ios.go (lines 170-173)
t = transport.NewCompressedTransport(yandex.NewYandexDocsTransport(docURL, config))

These integration points demonstrate that every transport automatically benefits from LZ4 compression without requiring changes to the underlying transport implementations.

Performance Characteristics and Fallback Behavior

The compression layer prioritizes throughput over compression ratio by using the LZ4 algorithm, known for extremely fast compression and decompression speeds. The implementation includes intelligent fallbacks:

• Small packet optimization – Packets under 200 bytes bypass compression entirely, avoiding CPU overhead for trivial payloads
• Negative compression handling – When LZ4 produces output larger than the input (possible with incompressible data), the system transparently falls back to uncompressed transmission
• Error resilience – Decompression failures result in raw data passthrough, ensuring connectivity even with corrupted or malformed packets

Summary

• Decorator architecture – The CompressedTransport wrapper in transport/compressor.go wraps any concrete transport implementation without modifying underlying code
• Smart compression triggers – Only payloads exceeding 200 bytes are candidates for compression, and only if the compressed result is actually smaller
• Lightweight protocol – Single-byte markers (0x1F for compressed, 0x00 for raw) enable transparent protocol negotiation
• LZ4 implementation – Uses github.com/pierrec/lz4/v4 for high-speed compression with automatic fallback on errors or incompressible data
• Universal integration – Activated in main.go and export_ios.go, providing compression across all transport types including Yandex, OneMe, and Cups.online

Frequently Asked Questions

What compression algorithm does OpenFlux use?

OpenFlux uses LZ4 compression via the github.com/pierrec/lz4/v4 library. LZ4 was chosen for its extremely fast compression and decompression speeds, making it ideal for real-time network transport where latency matters more than maximum compression ratios.

At what payload size does OpenFlux start compressing data?

OpenFlux only considers compression for payloads larger than 200 bytes, defined by the MinCompressSize constant in transport/compressor.go. Packets at or below this threshold are prefixed with 0x00 and transmitted uncompressed to avoid unnecessary CPU overhead.

How does OpenFlux handle decompression errors?

The decompress function implements a safety fallback: if any error occurs during LZ4 decompression, the system passes the raw data through unchanged rather than failing the transport. This ensures that corrupted packets or protocol mismatches do not break the connection, though the application layer receives uncompressed data in these edge cases.

Can I disable compression for specific transports?

While the base transport/transport.go interface does not expose a toggle, you can disable compression simply by not wrapping the transport with NewCompressedTransport(). Since compression is implemented as an optional decorator rather than a core feature, using the underlying transport directly (e.g., yandex.NewYandexDocsTransport() without the wrapper) bypasses compression entirely while maintaining full functionality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}