How OpenFlux Uses gVisor to Build a Userspace TCP/IP Stack

OpenFlux leverages the gVisor networking library to implement a complete TCP/IP stack in userspace, enabling virtual network interfaces and standard Go socket operations without kernel modifications.

OpenFlux (available at p1neappleXpress/OpenFlux) creates an isolated network environment by embedding gVisor's pkg/tcpip implementation. This design allows the application to intercept, route, and process TCP traffic entirely in userspace, supporting both client tunnels and exit node configurations through a dual-NIC architecture.

Architecture Overview

The core implementation resides in tunnel/tunnel.go, where OpenFlux instantiates a stack.Stack from gvisor.dev/gvisor/pkg/tcpip/stack. The stack is configured with IPv4 and TCP protocol factories to create a self-contained networking environment:

t.gvisorStack = stack.New(stack.Options{
    NetworkProtocols:   []stack.NetworkProtocolFactory{ipv4.NewProtocol},
    TransportProtocols: []stack.TransportProtocolFactory{tcp.NewProtocol},
})

This initialization (lines 58–62 in tunnel/tunnel.go) establishes the foundation for all subsequent network operations, enabling the userspace stack to handle packet processing independently of the host kernel.

Virtual Network Interface Configuration

OpenFlux implements two virtual Network Interface Cards (NICs) to separate tunnel traffic from internet-bound traffic. Each NIC connects to a distinct link endpoint that bridges the gVisor stack with OpenFlux's transport layer.

The Tunnel NIC (NIC 1)

The Tunnel NIC uses a custom TunnelLinkEndpoint (defined in tunnel/endpoint.go) to exchange raw IP packets with the OpenFlux transport layer. This endpoint implements stack.LinkEndpoint and provides the entry point for all tunneled traffic.

The Internet NIC (NIC 2)

When operating as an exit node, OpenFlux creates a second NIC using RawSocketEndpoint. This interface forwards packets directly to the host OS through raw sockets, allowing the exit node to bridge traffic between the tunnel and the public internet.

TCP Buffer Configuration

Before handling traffic, OpenFlux tunes the stack's buffer limits via SetTCPBuffers. This method applies configurable TCPBufMin, TCPBufDefault, and TCPBufMax values to the TCP transport protocol (lines 39–48 in tunnel/tunnel.go):

tcpProtocol := t.gvisorStack.TransportProtocolInstance(tcp.ProtocolNumber).(*tcp.Protocol)
tcpProtocol.SetSendBufferSizeRange(min, def, max)
tcpProtocol.SetReceiveBufferSizeRange(min, def, max)

Proper buffer sizing ensures efficient throughput while preventing memory exhaustion in the userspace stack.

Client vs. Exit Node Routing Logic

OpenFlux configures the stack differently depending on whether the node operates as a client or an exit relay.

Exit Node Configuration

The setupExitNode function (lines 95–127) initializes the internet-facing NIC and enables packet forwarding:

t.gvisorStack.CreateNIC(internetNIC, rawEP)
t.gvisorStack.AddProtocolAddress(internetNIC, tcpip.ProtocolAddress{
    Protocol: ipv4.ProtocolNumber,
    AddressWithPrefix: tcpip.AddressWithPrefix{
        Address:   tcpip.AddrFrom4(ip),
        PrefixLen: 24,
    },
})
t.gvisorStack.SetForwardingDefaultAndAllNICs(ipv4.ProtocolNumber, true)

This configuration assigns the exit node's external IP to NIC 2 and installs routes for both the tunnel subnet (10.10.10.0/24) and the default internet route (0.0.0.0/0).

Client-Side Configuration

For client nodes, setupClient registers only the tunnel NIC with a static address of 10.10.10.2/24 and sets the default route to forward all traffic through the tunnel interface (lines 39–53). This ensures complete traffic isolation within the virtual network.

Data Path and Packet Flow

The bidirectional data flow relies on callback functions wired during stack initialization (lines 66–85):

Outbound path: When the gVisor stack sends packets, the TunnelLinkEndpoint triggers onOutgoingPacket, which forwards serialized IP data to the transport layer:

tunnelEP.onOutgoingPacket = func(data []byte) { trans.Send(data) }

Inbound path: Incoming packets from the transport layer are injected back into the stack via InjectInbound:

trans.Receive(func(data []byte) { tunnelEP.InjectInbound(data) })

This design decouples the TCP/IP implementation from the underlying transport mechanism, allowing OpenFlux to work over various transport protocols while maintaining standard TCP semantics.

Exposing Standard Go Network Interfaces

To provide a familiar programming interface, OpenFlux uses the gonet adapters from gvisor.dev/gvisor/pkg/tcpip/adapters/gonet. These adapters wrap the userspace stack to expose standard net.Conn and net.Listener interfaces.

Dialing outbound connections uses gonet.DialTCP (lines 72–77), which creates TCP connections inside the gVisor stack and routes them through the appropriate NIC:

conn, err := gonet.DialTCP(t.gvisorStack, localAddr, remoteAddr, ipv4.ProtocolNumber)

Listening for incoming traffic uses gonet.ListenTCP (lines 81–86) to bind to addresses within the virtual stack:

ln, err := gonet.ListenTCP(t.gvisorStack, fullAddr, ipv4.ProtocolNumber)

Both methods return objects that satisfy Go's standard networking interfaces while operating entirely within the userspace TCP/IP implementation.

Observability and Debugging

A background goroutine (printStats) periodically exports stack statistics via t.gvisorStack.Stats() (lines 88–101). This functionality tracks metrics including:

• Active socket count
• Established connections
• TCP retransmits
• Dropped packets

These statistics help operators monitor the health of the userspace network and diagnose connectivity issues without kernel-level debugging tools.

Practical Usage Example

The following example demonstrates how client applications and exit nodes instantiate the TCP tunnel and interact with the gVisor-backed network stack:

// Create a tunnel for a client (non‑exit node)
clientTunnel := tunnel.NewTCPTunnel(myTransport, false)

// Open a TCP connection to a remote host via the gVisor stack
conn, err := clientTunnel.DialTCP("example.com:443")
if err != nil { log.Fatal(err) }
defer conn.Close()

// On an exit node, start listening for incoming TCP connections
ln, err := clientTunnel.ListenTCP(8080)
if err != nil { log.Fatal(err) }
for {
    c, _ := ln.Accept()
    go handle(c) // c is a *net.TCPConn backed by the gVisor stack
}

Summary

• OpenFlux implements a userspace TCP/IP stack using gVisor's pkg/tcpip/stack package in tunnel/tunnel.go.
• The architecture uses two virtual NICs: a custom TunnelLinkEndpoint for tunnel traffic and a RawSocketEndpoint for exit node internet access.
• gonet adapters bridge the gap between gVisor's internal networking and Go's standard net package interfaces.
• Buffer tuning via SetTCPBuffers optimizes memory usage for TCP connections.
• The dual-mode design supports both client tunnels (isolated routing) and exit nodes (full internet bridging) through conditional NIC configuration.

Frequently Asked Questions

What is gVisor and why does OpenFlux use it?

gVisor is an application kernel for containers that provides a sandboxed environment, including a pure Go implementation of a TCP/IP stack. OpenFlux uses gVisor to intercept and manage network traffic entirely in userspace, eliminating the need for kernel modules or TUN device management while maintaining full TCP protocol compatibility.

How does OpenFlux route packets between the tunnel and the internet?

OpenFlux routes packets through NIC-level forwarding enabled by SetForwardingDefaultAndAllNICs. In exit node mode, packets arriving from the tunnel (NIC 1) are processed by the gVisor stack and forwarded to the internet-facing raw socket (NIC 2). The stack maintains separate routing tables for the tunnel subnet (10.10.10.0/24) and default internet routes.

Can OpenFlux handle UDP traffic with this implementation?

Based on the current source code in tunnel/tunnel.go, OpenFlux specifically initializes the stack with tcp.NewProtocol only (line 61). While gVisor supports UDP through udp.NewProtocol, the current implementation focuses exclusively on TCP traffic. UDP support would require adding the UDP protocol factory during stack initialization.

How does the userspace stack impact performance compared to kernel networking?

The userspace implementation incurs additional context switching and memory copying between the gVisor stack and the transport layer. However, the dedicated TCP buffer configuration (TCPBufMin/Default/Max) allows fine-tuned memory allocation, and the elimination of kernel syscalls for packet processing can reduce latency in high-throughput tunnel scenarios. The trade-off provides greater flexibility and portability across operating systems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}