How Hysteria Core Handles Network Traffic: A Deep Dive into QUIC-Based Proxy Architecture

Hysteria tunnels all TCP and UDP traffic through a single authenticated QUIC connection using HTTP/3 for control signaling, multiplexing streams and datagrams with configurable congestion control and traffic accounting hooks.

The Hysteria proxy core (apernet/hysteria) reimplements traditional VPN traffic handling by leveraging quic-go to establish a single long-lived QUIC connection between client and server. Instead of managing multiple TCP sockets, Hysteria handles network traffic by treating all user data as application-level frames (for TCP) or datagrams (for UDP) flowing through this unified transport layer.

QUIC Foundation and HTTP/3 Control Plane

At the transport layer, Hysteria relies on QUIC (RFC 9000) via the quic-go library, using HTTP/3 exclusively for the initial handshake and authentication. This design eliminates head-of-line blocking and enables native stream multiplexing over UDP.

The connection lifecycle begins in core/client/client.go, where clientImpl.connect initiates the QUIC handshake and immediately sends an HTTP POST request containing the client’s authentication token and bandwidth requirements. On the server side, core/server/server.go implements h3sHandler.ServeHTTP to validate these credentials. Upon success, the server returns capability flags—including UDP support status and receive bandwidth limits—before transitioning the connection to the data phase.

Three-Phase Traffic Handling

Once authenticated, Hysteria manages user traffic through three distinct mechanisms:

Phase 1: TCP Stream Proxying

For TCP traffic, Hysteria opens new QUIC streams within the existing connection. When a client calls clientImpl.TCP, the implementation invokes clientImpl.openStream and sends a FrameTypeTCPRequest frame defined in core/internal/protocol/proxy.go.

The frame structure—written via protocol.WriteTCPRequest—contains the target address, padding for traffic shaping, and metadata. The server’s ProxyStreamHijacker intercepts this frame type and delegates to handleTCPRequest, which:

1. Parses the target using protocol.ReadTCPRequest
2. Optionally executes a RequestHook for filtering
3. Dials the destination via the configured outbound
4. Returns a TCP response frame via protocol.WriteTCPResponse
5. Proxies data bidirectionally using copyTwoWay or copyTwoWayEx (the latter for traffic-logging support)

The client receives the server response through protocol.ReadTCPResponse and returns a net.Conn-compatible wrapper (tcpConn) from core/internal/utils/qstream.go, allowing standard Go I/O operations over the QUIC stream.

Phase 2: UDP Datagram Transport

UDP traffic utilizes QUIC’s unreliable datagram extension (RFC 9000 §19.2) rather than streams. After authentication, the server initializes a udpSessionManager that owns a udpIOImpl instance.

On the client side, clientImpl.UDP returns a HyUDPConn interface. Outbound packets are wrapped in a protocol.UDPMessage structure—defined in core/internal/protocol/proxy.go—which includes session IDs, packet IDs, target addresses, and payloads. These messages are serialized via UDPMessage.Serialize and transmitted using quic.Conn.SendDatagram.

The server-side udpIOImpl.ReceiveMessage polls quic.Conn.ReceiveDatagram, deserializes messages using protocol.ParseUDPMessage, and forwards them to the outbound UDP endpoint. Replies follow the reverse path, enabling stateful UDP session management without maintaining independent socket pairs.

Phase 3: Path MTU Discovery (Optional)

While disabled by default, Hysteria includes an optional Path MTU Discovery (PMTUD) subsystem in core/internal/pmtud/ to optimize packet sizes for the underlying QUIC connection.

Congestion Control and Bandwidth Enforcement

Both client and server invoke the core/internal/congestion package immediately after authentication to enforce negotiated bandwidth limits.

If the client specifies a receive bandwidth (Rx), the server selects between two modes:

• congestion.UseConfigured: Applies standard QUIC congestion controllers (BBR or Cubic)
• congestion.UseBrutal: Enforces a hard bandwidth cap using Hysteria’s aggressive rate-limiting algorithm

These calls occur in h3sHandler.ServeHTTP on the server and clientImpl.connect on the client. When a TrafficLogger determines that a quota is exceeded, the connection terminates with HTTP/3 error code 0x107 (closeErrCodeTrafficLimitReached), checked in both TCP and UDP I/O paths including udpIOImpl.ReceiveMessage and udpIOImpl.SendMessage.

Traffic Logging and Event Hooks

Hysteria exposes two primary interfaces for observability:

• TrafficLogger: Handles per-byte accounting via TraceStream/UntraceStream (TCP) and LogTraffic (UDP)
• EventLogger: Provides audit hooks including Connect, Disconnect, TCPRequest, TCPError, UDPRequest, and UDPError

These interfaces enable advanced use cases such as per-user quota enforcement, request filtering via RequestHook hooks, and dynamic outbound selection. The hooks are invoked at critical points in core/server/server.go during stream establishment and data transfer.

Implementation Examples

Establishing an Authenticated Client Connection

import (
	"log"
	"github.com/apernet/hysteria/core/v2/client"
)

func main() {
	// Load configuration
	cfg, _ := client.LoadConfig("client.yaml")

	// Create client - performs QUIC handshake and HTTP/3 authentication
	c, info, err := client.NewClient(cfg)
	if err != nil {
		log.Fatalf("handshake failed: %v", err)
	}
	log.Printf("Connected - UDP enabled: %v, Tx limit: %d", info.UDPEnabled, info.Tx)
	
	// The client now maintains a single QUIC connection for all traffic
}

This example triggers clientImpl.connect in core/client/client.go, which handles the HTTP POST authentication and negotiates congestion control parameters.

Tunneling TCP Through QUIC Streams

// Open a TCP tunnel to example.com:80
conn, err := c.TCP("example.com:80")
if err != nil {
	log.Fatalf("TCP tunnel error: %v", err)
}
defer conn.Close()

// Use as standard net.Conn
conn.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
buf := make([]byte, 4096)
n, _ := conn.Read(buf)
log.Printf("Response: %s", buf[:n])

Under the hood, this calls clientImpl.TCP, which invokes openStream and exchanges protocol frames defined in core/internal/protocol/proxy.go with the server’s handleTCPRequest.

Transmitting UDP Datagrams

// Obtain UDP session manager
udp, err := c.UDP()
if err != nil {
	log.Fatalf("UDP not enabled: %v", err)
}
defer udp.Close()

// Send DNS query to 8.8.8.8:53
payload := []byte{ /* DNS packet bytes */ }
if err := udp.Send(payload, "8.8.8.8:53"); err != nil {
	log.Fatalf("send error: %v", err)
}

// Receive response
resp, src, err := udp.Receive()
if err != nil {
	log.Fatalf("receive error: %v", err)
}
log.Printf("Got %d bytes from %s", len(resp), src)

This utilizes clientImpl.UDP and the udpIOImpl structure, serializing packets via protocol.UDPMessage and transmitting them as QUIC datagrams.

Implementing Custom Traffic Quotas

type quotaLogger struct{}

func (l *quotaLogger) LogTraffic(authID string, tx, rx uint64) bool {
	// Return false to disconnect client when quota exceeded
	if rx > 10<<20 { // 10 MiB limit
		return false
	}
	return true
}

func main() {
	cfg, _ := client.LoadConfig("client.yaml")
	cfg.TrafficLogger = &quotaLogger{}
	// Client creation proceeds with quota enforcement...
}

The LogTraffic method is invoked for every TCP stream and UDP packet exchange, as implemented in the server-side I/O paths.

Summary

• Hysteria handles network traffic over a single QUIC connection using HTTP/3 for authentication, eliminating the overhead of multiple TCP handshakes.
• TCP proxying uses QUIC bidirectional streams with custom framing in core/internal/protocol/proxy.go, wrapping them in net.Conn-compatible interfaces.
• UDP forwarding leverages QUIC unreliable datagrams and the UDPMessage protocol for session management without socket overhead.
• Bandwidth enforcement occurs through the congestion package, supporting both standard controllers and Brutal mode with hard caps.
• Extensibility is provided via TrafficLogger and EventLogger hooks integrated into core/server/server.go, enabling quota management and request filtering.

Frequently Asked Questions

How does Hysteria differ from traditional TCP-based VPNs?

Traditional VPNs encapsulate IP packets or proxy TCP connections through separate TCP sessions, suffering from head-of-line blocking during congestion. Hysteria handles network traffic through a single QUIC connection that multiplexes multiple streams and datagrams over UDP, providing built-in congestion control, faster handshakes via TLS 1.3, and native stream isolation without TCP’s three-way overhead per connection.

What is the Brutal congestion control mode in Hysteria?

Brutal mode (congestion.UseBrutal) is an aggressive bandwidth-limiting algorithm that enforces hard caps on throughput regardless of network conditions, unlike BBR or Cubic which adapt to available bandwidth. It is activated when the client specifies strict bandwidth limits, ensuring that Hysteria traffic does not exceed allocated quotas even on high-capacity links.

How does Hysteria maintain UDP session state over QUIC?

Hysteria implements a udpSessionManager that maps QUIC datagrams to logical UDP sessions using the UDPMessage structure in protocol/proxy.go. Each datagram carries session and packet identifiers, allowing the server’s udpIOImpl to correlate incoming and outgoing packets without maintaining traditional UDP socket pairs, while ReceiveMessage and SendMessage handle asynchronous I/O over the single QUIC connection.

Where are traffic limits enforced in the Hysteria core?

Bandwidth limits are enforced in core/server/server.go within h3sHandler.ServeHTTP and in core/client/client.go within clientImpl.connect through the congestion.UseConfigured and congestion.UseBrutal functions. When quotas are breached, the TrafficLogger triggers connection closure with error code 0x107 in both TCP stream handlers and UDP I/O implementations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}