Network Protocols Used by Hysteria Core: QUIC, HTTP/3, and Custom Binary Framing Explained

Hysteria core employs a custom-tuned QUIC transport (RFC 9000) with TLS 1.3 encryption, using HTTP/3 for authentication handshakes and lightweight binary frames for TCP/UDP proxy traffic.

The apernet/hysteria repository provides a high-performance proxy core that circumvents network restrictions using a unique protocol stack. Understanding the network protocols used by Hysteria core reveals why it achieves lower latency and better obfuscation compared to traditional TCP-based proxies.

The Core Protocol Stack

Hysteria 2 multiplexes all traffic—including authentication, TCP forwarding, and UDP tunneling—over a single QUIC connection. This design eliminates head-of-line blocking and provides built-in encryption by default.

QUIC as the Transport Layer (RFC 9000)

QUIC serves as the sole transport protocol for both control and data traffic. The core explicitly enables datagram support to handle UDP forwarding efficiently.

In core/server/server.go (lines 50-60), the server initializes a quic.Listener with EnableDatagrams: true, allowing the transport to carry both reliable streams (for TCP proxying) and unreliable datagrams (for UDP traffic) within the same connection.

TLS 1.3 Security Layer

All QUIC packets are encrypted using TLS 1.3, which runs over QUIC rather than TCP. The core prepares the TLS configuration via convertToStdTLSConfig in core/server/server.go (lines 31-44), handling server authentication and optional client certificate verification. This provides confidentiality and integrity for the entire protocol stack without requiring additional encryption layers.

HTTP/3 Control Channel (RFC 9204)

Authentication and initial handshake occur over HTTP/3, which leverages QUIC streams. The server runs an http3.Server (defined in core/server/server.go, lines 43-51) that listens for POST requests on the root path (/).

Clients submit authentication credentials via HTTP/3 headers, validated by the server before allowing access to the proxy functionality. This approach reuses standard HTTP semantics while gaining QUIC's speed and reliability advantages.

Custom Binary Framing for Data

Once authenticated, TCP and UDP proxy traffic uses custom binary frames multiplexed over QUIC. These lightweight structures minimize overhead compared to HTTP request/response cycles:

• TCP frames (type 0x401) carry destination addresses and payload data over QUIC streams
• UDP messages carry session IDs, packet IDs, fragmentation metadata, and payload over QUIC datagrams

Both formats use QUIC variable-length integers (varints) for compact encoding of lengths and identifiers, as implemented in core/internal/protocol/proxy.go (lines 26-30).

Protocol Implementation Details

The binary framing protocol defined in core/internal/protocol/proxy.go handles the actual data transfer after HTTP/3 authentication completes.

TCP Request and Response Frames

TCP proxying uses a simple request-response model over QUIC bidirectional streams. The frame structure (lines 14-38 in proxy.go) includes:

• A fixed frame type identifier (0x401)
• The target address as a length-prefixed string
• Optional padding for traffic shaping

The server responds with a similar frame indicating success or failure before entering a raw data relay mode.

UDP Message Structure

UDP forwarding presents unique challenges due to packet boundaries and potential fragmentation. The UDP message format (lines 51-67 in proxy.go) contains:

• Session ID: Identifies the client session
• Packet ID: Allows reordering and deduplication
• Fragment ID: Supports splitting large UDP packets across multiple QUIC datagrams
• Payload: The actual UDP packet content

This design ensures unreliable UDP semantics are preserved even when transported over QUIC's reliable streams or datagrams.

Variable-Length Integer Encoding

Throughout the custom protocol, Hysteria uses QUIC varint encoding to minimize wire size. This compact representation (defined in the QUIC specification) efficiently encodes small integers (like port numbers and frame lengths) using fewer bytes than fixed-width alternatives.

Practical Code Examples

Sending TCP Request Frames

To initiate a TCP proxy connection, the client constructs a request frame using the protocol package:

import (
	"github.com/apernet/hysteria/core/v2/internal/protocol"
	"io"
)

// addr is the target address, e.g. "example.com:443"
func sendTCPRequest(w io.Writer, addr string) error {
	// WriteTCPRequest serializes the binary frame defined in proxy.go
	return protocol.WriteTCPRequest(w, addr)
}

Implementation reference: protocol.WriteTCPRequest in core/internal/protocol/proxy.go (lines 69-84).

Parsing UDP Datagrams

When receiving UDP traffic from the QUIC connection, the server parses the custom frame format:

import (
	"github.com/apernet/hysteria/core/v2/internal/protocol"
)

// raw contains the datagram payload received from a QUIC connection
func decodeUDPMessage(raw []byte) (*protocol.UDPMessage, error) {
	return protocol.ParseUDPMessage(raw)
}

Implementation reference: protocol.ParseUDPMessage in core/internal/protocol/proxy.go (lines 93-101).

Initializing the HTTP/3 Server

Server startup involves configuring TLS, QUIC, and HTTP/3 layers:

import (
	"github.com/apernet/hysteria/core/v2"
	"github.com/apernet/hysteria/core/v2/server"
)

func runServer(cfg *core.Config) error {
	// NewServer creates the TLS-QUIC listener and HTTP/3 handler
	srv, err := server.NewServer(cfg)
	if err != nil {
		return err
	}
	// Serve blocks, handling HTTP/3 authentication and proxy frames
	return srv.Serve()
}

Implementation reference: server.NewServer and http3.Server in core/server/server.go (lines 46-110).

Critical Source Files

These files define the network protocols used by Hysteria core:

• core/internal/protocol/proxy.go – Defines binary formats for TCP request/response frames and UDP datagram structures, plus varint encoding helpers.
• core/internal/protocol/http.go – Contains helpers mapping Hysteria authentication data to HTTP/3 headers.
• core/server/server.go – Implements the TLS-QUIC listener, HTTP/3 server setup, and stream/datagram routing to proxy handlers.
• core/client/client.go – Mirrors the server implementation, constructing the HTTP/3 transport and managing custom proxy frames.
• extras/sniff/internal/quic/ – Utilities for detecting QUIC packets (used by the traffic sniffing feature).

Summary

• QUIC (RFC 9000) with datagram support serves as the exclusive transport layer, handling both streams and unreliable datagrams.
• TLS 1.3 encrypts all traffic and handles authentication without additional overhead.
• HTTP/3 (RFC 9204) manages the initial authentication handshake via POST requests on a fixed endpoint.
• Custom binary frames carry TCP and UDP proxy payloads efficiently over QUIC, using varint encoding for compactness.
• No raw TCP or UDP protocols are used in the core transport; traditional protocols only appear at the proxy endpoints.

Frequently Asked Questions

Does Hysteria core use raw TCP or UDP for its transport layer?

No. According to the source code in core/server/server.go, Hysteria core exclusively uses QUIC (RFC 9000) as its transport protocol. While the proxy forwards traffic to TCP and UDP endpoints, the client-to-server communication always occurs over QUIC with EnableDatagrams: true configured to support UDP forwarding.

Why does Hysteria use HTTP/3 for authentication instead of a custom protocol?

HTTP/3 provides standardized request/response semantics and stream management while leveraging QUIC's performance benefits. The server implementation in core/server/server.go (lines 43-51) uses http3.Server to handle POST requests on the root path, allowing credentials to pass via standard HTTP headers rather than inventing a new handshake mechanism.

What is the significance of the 0x401 frame type in TCP proxying?

The hex value 0x401 identifies TCP proxy request and response frames in Hysteria's custom binary protocol, defined in core/internal/protocol/proxy.go (lines 14-38). This type discriminator distinguishes proxy control frames from other traffic types multiplexed over the same QUIC connection.

How does Hysteria handle UDP packet fragmentation?

The custom UDP message frame includes Packet ID and Fragment ID fields (specified in core/internal/protocol/proxy.go, lines 51-67). These allow large UDP datagrams to be split across multiple QUIC datagrams and reassembled at the destination, preserving the original packet boundaries despite QUIC's underlying packetization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}