How Xray-core Implements QUIC and HTTP/3 Support for the Hysteria Protocol

Xray-core implements Hysteria by layering HTTP/3 authentication over QUIC streams and datagrams, with TCP traffic prefixed by a custom frame type (0x401) and UDP traffic carried directly via QUIC datagrams.

The Hysteria protocol in Xray-core delivers high-performance proxying by combining QUIC (via quic-go) with HTTP/3 for the initial handshake. This implementation enables both reliable TCP streams and unreliable UDP datagrams over a single QUIC connection, with sophisticated congestion control negotiation between client and server.

---

How Xray-core Hysteria Uses QUIC as the Transport Foundation

QUIC Connection Setup with Datagram Support

Both server and client in Xray-core configure QUIC with datagrams enabled to support UDP traffic. The configuration is built in dialer.go (client) and hub.go (server):

quicConfig := &quic.Config{
    EnableDatagrams:      true,
    MaxDatagramFrameSize: MaxDatagramFrameSize, // 1200 bytes in config.go:L15
    InitialStreamReceiveWindow:     quicParams.InitStreamReceiveWindow,
    MaxStreamReceiveWindow:         quicParams.MaxStreamReceiveWindow,
    InitialConnectionReceiveWindow: quicParams.InitConnReceiveWindow,
    MaxConnectionReceiveWindow:     quicParams.MaxConnReceiveWindow,
    MaxIdleTimeout:     time.Duration(quicParams.MaxIdleTimeout) * time.Second,
    KeepAlivePeriod:    time.Duration(quicParams.KeepAlivePeriod) * time.Second,
    DisablePathMTUDiscovery: quicParams.DisablePathMtuDiscovery,
}

The MaxDatagramFrameSize is capped at 1200 bytes — the typical path MTU for UDP-based QUIC — to prevent fragmentation issues.

---

HTTP/3 Authentication Handshake in Xray-core Hysteria

Server-Side HTTP/3 Server

In transport/internet/hysteria/hub.go, the server wraps the QUIC listener with an HTTP/3 server:

qListener, err := quic.Listen(pktConn, tlsConfig.GetTLSConfig(), quicConfig) // hub.go:L30-L33

h3 := http3.Server{
    Handler:          handler,
    StreamDispatcher: handler.StreamDispatcher,
}
err := h3.ServeQUICConn(conn) // hub.go:L84-L86

The http3.Server reuses the QUIC connection and dispatches inbound streams to the custom httpHandler.

The /auth POST Request

Both client and server communicate via a single POST /auth request to https://hysteria/auth. The server handler in hub.go processes this:

if r.Method == http.MethodPost && r.Host == URLHost && r.URL.Path == URLPath {
    // Authentication logic
    w.Header().Set(ResponseHeaderUDPEnabled, 
        strconv.FormatBool(hyCtx.RequireDatagramFromContext(h.ctx)))
    w.Header().Set(CommonHeaderCCRX, 
        strconv.FormatUint(h.quicParams.BrutalDown, 10))
    w.WriteHeader(StatusAuthOK) // hub.go:L60-L71
}

The client in dialer.go builds and sends the same request:

req := &http.Request{
    Method: http.MethodPost,
    URL: &url.URL{
        Scheme: "https",
        Host:   URLHost,
        Path:   URLPath,
    },
    Header: http.Header{
        RequestHeaderAuth:   []string{c.config.Auth},
        CommonHeaderCCRX:    []string{strconv.FormatUint(quicParams.BrutalDown, 10)},
        CommonHeaderPadding: []string{authRequestPadding.String()},
    },
}
resp, err := rt.RoundTrip(req) // dialer.go:L66-L86

The response headers communicate:

• UDP support: Hysteria-UDP header
• Congestion control parameters: Hysteria-CC-RX for "brutal" mode bandwidth

---

TCP over QUIC Streams with Custom Frame Type

Client-Side First-Frame Marker

After authentication, TCP traffic flows over QUIC streams. The client in conn.go prefixes the first write with a custom frame type:

func (i *interConn) Write(b []byte) (int, error) {
    if i.client {
        buf := make([]byte, 0, quicvarint.Len(FrameTypeTCPRequest)+len(b))
        buf = quicvarint.Append(buf, FrameTypeTCPRequest) // 0x401, config.go:L29
        buf = append(buf, b...)
        _, err := i.stream.Write(buf) // conn.go:L39-L44
        i.client = false
        return len(b), nil
    }
    return i.stream.Write(b)
}

The FrameTypeTCPRequest constant (0x401) is defined in config.go:

const FrameTypeTCPRequest = 0x401 // config.go:L29

Server-Side Stream Dispatcher

The server's StreamDispatcher in hub.go handles this framing:

func (h *httpHandler) StreamDispatcher(conn quic.Connection, stream quic.Stream) {
    // Read leading varint
    frameType, err := quicvarint.Read(stream)
    if err != nil {
        return
    }
    if frameType != FrameTypeTCPRequest {
        // Handle as standard HTTP/3 stream
        return
    }
    // Discard frame type, treat remainder as TCP data
    // hub.go:L240-L255
}

---

UDP over QUIC Datagrams

Server-Side UDP Session Manager

When the server enables UDP support, it creates a udpSessionManagerServer in hub.go:

if hyCtx.RequireDatagramFromContext(h.ctx) {
    udpSM := &udpSessionManagerServer{
        conn:           h.conn,
        m:              make(map[uint32]*InterUdpConn),
        addConn:        h.addConn,
        stopCh:         make(chan struct{}),
        udpIdleTimeout: time.Duration(h.config.UdpIdleTimeout) * time.Second,
        user:           h.user,
    }
    go udpSM.clean()  // garbage collect idle sessions
    go udpSM.run()    // receive and dispatch datagrams
} // hub.go:L285-L306

The run method receives QUIC datagrams:

func (u *udpSessionManagerServer) run() {
    for {
        data, err := u.conn.ReceiveDatagram()
        if err != nil {
            return
        }
        // Extract 4-byte session ID prefix, route to appropriate InterUdpConn
        u.feed(data)
    }
}

Client-Side UDP Session Manager

The client similarly creates a udpSessionManagerClient in dialer.go:

if serverUdp {
    c.udpSM = &udpSessionManagerClient{
        conn: quicConn,
        m:    make(map[uint32]*InterUdpConn),
        next: 1,
    }
    go c.udpSM.run()
} // dialer.go:L119-L128

UDP Datagram Format

UDP packets are serialized in proxy/hysteria/protocol.go. The client-side InterUdpConn.Write in conn.go sends:

func (u *InterUdpConn) Write(b []byte) (int, error) {
    if u.manager.isClient {
        // Prepend 4-byte session ID
        pkt := make([]byte, 4+len(b))
        binary.BigEndian.PutUint32(pkt, u.id)
        copy(pkt[4:], b)
        return len(b), u.manager.conn.SendDatagram(pkt) // conn.go:L28-L35
    }
    // ...
}

The proxy/hysteria/protocol.go defines the full UDP message format including packetID, fragID, and fragCount for fragmentation support.

---

Congestion Control Negotiation

After the /auth handshake, both sides apply congestion control to the QUIC connection. The logic in dialer.go (client) and hub.go (server) follows:

switch quicParams.Congestion {
case "reno":
    // Default QUIC congestion, no action needed
case "bbr":
    congestion.UseBBR(quicConn, bbr.Profile(quicParams.BbrProfile))
case "brutal", "":
    if serverAuto == "auto" || quicParams.BrutalUp == 0 || serverDown == 0 {
        // Fallback to BBR if auto-negotiation or bandwidth unknown
        congestion.UseBBR(quicConn, bbr.Profile(quicParams.BbrProfile))
    } else {
        // Use brutal with the minimum of client upload and server advertised download
        congestion.UseBrutal(quicConn, min(quicParams.BrutalUp, serverDown))
    }
}

The congestion package at transport/internet/hysteria/congestion/ provides UseBBR and UseBrutal wrappers that configure the underlying quic-go connection.

---

Summary

• QUIC foundation: Xray-core uses quic-go with datagrams enabled, configuring receive windows, idle timeout, and MTU discovery in dialer.go and hub.go.

• HTTP/3 handshake: A single POST /auth request over HTTP/3 negotiates authentication, UDP support (Hysteria-UDP header), and congestion control bandwidth (Hysteria-CC-RX).

• TCP transport: After handshake, TCP flows over raw QUIC streams with a custom 0x401 frame type prefix to distinguish proxy data from HTTP/3 traffic.

• UDP transport: When enabled, UDP uses QUIC datagrams with a 4-byte session ID prefix, managed by udpSessionManagerServer/udpSessionManagerClient that present net.Conn interfaces.

• Congestion control: Supports reno, bbr, and brutal algorithms, with auto-negotiation based on the /auth handshake bandwidth advertisement.

---

Frequently Asked Questions

What QUIC library does Xray-core use for Hysteria?

Xray-core uses quic-go as its underlying QUIC library. This is evident from import statements and function calls like quic.Listen(), quic.DialEarly(), and quic.Config throughout hub.go and dialer.go. The quic-go library provides the HTTP/3 implementation (http3.Server and http3.Transport) that Xray-core layers its Hysteria protocol on top of.

Why does Hysteria use HTTP/3 only for the initial handshake?

HTTP/3 is used only for the /auth POST request because it provides a convenient request-response abstraction for negotiating authentication, UDP support, and congestion control parameters. After this handshake completes, Xray-core bypasses HTTP/3 entirely for data transfer. TCP traffic uses raw QUIC streams with a custom 0x401 frame prefix, and UDP traffic uses QUIC datagrams directly. This design minimizes overhead while leveraging HTTP/3's standard semantics for initial negotiation.

How does Xray-core distinguish Hysteria TCP streams from regular HTTP/3 traffic?

Xray-core uses a custom frame type prefix (0x401, defined as FrameTypeTCPRequest in config.go:L29). On the client side, the first write to a new QUIC stream prepends this varint-encoded frame type in conn.go:L39-L44. The server's StreamDispatcher in hub.go:L240-L255 reads this leading varint, recognizes 0x401, and strips it before treating the remainder as proxy data. This allows Hysteria to coexist with standard HTTP/3 on the same QUIC connection.

What happens if the server and client disagree on UDP support?

UDP support is unilaterally determined by the server during the /auth handshake. The server sets the Hysteria-UDP response header based on hyCtx.RequireDatagramFromContext(h.ctx) in hub.go:L65. The client in dialer.go:L87 parses this header as serverUdp. If the server disables UDP (serverUdp == false), the client will not create its udpSessionManagerClient and all UDP traffic will fail. The client cannot force UDP if the server refuses it.

How does the "brutal" congestion control mode work?

The "brutal" congestion control is a bandwidth-based algorithm that shapes traffic to a fixed rate. After the /auth handshake, the client and server each run the selection logic in dialer.go:L98-L119 and hub.go:L88-L119. For "brutal" mode, they use congestion.UseBrutal(quicConn, min(clientUpload, serverDownload)) — the actual sending rate is capped at the minimum of what the client can upload and what the server advertised it can receive. This creates a fixed-rate tunnel that avoids traditional congestion-control back-off, optimized for high-bandwidth, high-latency networks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}