How to Implement a Custom Application Protocol on Top of XQUIC: A Complete Guide

You implement a custom application protocol on top of XQUIC by registering an ALPN string with the engine using xqc_engine_register_alpn(), which binds your protocol-specific callback functions for connection, stream, and datagram events to that identifier.

The alibaba/xquic library provides a pluggable framework for building custom transport protocols over QUIC. To implement a custom application protocol on top of XQUIC, you leverage the ALPN (Application-Layer Protocol Negotiation) registration system that decouples transport handling from application logic. This approach allows you to define protocol-specific behaviors through structured callbacks while XQUIC manages the underlying QUIC connection lifecycle.

Understanding XQUIC's ALPN Registration Framework

XQUIC uses a callback-driven architecture where protocols register themselves via the xqc_engine_register_alpn() function defined in src/transport/xqc_engine.c. When a TLS handshake completes and negotiates an ALPN string, the engine looks up the registered callbacks and associated context through xqc_engine_get_alpn_ctx(), then dispatches events to your implementation.

The registration stores three critical components in an xqc_alpn_registration_t structure:

• The ALPN string identifying your protocol
• The callback tables (xqc_app_proto_callbacks_t) containing function pointers
• The user-defined context holding your protocol state

This mechanism powers the built-in HTTP/3 implementation in src/http3/xqc_h3_ctx.c and the HQ demo protocol in demo/xqc_hq_ctx.c, providing concrete reference implementations.

Step-by-Step Implementation Guide

Step 1: Define Your ALPN String

Choose a unique ASCII string (e.g., "myproto") not exceeding XQC_MAX_ALPN_LEN characters. This identifier must be distinct from standard protocols like "h3" or "hq-interop" and will be validated during the xqc_engine_register_alpn() call.

Step 2: Create the Protocol Context Structure

Define a context structure to maintain per-engine state. While HTTP/3 uses complex state machines in src/http3/xqc_h3_ctx.c, a simple protocol might only need configuration flags or connection counters.

Step 3: Implement Connection Callbacks

Define functions matching the signatures required by xqc_app_proto_callbacks_t to handle connection lifecycle events. You must implement connection creation and connection close callbacks:

• conn_create_notify: Invoked when a new xqc_connection_t is established
• conn_close_notify: Invoked when the connection terminates

Step 4: Implement Stream Callbacks

Implement the four required stream callbacks that operate on individual xqc_stream_t handles:

• stream_create_notify: Called when a peer opens a new stream
• stream_read_notify: Triggered when data is available; use xqc_stream_recv() to read
• stream_write_notify: Indicates the stream is ready for transmission; use xqc_stream_send() to write
• stream_close_notify: Called when the stream closes

Step 5: Populate the Callbacks Structure

Fill an xqc_app_proto_callbacks_t instance with pointers to your functions. Set the conn_cbs and stream_cbs members. Leave dgram_cbs NULL unless implementing QUIC datagram support as shown in src/http3/xqc_h3_ext_dgram.c.

Step 6: Register with the Engine

Call xqc_engine_register_alpn() during initialization, passing your ALPN string, the callbacks structure, and your context pointer. The engine stores this registration in its internal alpn_reg_list for runtime dispatch.

Complete Working Example: Building an Echo Protocol

The following example implements a minimal "echo" protocol named myproto that returns received data to the sender.

File: myproto.h

#ifndef MYPROTO_H
#define MYPROTO_H

#include <xquic/xquic.h>

typedef struct {
    /* per‑engine data, e.g. configuration */
    int placeholder;
} myproto_ctx_t;

/* Connection callbacks */
int myproto_conn_create(xqc_connection_t *c, const xqc_cid_t *cid,
                        void *user_data, void *proto_data);
int myproto_conn_close(xqc_connection_t *c, const xqc_cid_t *cid,
                       void *user_data, void *proto_data);

/* Stream callbacks */
int myproto_stream_read(xqc_stream_t *s, void *user_data);
int myproto_stream_write(xqc_stream_t *s, void *user_data);
int myproto_stream_create(xqc_stream_t *s, void *user_data);
int myproto_stream_close(xqc_stream_t *s, void *user_data);

/* Helper to register the protocol */
xqc_int_t myproto_register(xqc_engine_t *engine);

#endif /* MYPROTO_H */

File: myproto.c

#include "myproto.h"
#include <stdio.h>

/* ---------- Connection callbacks ---------- */
int
myproto_conn_create(xqc_connection_t *c, const xqc_cid_t *cid,
                    void *user_data, void *proto_data)
{
    /* No special per‑connection state needed for this demo */
    (void)c; (void)cid; (void)user_data; (void)proto_data;
    printf("[myproto] connection created\n");
    return 0;
}

int
myproto_conn_close(xqc_connection_t *c, const xqc_cid_t *cid,
                   void *user_data, void *proto_data)
{
    (void)c; (void)cid; (void)user_data; (void)proto_data;
    printf("[myproto] connection closed\n");
    return 0;
}

/* ---------- Stream callbacks ---------- */
int
myproto_stream_create(xqc_stream_t *s, void *user_data)
{
    (void)s; (void)user_data;
    printf("[myproto] stream created\n");
    return 0;
}

int
myproto_stream_read(xqc_stream_t *s, void *user_data)
{
    unsigned char buf[1024];
    ssize_t n = xqc_stream_recv(s, buf, sizeof(buf));
    if (n > 0) {
        /* Echo the data back */
        xqc_stream_send(s, buf, (size_t)n);
        printf("[myproto] echoed %zd bytes\n", n);
    }
    return 0;
}

int
myproto_stream_write(xqc_stream_t *s, void *user_data)
{
    /* Nothing special – write notifications are handled by the read side in this echo demo */
    (void)s; (void)user_data;
    return 0;
}

int
myproto_stream_close(xqc_stream_t *s, void *user_data)
{
    (void)s; (void)user_data;
    printf("[myproto] stream closed\n");
    return 0;
}

/* ---------- Registration helper ---------- */
xqc_int_t
myproto_register(xqc_engine_t *engine)
{
    static const xqc_app_proto_callbacks_t myproto_cbs = {
        .conn_cbs   = {
            .conn_create_notify = myproto_conn_create,
            .conn_close_notify  = myproto_conn_close,
        },
        .stream_cbs = {
            .stream_create_notify = myproto_stream_create,
            .stream_read_notify   = myproto_stream_read,
            .stream_write_notify  = myproto_stream_write,
            .stream_close_notify  = myproto_stream_close,
        },
        /* .dgram_cbs left NULL – not using datagrams */
    };

    /* Allocate a static context – could be malloc’ed if needed */
    static myproto_ctx_t ctx = { .placeholder = 0 };
    return xqc_engine_register_alpn(engine,
                                   "myproto", strlen("myproto"),
                                   &myproto_cbs,
                                   &ctx);
}

File: main.c (application entry)

#include <xquic/xquic.h>
#include "myproto.h"

int main(void)
{
    /* 1. Create engine (the usual XQUIC bootstrap) */
    xqc_engine_t *engine = xqc_engine_create(...);
    if (!engine) return -1;

    /* 2. Register the custom protocol */
    if (myproto_register(engine) != XQC_OK) {
        fprintf(stderr, "Failed to register myproto\n");
        return -1;
    }

    /* 3. Run the event loop – xqc_engine_main_logic() is called from your timer/IO
       integration (as described in the XQUIC README). */
    while (running) {
        xqc_engine_main_logic(engine);
        /* poll sockets, handle timers, etc. */
    }

    xqc_engine_destroy(engine);
    return 0;
}

Key Source Files for Reference

• src/transport/xqc_engine.c: Contains xqc_engine_register_alpn() and xqc_engine_get_alpn_ctx() implementations that manage the alpn_reg_list
• include/xquic/xquic.h: Public API declarations for engine creation, stream I/O, and callback type definitions
• src/http3/xqc_h3_ctx.c: HTTP/3 registration reference showing production callback implementations for connection and stream events
• demo/xqc_hq_ctx.c: HQ protocol demo demonstrating ALPN registration patterns and context handling for two different ALPN values

Summary

• Register custom protocols using xqc_engine_register_alpn() with a unique ALPN string and xqc_app_proto_callbacks_t structure
• Implement connection callbacks (conn_create_notify, conn_close_notify) for lifecycle events and stream callbacks (stream_read_notify, stream_write_notify, etc.) for data handling
• Store protocol state in a custom context passed during registration and retrieved via xqc_engine_get_alpn_ctx()
• Reference built-in implementations in src/http3/xqc_h3_ctx.c for production patterns and demo/xqc_hq_ctx.c for integration examples
• XQUIC automatically dispatches events to your callbacks based on the ALPN negotiated during the TLS handshake

Frequently Asked Questions

What is the maximum length for an ALPN string in XQUIC?

XQUIC enforces a maximum ALPN length defined by XQC_MAX_ALPN_LEN. Your ALPN string must be valid ASCII and fit within this limit, validated during the xqc_engine_register_alpn() call in src/transport/xqc_engine.c.

Can I register multiple custom protocols simultaneously?

Yes. The engine maintains an internal list (alpn_reg_list) that supports multiple concurrent registrations. Each protocol requires a unique ALPN string and its own callback set and context, as demonstrated in demo/xqc_hq_ctx.c which registers multiple HQ variants.

How do I unregister a protocol when shutting down?

Call xqc_engine_unregister_alpn() with your ALPN string to remove the registration and clean up associated resources. While optional, this is recommended for clean application shutdown and resource management.

Do I need to implement datagram callbacks for my custom protocol?

No. The dgram_cbs field in xqc_app_proto_callbacks_t is optional. Leave it NULL if your protocol uses only streams, as shown in the echo example above. Implement datagram callbacks only if you require unreliable message transmission via QUIC datagrams, similar to the implementation in src/http3/xqc_h3_ext_dgram.c.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}