# How to Implement a Custom Application Protocol on Top of XQUIC: A Complete Guide

> Learn how to implement a custom application protocol on top of XQUIC by registering an ALPN string and binding callback functions for events. A complete guide for developers.

- Repository: [Alibaba/xquic](https://github.com/alibaba/xquic)
- Tags: how-to-guide
- Published: 2026-02-24

---

**You implement a custom application protocol on top of XQUIC by registering an ALPN string with the engine using `xqc_engine_register_alpn()`, which binds your protocol-specific callback functions for connection, stream, and datagram events to that identifier.**

The **alibaba/xquic** library provides a pluggable framework for building custom transport protocols over QUIC. To implement a custom application protocol on top of XQUIC, you leverage the ALPN (Application-Layer Protocol Negotiation) registration system that decouples transport handling from application logic. This approach allows you to define protocol-specific behaviors through structured callbacks while XQUIC manages the underlying QUIC connection lifecycle.

## Understanding XQUIC's ALPN Registration Framework

XQUIC uses a callback-driven architecture where protocols register themselves via the `xqc_engine_register_alpn()` function defined in [`src/transport/xqc_engine.c`](https://github.com/alibaba/xquic/blob/main/src/transport/xqc_engine.c). When a TLS handshake completes and negotiates an ALPN string, the engine looks up the registered callbacks and associated context through `xqc_engine_get_alpn_ctx()`, then dispatches events to your implementation.

The registration stores three critical components in an `xqc_alpn_registration_t` structure:

- The **ALPN string** identifying your protocol
- The **callback tables** (`xqc_app_proto_callbacks_t`) containing function pointers
- The **user-defined context** holding your protocol state

This mechanism powers the built-in HTTP/3 implementation in [`src/http3/xqc_h3_ctx.c`](https://github.com/alibaba/xquic/blob/main/src/http3/xqc_h3_ctx.c) and the HQ demo protocol in [`demo/xqc_hq_ctx.c`](https://github.com/alibaba/xquic/blob/main/demo/xqc_hq_ctx.c), providing concrete reference implementations.

## Step-by-Step Implementation Guide

### Step 1: Define Your ALPN String

Choose a unique ASCII string (e.g., `"myproto"`) not exceeding `XQC_MAX_ALPN_LEN` characters. This identifier must be distinct from standard protocols like `"h3"` or `"hq-interop"` and will be validated during the `xqc_engine_register_alpn()` call.

### Step 2: Create the Protocol Context Structure

Define a context structure to maintain per-engine state. While HTTP/3 uses complex state machines in [`src/http3/xqc_h3_ctx.c`](https://github.com/alibaba/xquic/blob/main/src/http3/xqc_h3_ctx.c), a simple protocol might only need configuration flags or connection counters.

### Step 3: Implement Connection Callbacks

Define functions matching the signatures required by `xqc_app_proto_callbacks_t` to handle connection lifecycle events. You must implement **connection creation** and **connection close** callbacks:

- `conn_create_notify`: Invoked when a new `xqc_connection_t` is established
- `conn_close_notify`: Invoked when the connection terminates

### Step 4: Implement Stream Callbacks

Implement the four required stream callbacks that operate on individual `xqc_stream_t` handles:

- `stream_create_notify`: Called when a peer opens a new stream
- `stream_read_notify`: Triggered when data is available; use `xqc_stream_recv()` to read
- `stream_write_notify`: Indicates the stream is ready for transmission; use `xqc_stream_send()` to write
- `stream_close_notify`: Called when the stream closes

### Step 5: Populate the Callbacks Structure

Fill an `xqc_app_proto_callbacks_t` instance with pointers to your functions. Set the `conn_cbs` and `stream_cbs` members. Leave `dgram_cbs` NULL unless implementing QUIC datagram support as shown in [`src/http3/xqc_h3_ext_dgram.c`](https://github.com/alibaba/xquic/blob/main/src/http3/xqc_h3_ext_dgram.c).

### Step 6: Register with the Engine

Call `xqc_engine_register_alpn()` during initialization, passing your ALPN string, the callbacks structure, and your context pointer. The engine stores this registration in its internal `alpn_reg_list` for runtime dispatch.

## Complete Working Example: Building an Echo Protocol

The following example implements a minimal "echo" protocol named `myproto` that returns received data to the sender.

**File:** [`myproto.h`](https://github.com/alibaba/xquic/blob/main/myproto.h)

```c
#ifndef MYPROTO_H
#define MYPROTO_H

#include <xquic/xquic.h>

typedef struct {
    /* per‑engine data, e.g. configuration */
    int placeholder;
} myproto_ctx_t;

/* Connection callbacks */
int myproto_conn_create(xqc_connection_t *c, const xqc_cid_t *cid,
                        void *user_data, void *proto_data);
int myproto_conn_close(xqc_connection_t *c, const xqc_cid_t *cid,
                       void *user_data, void *proto_data);

/* Stream callbacks */
int myproto_stream_read(xqc_stream_t *s, void *user_data);
int myproto_stream_write(xqc_stream_t *s, void *user_data);
int myproto_stream_create(xqc_stream_t *s, void *user_data);
int myproto_stream_close(xqc_stream_t *s, void *user_data);

/* Helper to register the protocol */
xqc_int_t myproto_register(xqc_engine_t *engine);

#endif /* MYPROTO_H */

```

**File:** [`myproto.c`](https://github.com/alibaba/xquic/blob/main/myproto.c)

```c
#include "myproto.h"
#include <stdio.h>

/* ---------- Connection callbacks ---------- */
int
myproto_conn_create(xqc_connection_t *c, const xqc_cid_t *cid,
                    void *user_data, void *proto_data)
{
    /* No special per‑connection state needed for this demo */
    (void)c; (void)cid; (void)user_data; (void)proto_data;
    printf("[myproto] connection created\n");
    return 0;
}

int
myproto_conn_close(xqc_connection_t *c, const xqc_cid_t *cid,
                   void *user_data, void *proto_data)
{
    (void)c; (void)cid; (void)user_data; (void)proto_data;
    printf("[myproto] connection closed\n");
    return 0;
}

/* ---------- Stream callbacks ---------- */
int
myproto_stream_create(xqc_stream_t *s, void *user_data)
{
    (void)s; (void)user_data;
    printf("[myproto] stream created\n");
    return 0;
}

int
myproto_stream_read(xqc_stream_t *s, void *user_data)
{
    unsigned char buf[1024];
    ssize_t n = xqc_stream_recv(s, buf, sizeof(buf));
    if (n > 0) {
        /* Echo the data back */
        xqc_stream_send(s, buf, (size_t)n);
        printf("[myproto] echoed %zd bytes\n", n);
    }
    return 0;
}

int
myproto_stream_write(xqc_stream_t *s, void *user_data)
{
    /* Nothing special – write notifications are handled by the read side in this echo demo */
    (void)s; (void)user_data;
    return 0;
}

int
myproto_stream_close(xqc_stream_t *s, void *user_data)
{
    (void)s; (void)user_data;
    printf("[myproto] stream closed\n");
    return 0;
}

/* ---------- Registration helper ---------- */
xqc_int_t
myproto_register(xqc_engine_t *engine)
{
    static const xqc_app_proto_callbacks_t myproto_cbs = {
        .conn_cbs   = {
            .conn_create_notify = myproto_conn_create,
            .conn_close_notify  = myproto_conn_close,
        },
        .stream_cbs = {
            .stream_create_notify = myproto_stream_create,
            .stream_read_notify   = myproto_stream_read,
            .stream_write_notify  = myproto_stream_write,
            .stream_close_notify  = myproto_stream_close,
        },
        /* .dgram_cbs left NULL – not using datagrams */
    };

    /* Allocate a static context – could be malloc’ed if needed */
    static myproto_ctx_t ctx = { .placeholder = 0 };
    return xqc_engine_register_alpn(engine,
                                   "myproto", strlen("myproto"),
                                   &myproto_cbs,
                                   &ctx);
}

```

**File:** [`main.c`](https://github.com/alibaba/xquic/blob/main/main.c) (application entry)

```c
#include <xquic/xquic.h>
#include "myproto.h"

int main(void)
{
    /* 1. Create engine (the usual XQUIC bootstrap) */
    xqc_engine_t *engine = xqc_engine_create(...);
    if (!engine) return -1;

    /* 2. Register the custom protocol */
    if (myproto_register(engine) != XQC_OK) {
        fprintf(stderr, "Failed to register myproto\n");
        return -1;
    }

    /* 3. Run the event loop – xqc_engine_main_logic() is called from your timer/IO
       integration (as described in the XQUIC README). */
    while (running) {
        xqc_engine_main_logic(engine);
        /* poll sockets, handle timers, etc. */
    }

    xqc_engine_destroy(engine);
    return 0;
}

```

## Key Source Files for Reference

- [`src/transport/xqc_engine.c`](https://github.com/alibaba/xquic/blob/main/src/transport/xqc_engine.c): Contains `xqc_engine_register_alpn()` and `xqc_engine_get_alpn_ctx()` implementations that manage the `alpn_reg_list`
- [`include/xquic/xquic.h`](https://github.com/alibaba/xquic/blob/main/include/xquic/xquic.h): Public API declarations for engine creation, stream I/O, and callback type definitions
- [`src/http3/xqc_h3_ctx.c`](https://github.com/alibaba/xquic/blob/main/src/http3/xqc_h3_ctx.c): HTTP/3 registration reference showing production callback implementations for connection and stream events
- [`demo/xqc_hq_ctx.c`](https://github.com/alibaba/xquic/blob/main/demo/xqc_hq_ctx.c): HQ protocol demo demonstrating ALPN registration patterns and context handling for two different ALPN values

## Summary

- Register custom protocols using `xqc_engine_register_alpn()` with a unique ALPN string and `xqc_app_proto_callbacks_t` structure
- Implement connection callbacks (`conn_create_notify`, `conn_close_notify`) for lifecycle events and stream callbacks (`stream_read_notify`, `stream_write_notify`, etc.) for data handling
- Store protocol state in a custom context passed during registration and retrieved via `xqc_engine_get_alpn_ctx()`
- Reference built-in implementations in [`src/http3/xqc_h3_ctx.c`](https://github.com/alibaba/xquic/blob/main/src/http3/xqc_h3_ctx.c) for production patterns and [`demo/xqc_hq_ctx.c`](https://github.com/alibaba/xquic/blob/main/demo/xqc_hq_ctx.c) for integration examples
- XQUIC automatically dispatches events to your callbacks based on the ALPN negotiated during the TLS handshake

## Frequently Asked Questions

### What is the maximum length for an ALPN string in XQUIC?

XQUIC enforces a maximum ALPN length defined by `XQC_MAX_ALPN_LEN`. Your ALPN string must be valid ASCII and fit within this limit, validated during the `xqc_engine_register_alpn()` call in [`src/transport/xqc_engine.c`](https://github.com/alibaba/xquic/blob/main/src/transport/xqc_engine.c).

### Can I register multiple custom protocols simultaneously?

Yes. The engine maintains an internal list (`alpn_reg_list`) that supports multiple concurrent registrations. Each protocol requires a unique ALPN string and its own callback set and context, as demonstrated in [`demo/xqc_hq_ctx.c`](https://github.com/alibaba/xquic/blob/main/demo/xqc_hq_ctx.c) which registers multiple HQ variants.

### How do I unregister a protocol when shutting down?

Call `xqc_engine_unregister_alpn()` with your ALPN string to remove the registration and clean up associated resources. While optional, this is recommended for clean application shutdown and resource management.

### Do I need to implement datagram callbacks for my custom protocol?

No. The `dgram_cbs` field in `xqc_app_proto_callbacks_t` is optional. Leave it NULL if your protocol uses only streams, as shown in the echo example above. Implement datagram callbacks only if you require unreliable message transmission via QUIC datagrams, similar to the implementation in [`src/http3/xqc_h3_ext_dgram.c`](https://github.com/alibaba/xquic/blob/main/src/http3/xqc_h3_ext_dgram.c).