Security Implications of Different Proxy Protocols in Xray-core: VMess, VLESS, Trojan, and Shadowsocks Explained

Xray-core implements distinct security models across its proxy protocols—VMess provides AES-GCM encryption but lacks forward secrecy, VLESS offers optional post-quantum ML-KEM-X25519 key exchange for forward secrecy, Trojan inherits TLS security with static password authentication, Shadowsocks relies on AEAD ciphers with per-packet integrity, while HTTP/SOCKS and the PROXY protocol introduce minimal or spoofable security guarantees that require careful transport-layer hardening.

Xray-core from the XTLS project is a modular, protocol-agnostic proxy engine deployed for circumventing network censorship and securing traffic flows. Understanding the security implications of different proxy protocols in Xray-core is essential for threat modeling, as each protocol implements confidentiality, integrity, authentication, and traffic-analysis resistance through distinct cryptographic primitives and authentication mechanisms defined in specific source files throughout the repository.

VMess Protocol Security Analysis

VMess is the original Xray protocol that encrypts traffic using AES-128/256-GCM with per-connection encryption keys derived from a shared secret. In proxy/vmess/encoding/server.go, the server implements AEAD length encryption using responseBodyKey and responseBodyIV parameters generated per session【proxy/vmess/encoding/server.go†L335-L363】.

Authentication relies on a 16-byte UserID (UUID) that must exist in the inbound user list defined in proxy/vmess/inbound/config.pb.go. The protocol includes replay protection through SessionID validation against per-user counters, and optional random padding (paddingValue in proxy/vmess/outbound/outbound.go) to obscure fixed-size headers.

A critical limitation is the lack of forward secrecy. Since VMess depends on a symmetric secret, compromise of the UUID and keys allows decryption of all historical traffic. The proprietary nature of the protocol also limits formal cryptographic analysis.

VLESS Protocol Security Considerations

VLESS functions as a lightweight successor with configurable decryption methods. The configuration parsing in infra/conf/vless.go (lines 94-138) supports decryption values of "none" or "mlkem768x25519plus"【infra/conf/vless.go†L94-L138】.

When decryption is set to "none", the protocol provides no confidentiality and must be paired with a TLS-based transport. For post-quantum security, the ML-KEM-768 + X25519 hybrid mode generates ephemeral key pairs per connection, delivering forward secrecy even if the server private key is compromised. The outbound logic in proxy/vless/outbound/outbound.go handles Xor-based encryption modes and optional XOR-Pub masks based on the Encryption field.

VLESS validates PROXY protocol versions in infra/conf/vless.go lines 196-199, rejecting xver values greater than 2 to prevent malformed header attacks. However, when operating in stream mode without TLS, traffic remains vulnerable to passive eavesdropping.

Trojan Protocol Security Model

Trojan encapsulates all traffic within TLS (or REALITY), with the TLS configuration built from transport/internet/tls and applied in the listener at transport/internet/tcp/hub.go. Authentication occurs through a pre-shared password sent as the first HTTP header after the TLS handshake, validated in proxy/trojan/client.go against the server configuration in infra/conf/trojan.go.

The protocol supports modern cipher suites, ALPN negotiation, and Reality obfuscation. However, Trojan relies on a static password; compromise of this secret allows impersonation. Forward secrecy depends entirely on the underlying TLS configuration using ECDHE cipher suites rather than protocol-native mechanisms.

Shadowsocks AEAD Implementation

Shadowsocks in Xray-core implements AEAD ciphers including AES-GCM and ChaCha20-Poly1305. The NewEncryptionWriter method in proxy/shadowsocks/config.go constructs these cipher instances, while key derivation from user passwords uses PBKDF2 as implemented in proxy/shadowsocks/validator.go.

Security characteristics include implicit authentication through AEAD tags—decryption fails with incorrect keys. However, the protocol provides only per-packet integrity rather than full stream integrity, and packet lengths remain visible unless the user explicitly enables config.Padding padding options.

HTTP and SOCKS Plain-Text Risks

The HTTP (proxy/http/server.go) and SOCKS (proxy/socks/server.go) protocols provide no encryption at the proxy layer. Authentication is optional, implemented through username/password combinations in proxy/socks/config.go via the Account.Equals method.

Without an encrypted transport such as TLS, WebSocket, or Reality, credentials and payload traverse the network in clear text. These protocols should never be exposed directly to untrusted networks without underlying transport encryption.

PROXY Protocol Security Implications

The HAProxy PROXY protocol preserves original client IP addresses through header injection. Xray-core processes this in transport/internet/tcp/hub.go (lines 71-73) for inbound acceptance and injects headers in proxy/freedom/freedom.go (lines 214-221) for outbound forwarding.

While useful for logging and access control, the header is spoofable by any client reaching the listener directly. Malformed headers are rejected through version validation (enforcing xver ≤ 2), but exposed listeners accepting PROXY protocol can facilitate IP address spoofing to bypass ACLs. Best practice requires placing a trusted edge proxy (HAProxy or Nginx) before Xray and enabling acceptProxyProtocol only on trusted internal paths.

Security Trade-offs Comparison

The following summarizes the security properties across Xray-core protocols:

• VMess: Strong confidentiality (AES-GCM) and integrity via AEAD, UUID authentication, good traffic-analysis resistance with padding, but no forward secrecy due to static symmetric keys.
• VLESS (none): No confidentiality or integrity without underlying TLS transport; UUID authentication only.
• VLESS (ML-KEM): Post-quantum confidentiality and integrity via ML-KEM-768+X25519 hybrid, forward secrecy through ephemeral keys, and optional padding.
• Trojan: TLS-wrapped confidentiality and integrity, password authentication, forward secrecy dependent on TLS ECDHE configuration.
• Shadowsocks: AEAD confidentiality and integrity, pre-shared key authentication, moderate traffic-analysis resistance with optional padding.
• HTTP/SOCKS: No native encryption or integrity; vulnerable to eavesdropping without TLS transport.
• PROXY Protocol: Metadata carrier only; introduces spoofing risks if exposed to untrusted clients.

Code Configuration Examples

VMess Inbound with AEAD Encryption

{
  "protocol": "vmess",
  "settings": {
    "clients": [
      {
        "id": "6d9d2e5b-4c1a-4a6c-9a2b-1f7c1b2e3c4d",
        "alterId": 64
      }
    ]
  }
}

Source: infra/conf/vmess.go builds the proxy/vmess/inbound.Config.

VLESS with Post-Quantum Decryption

{
  "protocol": "vless",
  "settings": {
    "vnext": [
      {
        "address": "example.com",
        "port": 443,
        "users": [
          {
            "id": "c0ffee00-1234-5678-9abc-def012345678",
            "encryption": "mlkem768x25519plus.native.xorpub-30s-none",
            "flow": "xtls-rprx-vision"
          }
        ]
      }
    ]
  },
  "streamSettings": {
    "network": "tls"
  }
}

Source: Parsing logic in infra/conf/vless.go (lines 94-138) and outbound implementation in proxy/vless/outbound/outbound.go.

Trojan with TLS and Password Authentication

{
  "protocol": "trojan",
  "settings": {
    "clients": [
      { "password": "s3cr3tP@ssw0rd" }
    ]
  },
  "streamSettings": {
    "network": "tls",
    "security": "tls",
    "tlsSettings": {
      "alpn": ["http/1.1"]
    }
  }
}

Source: infra/conf/trojan.go validates passwords and constructs the inbound configuration.

Shadowsocks with AEAD-AES-128-GCM

{
  "protocol": "shadowsocks",
  "settings": {
    "method": "aes-128-gcm",
    "password": "myStrongPwd"
  }
}

Source: Cipher instantiation occurs in proxy/shadowsocks/config.go via NewEncryptionWriter.

Freedom Outbound with PROXY Protocol

{
  "protocol": "freedom",
  "settings": {
    "domainStrategy": "use_ip",
    "proxyProtocol": 1
  }
}

Source: Header construction in proxy/freedom/freedom.go lines 214-221 using proxyproto.HeaderProxyFromAddrs.

Summary

• VMess offers robust encryption through AES-GCM but relies on static keys, making historical traffic vulnerable to key compromise.
• VLESS provides the strongest security when configured with ML-KEM-768+X25519 hybrid encryption, delivering post-quantum forward secrecy.
• Trojan security depends entirely on TLS configuration and password secrecy, lacking protocol-native forward secrecy.
• Shadowsocks implements modern AEAD ciphers but requires explicit padding to resist traffic analysis.
• HTTP and SOCKS protocols transmit data in clear text unless wrapped in TLS, Reality, or other encrypted transports.
• The PROXY protocol exposes IP spoofing attack surfaces when listeners are directly internet-facing; deploy only behind trusted edge proxies with acceptProxyProtocol enabled selectively.

Frequently Asked Questions

Does VMess provide forward secrecy?

No. VMess derives encryption keys (responseBodyKey and responseBodyIV) from a static UUID shared between client and server. If this secret is compromised, an attacker can decrypt all past and future traffic captured from the network. The protocol lacks ephemeral key exchange mechanisms found in modern handshake protocols.

Is VLESS without encryption secure?

VLESS configured with "decryption": "none" provides no confidentiality, integrity, or authentication at the protocol layer. Security depends entirely on the underlying transport—typically TLS or Reality. Without these, usernames, passwords, and payload data transmit in clear text, making the configuration vulnerable to passive eavesdropping and active manipulation.

What are the risks of enabling the PROXY protocol in Xray-core?

The PROXY protocol (HAProxy header) can be spoofed if the Xray listener accepts direct connections from untrusted clients. An attacker can inject arbitrary source IP addresses to bypass IP-based access controls or pollute logging data. Xray-core mitigates malformed header attacks by validating version fields (rejecting xver > 2) in transport/internet/tcp/hub.go, but you should only enable acceptProxyProtocol behind a trusted edge proxy like HAProxy or Nginx that validates client addresses.

Which Xray-core protocol offers post-quantum security?

VLESS with the mlkem768x25519plus encryption setting provides post-quantum confidentiality through a hybrid of ML-KEM-768 (Kyber) and X25519 key exchange. This configuration, parsed in infra/conf/vless.go and implemented in proxy/vless/outbound/outbound.go, generates ephemeral keys per connection, achieving forward secrecy resistant to both classical and quantum computational attacks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}