How to Configure TLS/SSL Encryption with Custom Certificates in frp

Configure custom TLS certificates in frp by setting transport.tls.certFile and transport.tls.keyFile in both frps.toml and frpc.toml, and enable mutual TLS by specifying transport.tls.trustedCaFile on both sides to verify peer certificates.

frp (Fast Reverse Proxy) secures control-plane and data-plane connections through built-in TLS handling defined in TOML configuration files. By supplying your own certificate authority (CA) and key pairs, you replace the default auto-generated self-signed certificates with organization-trusted credentials, enforcing encryption and optional mutual authentication between the server (frps) and client (frpc).

Understanding frp's TLS Architecture

The TLS implementation resides in the transport layer and is driven by configuration structures defined in the source code. According to the frp repository, the core TLSConfig struct in pkg/config/v1/common.go (lines 87-96) provides the foundational fields used by both components:

type TLSConfig struct {
    CertFile       string `json:"certFile,omitempty"`       // path to cert
    KeyFile        string `json:"keyFile,omitempty"`        // path to private key
    TrustedCaFile  string `json:"trustedCaFile,omitempty"`  // CA for verification
    ServerName     string `json:"serverName,omitempty"`     // custom SNI name
}

The server extends this via TLSServerConfig in pkg/config/v1/server.go (lines 99-104), adding a Force boolean that rejects non-TLS connections. The client uses TLSClientConfig in pkg/config/v1/client.go (lines 161-168), which includes an Enable switch to activate TLS for the control connection.

During initialization, the server and client invoke NewServerTLSConfig and NewClientTLSConfig respectively from pkg/transport/tls.go. If certificate paths are empty, these functions automatically generate temporary self-signed pairs via newRandomTLSKeyPair; otherwise, they load your custom files from disk.

Server-Side TLS Configuration (frps.toml)

To enforce TLS on the server, populate the transport.tls section in frps.toml. Setting transport.tls.force = true ensures the server accepts only encrypted connections, rejecting any plain-text attempts.

[common]
bindPort = 7000

[transport.tls]
force = true
certFile = "/etc/frp/server.crt"
keyFile = "/etc/frp/server.key"
trustedCaFile = "/etc/frp/ca.crt"

• force: When true, the server requires TLS handshakes for all incoming control connections. As implemented in ServerTransportConfig.Complete() (pkg/config/v1/server.go), this flag automatically activates when trustedCaFile is provided.
• certFile / keyFile: Paths to the server’s PEM-encoded certificate and private key. If omitted, NewServerTLSConfig generates a random temporary pair.
• trustedCaFile: Enables mutual TLS (mTLS). When present, the server loads the CA and sets tls.RequireAndVerifyClientCert, demanding that every client present a valid certificate signed by this authority.

Client-Side TLS Configuration (frpc.toml)

The client activates TLS with the enable flag and specifies verification parameters under transport.tls in frpc.toml.

[common]
serverAddr = "frps.example.com"
serverPort = 7000

[transport.tls]
enable = true
certFile = "/etc/frp/client.crt"
keyFile = "/etc/frp/client.key"
trustedCaFile = "/etc/frp/ca.crt"
serverName = "frps.example.com"

• enable: Activates TLS for the control connection. This defaults to true in versions after v0.50.
• certFile / keyFile: Optional client certificate and key. Required when the server enforces mTLS via trustedCaFile.
• trustedCaFile: Path to the CA certificate used to verify the server’s identity. When provided, NewClientTLSConfig sets this as RootCAs and disables InsecureSkipVerify.
• serverName: Specifies the Server Name Indication (SNI) and the hostname used for certificate verification. Set this to match the Common Name (CN) or Subject Alternative Name (SAN) in the server certificate.

Generating Custom Certificates with OpenSSL

Before configuring frp, generate your own CA and signed certificates. The following commands create a self-signed CA, a server certificate, and an optional client certificate for mutual authentication:

# Generate a self-signed CA

openssl req -new -x509 -days 3650 -keyout ca.key -out ca.crt -nodes -subj "/CN=MyFRP-CA"

# Generate a server certificate signed by the CA

openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr -subj "/CN=frps.example.com"
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 3650

# Generate a client certificate (optional, for mutual auth)

openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr -subj "/CN=frpc.example.com"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 3650

Store the generated .crt and .key files on the respective hosts and reference them in your TOML configurations as shown above.

Advanced: Disabling the Custom TLS First Byte

By default, frp sends a magic byte (0x17) at the start of TLS connections to distinguish them from plain HTTP when sharing ports. If you bind vhostHTTPSPort to the same port as bindPort and encounter protocol detection issues, disable this behavior:

[transport.tls]
disableCustomTLSFirstByte = false

This setting is documented in the example configuration files conf/frps_full_example.toml and conf/frpc_full_example.toml within the repository.

Summary

• Define certificate paths in transport.tls.certFile and transport.tls.keyFile to use custom credentials instead of auto-generated ones.
• Enable server enforcement with transport.tls.force = true to reject unencrypted connections.
• Activate mutual TLS by providing transport.tls.trustedCaFile on both the server and client; this verifies peer identities in both directions.
• Set serverName on the client when the certificate’s CN differs from the DNS name used to reach the server.
• Apply to all traffic: The TLS configuration automatically encrypts both control channels and proxy data channels (TCP, UDP, HTTP/HTTPS) without additional settings.

Frequently Asked Questions

Does frp require manually generated certificates, or will it create them automatically?

frp automatically generates temporary self-signed certificates if certFile and keyFile are omitted. As seen in pkg/transport/tls.go, the function newRandomTLSKeyPair creates these on startup. However, for production deployments and mutual authentication, you should supply your own certificates signed by a trusted CA.

What is mutual TLS (mTLS) in frp, and how do I enable it?

Mutual TLS ensures both the server and client verify each other’s certificates. Enable mTLS by setting transport.tls.trustedCaFile on both frps and frpc. On the server, this triggers tls.RequireAndVerifyClientCert inside NewServerTLSConfig, requiring clients to present a valid certificate. On the client, providing the CA file ensures the server’s identity is validated against your private authority.

How do I resolve "certificate signed by unknown authority" errors?

This error occurs when the client cannot verify the server’s certificate against its trust store. Ensure transport.tls.trustedCaFile in frpc.toml points to the CA certificate that signed the server’s certificate. Additionally, verify that transport.tls.serverName matches the CN or SAN in the server certificate; mismatches cause hostname verification failures in NewClientTLSConfig.

Is TLS encryption limited to the control connection, or does it cover data traffic?

TLS applies to both control and data planes. Once the initial control connection is established using the transport.tls configuration, frp reuses these TLS settings for all subsequent proxy connections, including TCP, UDP, and HTTP/HTTPS tunnels. No separate TLS configuration is required for individual proxies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}