How to Use Full Query Logging (FQL) for Debugging and Replay in Apache Cassandra

Full Query Logging (FQL) captures every CQL statement—including bind values, timestamps, and protocol metadata—to a binary Chronicle log that you can inspect with fqltool or replay against any Cassandra cluster for debugging and load testing.

Full Query Logging is a built-in diagnostic feature of Apache Cassandra that records complete query information after execution finishes or times out. Unlike query tracing, which samples specific requests, FQL provides a lossless binary record of all traffic, making it ideal for auditing, troubleshooting production issues, and performing realistic workload replays during migration testing.

How Full Query Logging Works in Cassandra

The FQL architecture consists of a high-throughput logger that plugs into Cassandra's query event pipeline and writes to a lock-free binary format. All records are stored using Chronicle Wire, enabling fast sequential writes and deterministic replay.

Core Components

The implementation centers on three primary classes:

• FullQueryLogger – Located in src/java/org/apache/cassandra/fql/FullQueryLogger.java, this class implements QueryEvents.Listener and acts as the central dispatcher. It extracts metadata from each query event and writes structured records to a BinLog instance.
• FullQueryLoggerOptions – Defined in src/java/org/apache/cassandra/fql/FullQueryLoggerOptions.java, this POJO extends BinLogOptions and exposes configuration parameters via JMX and cassandra.yaml.
• BinLog – Found in org.apache.cassandra.utils.binlog, this low-level component provides the lock-free binary log used for persistence. FullQueryLogger initializes it via new BinLog.Builder() in its constructor.

Logged Data Structure

When FullQueryLogger handles a query event, it serializes the following fields into the binary log:

• VERSION – Log format version for compatibility checks.
• TYPE – Either SINGLE_QUERY or BATCH.
• PROTOCOL_VERSION – Native protocol version used by the client.
• QUERY_OPTIONS – Query flags and options.
• QUERY_START_TIME – Nanosecond-precision start timestamp.
• GENERATED_TIMESTAMP and GENERATED_NOW_IN_SECONDS – Server-side timestamps.
• KEYSPACE – Target keyspace context.
• QUERY – The raw CQL string.
• VALUES – Serialized bound parameters.

During replay, fqltool uses FQLQueryReader (in tools/fqltool/src/org/apache/cassandra/fqltool/FQLQueryReader.java) to validate the version via verifyVersion() and reconstruct either FQLQuery.Single or FQLQuery.Batch objects based on the readType() discriminator.

Enabling Full Query Logging

You can activate FQL dynamically without restarting the node, or configure it persistently for always-on auditing.

Method 1: Dynamic Activation via JMX

For ad-hoc debugging, enable the logger via JMX using FullQueryLogger.instance.enable(). This method registers the listener with QueryEvents.instance immediately:

import java.nio.file.Paths;
import org.apache.cassandra.fql.FullQueryLogger;

// Enable FQL via JMX or embedded code
FullQueryLogger.instance.enable(
    Paths.get("/var/log/cassandra/fql"), // log directory
    "daily",                               // roll cycle (Chronicle format)
    true,                                  // block: true ensures durability
    1000,                                  // max_queue_weight (backpressure)
    10L * 1024 * 1024,                     // max_log_size: 10 MiB per file
    "",                                    // optional archive command
    3                                      // max archive retries
);

Parameters follow Chronicle naming conventions: roll_cycle accepts values like daily, hourly, or minutely, and block determines whether writes wait for disk space or drop records under pressure.

Method 2: Persistent Configuration in cassandra.yaml

To enable FQL automatically on node startup, add the full_query_log_options block to cassandra.yaml:

full_query_log_options:
  enabled: true
  path: /var/log/cassandra/fql
  roll_cycle: daily
  block: true
  max_queue_weight: 1000
  max_log_size: 10485760  # 10 MiB

When the node initializes, Cassandra instantiates FullQueryLogger with these values via enableWithoutClean(), ensuring logging begins before client traffic arrives.

Disabling and Resetting the Log

To stop logging without restarting:

FullQueryLogger.instance.stop();  // Deregisters the listener and closes files

To clear existing log files after a debugging session, use:

FullQueryLogger.instance.reset("/var/log/cassandra/fql");

This deletes the Chronicle queue files for the specified path, preventing replay of stale data.

Replaying and Inspecting FQL Logs with fqltool

The fqltool utility (located in the tools/fqltool directory) reads the binary Chronicle logs and provides two primary modes: human-readable inspection and live cluster replay.

Printing Logs in Human-Readable Format

To inspect captured queries without executing them:

./tools/fqltool/fqltool --print /var/log/cassandra/fql/2023-03-15

This iterates through the log using FQLQueryIterator (from tools/fqltool/src/org/apache/cassandra/fqltool/FQLQueryIterator.java) and outputs formatted text showing timestamps, keyspaces, CQL strings, and hex-encoded bound values:

[2023-03-15 12:34:56.789] keyspace=users type=single-query protocol=4
  CQL: INSERT INTO users (id, name) VALUES (?, ?)
  VALUES: [0x00000001, 0x6a6f686e]

Executing Replay Against a Target Cluster

To replay logged traffic against a different cluster or version for regression testing:

./tools/fqltool/fqltool --replay /var/log/cassandra/fql/2023-03-15 \
                        --hosts 10.0.1.12,10.0.1.13 \
                        --port 9042

The tool reconstructs each FQLQuery object, preserving the original protocol version and bound parameters. For single queries, it creates simple statements; for batches, it builds BatchStatement objects with the recorded sub-queries. This ensures the replay matches the original workload characteristics, including timestamp behavior and consistency levels.

Programmatic Integration Examples

Enable FQL from Application Code

If running embedded Cassandra or managing nodes via a custom agent:

import java.nio.file.Paths;
import org.apache.cassandra.fql.FullQueryLogger;

public class EnableFQL {
    public static void main(String[] args) {
        FullQueryLogger.instance.enable(
            Paths.get("/tmp/cassandra-fql"),
            "daily",
            true,
            500,                 // moderate memory usage
            50L * 1024 * 1024,   // 50 MiB per file
            "",                  // no external archiving
            2                    // retry archiving twice
        );
        System.out.println("Full Query Logging enabled");
    }
}

Disable FQL via Jolokia HTTP API

When JMX is exposed via HTTP using Jolokia:

curl -X POST http://localhost:8778/jolokia/ \
     -d '{"type":"exec","mbean":"org.apache.cassandra.db:type=FullQueryLogger","operation":"stop"}'

Summary

• Full Query Logging in Apache Cassandra captures complete CQL statements, bind values, and metadata to a binary Chronicle log after query execution.
• Enable FQL dynamically via FullQueryLogger.instance.enable() in src/java/org/apache/cassandra/fql/FullQueryLogger.java, or persistently through the full_query_log_options section in cassandra.yaml.
• The binary format stores fields like PROTOCOL_VERSION, QUERY, VALUES, and GENERATED_TIMESTAMP, written using Chronicle Wire for high throughput.
• Use fqltool with the --print flag to inspect logs, or --replay to execute queries against a target cluster using the recorded protocol version and parameters.
• Disable logging with FullQueryLogger.instance.stop() and clean up old logs using reset(String path).

Frequently Asked Questions

What is the performance impact of enabling Full Query Logging?

Full Query Logging adds minimal latency because it writes asynchronously to a lock-free Chronicle queue, but it does increase disk I/O and heap pressure based on the max_queue_weight and max_log_size settings. According to the FullQueryLogger implementation, blocking mode (block: true) ensures durability but may slow down the request path if the disk saturates, while non-blocking mode drops records under pressure.

Can I replay FQL logs to a different Cassandra version?

Yes, fqltool validates the log format version via FQLQueryReader.verifyVersion() before replaying. As long as the target cluster supports the native protocol version recorded in the log (stored in the PROTOCOL_VERSION field), the replay will succeed. However, schema differences between the source and target clusters will cause individual statements to fail during replay.

Does Full Query Logging capture failed or timed-out queries?

Yes, FQL records queries after they finish or time out, capturing the complete statement and parameters regardless of success or failure. The listener registered in FullQueryLogger handles both normal completion and timeout events from QueryEvents, ensuring you have a complete audit trail of attempted operations.

Where are the FQL log files stored and how are they rotated?

Logs are stored in the path specified by the path parameter, with files rotated according to the roll_cycle setting (e.g., daily, hourly). The max_log_size parameter controls the maximum size of individual files. Once a file reaches this limit or the roll cycle triggers, Chronicle creates a new segment. You can specify an optional archive command in FullQueryLoggerOptions to compress or move old segments automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}