How openpilot Implements Robust Logging and Telemetry Collection with loggerd

openpilot's loggerd daemon captures all vehicle telemetry through a single poll loop that writes CAN, sensor, and camera data to rotating qlog segments while synchronizing video encoders and handling graceful shutdowns even during power loss.

The loggerd service in the commaai/openpilot repository serves as the central persistence layer for autonomous driving data, converting high-frequency ZMQ messages into structured route files. This C++ daemon manages the complex orchestration between sensor streams, video encoders, and user events to ensure complete data capture across every drive segment.

High-Throughput Message Ingestion Architecture

loggerd implements a thread-safe ingestion pipeline through a single poll loop that subscribes to all vehicle services. In system/loggerd/loggerd.cc, the loggerd_thread() function creates a SubSocket for every service marked with should_log in the services array defined in system/loggerd/loggerd.h.

The poll loop caps processing at 200 messages per socket per iteration to prevent lock-ups during burst conditions. Each message is either written directly to the qlog via s.logger.write() or routed to handle_encoder_msg() for video processing. This design ensures that CAN bus data, IMU readings, and GPS coordinates are persisted with minimal latency while maintaining system responsiveness.

Segment Rotation and File Management

Route recording is divided into fixed-length segments managed by the Logger class. Rotation occurs when all encoders signal completion of the current segment or when the SEGMENT_LENGTH timeout expires due to missing camera packets.

The rotation logic resides in rotate_if_needed(), which invokes logger_rotate() to call Logger::next(). This atomic operation creates a new qlog file and synchronizes the segment index across all writers. The system uses last_camera_seen_tms timestamps to ensure rotation only happens after valid video feed detection, preventing empty segments during camera initialization failures.

Video Encoding and Camera Synchronization

Each camera stream is handled by a RemoteEncoder instance that owns a VideoWriter. When handle_encoder_msg() receives an EncodeIndex, it validates the segment number against the current logger state and creates a new VideoWriter instance if a segment boundary is crossed.

The first I-frame triggers file creation in VideoWriter::write(), with subsequent frames written through the same interface. Encoder implementations vary by platform:

• v4l_encoder.cc – V4L2 hardware encoding for front/rear cameras
• jpeg_encoder.cc – JPEG fallback for low-bandwidth scenarios
• ffmpeg_encoder.cc – High-quality software encoding when hardware acceleration is unavailable

These files reside in system/loggerd/encoder/ and handle platform-specific frame processing before passing raw bytes to the video writer.

Audio Synchronization and Multiplexing

When the RecordAudio parameter is enabled, loggerd captures rawAudioData streams and multiplexes them into every video file marked with include_audio. The audio injection occurs in handle_preserve_segment() blocks where decoded audio packets are written directly after video frames to maintain lip-sync accuracy across the route recording.

User-Triggered Segment Preservation

Critical segments can be preserved beyond normal retention policies through the handle_preserve_segment() function. When messages like userBookmark or audioFeedback arrive, loggerd sets an extended attribute (PRESERVE_ATTR_NAME) on the segment's directory and updates the AthenadRecentlyViewedRoutes parameter.

External tools and the openpilot UI interact with this system through the Params class:

// Trigger preservation from user interface
Params params;
std::string routes = params.get("AthenadRecentlyViewedRoutes");
params.put("AthenadRecentlyViewedRoutes", routes + ",<segment_name>");

This mechanism ensures user-marked events remain available for later analysis and cloud upload regardless of storage management policies.

Configuration and Health Telemetry

Runtime behavior is controlled through the Params key-value store in common/params.h and common/params.cc. Key flags include:

• RecordAudio – Enables microphone stream logging
• CurrentRoute – Active route identifier
• AthenadRecentlyViewedRoutes – Preservation whitelist

loggerd emits periodic LOGD statements tracking message rates, byte throughput, and rotation events. These metrics are themselves written to the qlog, creating a self-referential audit trail of logging performance and system health.

Graceful Shutdown and Data Integrity

The ExitHandler (do_exit) coordinates clean termination across the logging pipeline. When shutdown is requested, the poll loop exits, flushes the current Logger instance, and closes all video writers. If a power failure is detected during this process, loggerd forces a sync() system call to guarantee that the latest segment reaches persistent storage before hardware power-down.

Code Examples

Subscribing to a New Service

Add entries to the services array in system/loggerd/loggerd.h to enable automatic logging:

ServiceInfo {
  const char *name = "roadEncodeData";   // ZMQ topic name
  bool should_log = true;                // Enable qlog persistence
  int decimation = 1;                    // Log every Nth message
  int queue_size = 1000;                 // Socket buffer depth
};

loggerd_thread() automatically creates the SubSocket and begins persistence without modifying the core loop logic.

Writing Video Frames

Encoder implementations write synchronized frames through the video writer interface:

// Inside handle_encoder_msg() after writer initialization
re.writer->write(
    (uint8_t*)frame_data.begin(),    // Raw encoded frame bytes
    frame_data.size(),
    timestamp_ms,                    // Presentation timestamp
    false,                           // Keyframe flag
    false                            // Forced I-frame flag
);

This method in system/loggerd/video_writer.cc handles platform-specific container formats while maintaining timestamp monotonicity.

Main Daemon Entry Point

The system manager launches loggerd through the standard main function:

int main(int argc, char **argv) {
  // Optimize CPU affinity on embedded hardware
  if (!Hardware::PC()) {
    util::set_core_affinity({0, 1, 2, 3});
  }
  
  // Enter blocking logging loop
  loggerd_thread();
  return 0;
}

Located in system/loggerd/loggerd.cc, this entry point configures real-time scheduling priorities before entering the message processing loop.

Summary

• Single-threaded poll architecture in loggerd_thread() handles all ZMQ subscriptions with a 200-message processing cap per iteration to prevent blocking.
• Atomic segment rotation via logger_rotate() and Logger::next() ensures qlog and video files remain synchronized across segment boundaries.
• Per-camera encoder abstraction through RemoteEncoder and VideoWriter supports V4L2, JPEG, and FFmpeg backends in system/loggerd/encoder/.
• Audio multiplexing injects rawAudioData into video containers when RecordAudio is enabled via the Params system.
• Segment preservation uses extended attributes and AthenadRecentlyViewedRoutes to protect user-marked segments from deletion.
• Power-fail safety is achieved through explicit sync() calls in the ExitHandler, guaranteeing data durability during abrupt shutdowns.

Frequently Asked Questions

How does loggerd handle high-frequency CAN bus data without dropping messages?

loggerd employs a non-blocking poll loop that processes up to 200 messages per socket per iteration, as implemented in system/loggerd/loggerd.cc. The SubSocket interfaces utilize ZMQ queueing with configurable buffer sizes defined in the ServiceInfo structure, allowing the system to absorb burst traffic while maintaining real-time write performance to disk.

What triggers a segment rotation in openpilot's logging system?

Segment rotation occurs when all video encoders report completion of the current segment or when the SEGMENT_LENGTH timeout expires. The rotate_if_needed() function checks encoder readiness flags and camera packet timestamps (last_camera_seen_tms) to ensure rotation only happens after valid video data is captured, preventing empty or corrupted segments.

How can developers preserve specific route segments for later analysis?

Developers can trigger preservation by sending userBookmark or audioFeedback messages, which cause handle_preserve_segment() to set the PRESERVE_ATTR_NAME extended attribute on the segment directory. Alternatively, writing to the AthenadRecentlyViewedRoutes parameter through the Params API marks segments for extended retention in the upload queue.

Where is audio data stored when RecordAudio is enabled?

When the RecordAudio parameter is active, audio samples from rawAudioData are multiplexed into every video file that requests audio inclusion (include_audio flag). The VideoWriter class in system/loggerd/video_writer.cc interleaves these packets immediately after video frames to maintain synchronization, storing them within the segment's video container rather than separate files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}