How the OpenPilot Persistent Params System Stores Configuration Across Reboots

The OpenPilot persistent params system stores configuration as individual files under /data/params/d, uses atomic write operations with fsync guarantees to prevent corruption, and distinguishes persistent from volatile keys through compile-time ParamKeyFlag annotations that control automatic cleanup during system lifecycle events.

The persistent params system in the commaai/openpilot repository provides a lightweight, crash-resistant key-value store designed for automotive environments where power loss can occur unexpectedly. This subsystem manages calibration data, user preferences, and temporary state using a file-based architecture that guarantees data survives reboots unless explicitly marked for deletion.

Architecture of the Persistent Params System

The implementation spans multiple layers, from Python wrappers to low-level C++ file operations:

• common/params_pyx.pyx: Python-level wrapper exposing the Params class, flag enums, and type-conversion helpers.
• common/params.cc: C++ implementation handling atomic file writes, directory locking, and flag-based cleanup.
• common/params.h: Declarations of the C++ API and the ParamKeyFlag enum.
• system/manager/manager.py: High-level driver that orchestrates cleanup of non-persistent keys during state transitions.
• common/tests/test_params.py: Unit tests verifying persistence semantics.

Each parameter key is defined in the generated header common/params_keys.h with specific flags that determine its lifetime behavior.

Persistent vs. Volatile Keys

The system distinguishes between settings that must survive indefinitely and those that should reset during specific events.

Compile-Time Flag Definitions

In common/params_pyx.pyx (lines 14-21), the ParamKeyFlag enum defines persistence behavior:

• PERSISTENT: The key is never deleted automatically and survives all reboots.
• CLEAR_ON_MANAGER_START: Removed when the system manager initializes.
• CLEAR_ON_ONROAD_TRANSITION: Cleared when transitioning to the on-road driving state.
• CLEAR_ON_OFFROAD_TRANSITION: Cleared when coming off-road.
• CLEAR_ON_IGNITION_ON: Removed when ignition powers on.

Keys are declared in common/params_keys.h with their respective flags set at build time. Only keys lacking the PERSISTENT flag participate in automatic cleanup operations.

Atomic Write Mechanism for Crash Safety

When Python code calls params.put(key, value), the wrapper in common/params_pyx.pyx validates the key and casts the value before invoking the C++ implementation in common/params.cc (lines 30-67).

The C++ layer implements atomic writes through a specific sequence:

1. Temporary File Creation: Writes data to a temporary file in the same directory.
2. Forced Sync: Calls fsync on the file descriptor to ensure data reaches physical storage.
3. Atomic Rename: Uses rename to move the temporary file to the final location, which is atomic on POSIX systems.
4. Directory Sync: Calls fsync on the parent directory to guarantee the directory entry is persisted.

This pattern ensures that even if power is lost mid-write, the existing valid file remains untouched, and readers never see partial data. The storage directory defaults to /data/params/d, created lazily by ensure_params_path().

Reading Stored Parameters

The retrieval flow, implemented in common/params_pyx.pyx (lines 14-31), converts raw bytes back to Python types using the CPP_2_PYTHON mapping table.

Calling Params.get(key, block=False) attempts to read the file contents immediately. If block=True, the call waits until the file becomes non-empty, useful for synchronizing across processes. The file name corresponds directly to the key name, making the storage structure transparent and debuggable.

Lifecycle Management and Automatic Cleanup

The system/manager/manager.py module (lines 31-36) orchestrates cleanup by invoking params.clear_all() with specific flags at well-defined moments:

params.clear_all(ParamKeyFlag.CLEAR_ON_MANAGER_START)
params.clear_all(ParamKeyFlag.CLEAR_ON_ONROAD_TRANSITION)
params.clear_all(ParamKeyFlag.CLEAR_ON_OFFROAD_TRANSITION)
params.clear_all(ParamKeyFlag.CLEAR_ON_IGNITION_ON)

This selective removal targets only keys without the PERSISTENT flag. Consequently, calibration data and user preferences remain intact across reboots, while temporary debugging flags or session-specific state resets appropriately.

Non-Blocking Thread-Safe Writes

For performance-critical paths, params.put_nonblocking() queues writes to a background thread (asyncWriteThread) defined in common/params.cc (lines 27-43). The queue eventually executes the same atomic put routine, ensuring that even asynchronous writes maintain the same crash-safety guarantees while preventing I/O blocking in the main execution path.

Practical Usage Examples

from openpilot.common.params import Params, ParamKeyFlag

# Create a Params instance using the default storage prefix

params = Params()

# Store a persistent value (key must be declared with PERSISTENT flag)

params.put("UserPreferredSpeed", 95)

# Retrieve the value later, even after system reboot

speed = params.get("UserPreferredSpeed", return_default=False)
print(f"Preferred speed: {speed}")  # Output: 95

# Store a volatile flag (cleared on manager start)

params.put_bool("TempDebugMode", True)

# Clear all non-persistent keys (as system manager does)

params.clear_all(ParamKeyFlag.CLEAR_ON_MANAGER_START)

# TempDebugMode is deleted, but UserPreferredSpeed remains

Summary

• Storage Format: One file per key under /data/params/d, with filenames matching key names.
• Persistence Control: Determined by ParamKeyFlag in common/params_keys.h; only PERSISTENT keys survive cleanup events.
• Crash Safety: Atomic writes use temporary files, fsync, and rename operations to prevent corruption.
• Lifecycle Integration: system/manager/manager.py clears volatile keys during ignition and state transitions.
• Concurrency: Background write queue enables non-blocking, thread-safe parameter updates.

Frequently Asked Questions

Where are parameter values physically stored on disk?

According to the commaai/openpilot source code in common/params.cc, each key is stored as a separate file under /data/params/d/ (or an alternative path set via environment variables). The filename matches the key name exactly, and the file contents store the value as raw bytes, making the database transparent and accessible with standard Unix tools.

What determines whether a parameter survives a reboot?

Persistence is defined at compile time in common/params_keys.h. Keys flagged with PERSISTENT in the ParamKeyFlag enum are excluded from all automatic cleanup operations. All other keys—marked with flags like CLEAR_ON_MANAGER_START or CLEAR_ON_ONROAD_TRANSITION—are removed when system/manager/manager.py invokes params.clear_all() during the corresponding lifecycle event.

How does the system prevent data corruption if power is lost during a write?

The C++ implementation in common/params.cc (lines 30-67) uses an atomic write pattern: data is written to a temporary file, fsync forces it to disk, rename moves it into place atomically, and a final directory fsync ensures the directory entry is persisted. Readers always see either the complete old file or the complete new file, never a partial write.

Can multiple threads write to the params system simultaneously?

Yes. The system provides thread-safe operations through two mechanisms: the atomic file rename ensures that even concurrent writes to the same key result in a valid state (last-write-wins), and put_nonblocking() queues writes to a dedicated background thread (asyncWriteThread), preventing I/O blocking while maintaining the same atomic persistence guarantees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}