# How Distilly Ensures Atomic Operations for Storage

> Distilly ensures atomic storage operations by writing to temp files and using os.replace for atomic renames. This guarantees readers see complete old or new versions, never partial writes.

- Repository: [Tianyi Zhou/distilly](https://github.com/titanwings/distilly)
- Tags: internals
- Published: 2026-09-10

---

**Distilly guarantees atomic storage operations by writing data to temporary files first, then using Python's `os.replace()` to perform an atomic rename that ensures readers always see either the complete old version or the complete new version, never a partially written file.**

Distilly is an open-source skill-generation framework developed by titanwings that requires reliable file persistence to prevent data corruption during concurrent access or system crashes. To achieve this, the codebase implements a robust atomic write pattern across its tooling modules, most notably in [`tools/research/xquik_public_posts.py`](https://github.com/titanwings/distilly/blob/main/tools/research/xquik_public_posts.py). This approach ensures that every storage operation maintains data integrity without requiring external locking mechanisms or database transactions.

## The Atomic Write Pattern in Distilly

Distilly's storage strategy follows a three-phase commit process that leverages POSIX filesystem semantics. This pattern ensures that no partially written file is ever exposed to other processes, even if the Python process crashes mid-write.

### Write to a Temporary Location

Before touching any target file, Distilly writes all data to a uniquely named temporary file created within the system's temporary directory. In [`tools/research/xquik_public_posts.py`](https://github.com/titanwings/distilly/blob/main/tools/research/xquik_public_posts.py), the code uses `tempfile.NamedTemporaryFile` with `delete=False` to persist the buffer to disk while keeping the filename accessible for the subsequent rename operation.

```python
import tempfile
from pathlib import Path

def prepare_temp(data: str) -> Path:
    with tempfile.NamedTemporaryFile(
        mode="w", encoding="utf-8", delete=False, suffix=".json"
    ) as tmp:
        tmp.write(data)
        return Path(tmp.name)

```

### Atomic Replacement with os.replace

Once the temporary file is fully written and flushed to disk, Distilly commits the change using `os.replace(src, dst)`. This system call performs an atomic rename on POSIX-compliant filesystems, meaning the destination path is swapped out in a single kernel operation that is indivisible and thread-safe.

```python
import os
from pathlib import Path

def atomic_write(temp_path: Path, target: Path) -> None:
    os.replace(temp_path, target)  # Atomic operation

```

### Failure Handling and Cleanup

If an exception occurs during the write phase before the atomic rename completes, the temporary file remains in the temp directory but is never moved to the target location. This ensures that the original file remains untouched and valid. The temporary files are typically cleaned up by the operating system's temp directory maintenance or can be explicitly removed in exception handlers.

## Implementation in xquik_public_posts.py

The research tool [`tools/research/xquik_public_posts.py`](https://github.com/titanwings/distilly/blob/main/tools/research/xquik_public_posts.py) exemplifies this pattern when persisting downloaded JSON data. Rather than writing directly to the repository path, it buffers content to a temporary file and commits it atomically.

```python
import json, os, tempfile
from pathlib import Path

def store_json(data: dict, target: Path) -> None:
    # Write to temporary file first

    with tempfile.NamedTemporaryFile(
        mode="w", encoding="utf-8", delete=False, suffix=".json"
    ) as tmp:
        json.dump(data, tmp, ensure_ascii=False, indent=2)
        tmp_path = Path(tmp.name)
    
    # Atomically replace the target

    os.replace(tmp_path, target)

```

This implementation guarantees that any concurrent reader opening `target` during the write operation will either see the previous complete version or, after the rename completes, the new complete version.

## Cross-Platform Atomicity Guarantees

While `os.replace()` abstracts underlying platform differences, the atomic guarantee holds across all supported platforms. On Linux and macOS, this maps directly to the atomic `rename(2)` system call. On Windows, it utilizes the `MoveFileEx` function with appropriate flags to ensure transactional behavior. This cross-platform consistency allows Distilly to operate reliably across development environments without platform-specific branching logic.

## Summary

- **Temporary-First Writes**: All data is written to `tempfile.NamedTemporaryFile` instances before touching target paths.
- **Atomic Commit**: `os.replace()` provides single-operation file swapping that prevents partial reads.
- **Failure Safety**: Crashes during write phase leave original files untouched; no database or external locks required.
- **Cross-Platform**: Works on Linux, macOS, and Windows through Python's `os` module abstraction.
- **Key Implementation**: The pattern is implemented in [`tools/research/xquik_public_posts.py`](https://github.com/titanwings/distilly/blob/main/tools/research/xquik_public_posts.py) for robust JSON persistence.

## Frequently Asked Questions

### Does Distilly use file locking for atomic operations?

No. According to the source code analysis of the titanwings/distilly repository, Distilly does not implement file locks or mutexes. Instead, it relies solely on the atomic rename semantics provided by `os.replace()`, which is sufficient for ensuring that readers never encounter partially written files.

### What happens if the Python process crashes during the atomic write?

If the process crashes after the temporary file is created but before `os.replace()` is called, the temporary file remains in the system's temp directory but the target file remains completely unchanged. When the process restarts or the temp directory is cleaned, the stale temporary file is removed without affecting the repository state.

### Is the atomic write pattern used throughout the entire Distilly codebase?

Yes. While [`tools/research/xquik_public_posts.py`](https://github.com/titanwings/distilly/blob/main/tools/research/xquik_public_posts.py) provides the clearest example of this pattern, other modules that generate artifacts—such as skill writers and version managers—follow the same temporary-file-then-replace strategy to maintain consistency across all storage operations.

### Why not use SQLite or a database for atomicity instead of file renaming?

Distilly maintains a lightweight, file-based architecture to avoid external dependencies and keep the toolchain pure-Python. The `os.replace()` approach provides sufficient atomicity for file-based content generation without the overhead of database engines, making the system more portable and easier to deploy in serverless or containerized environments.