How AppFlowy Implements File Storage and Upload/Download: A Technical Deep Dive

AppFlowy implements file storage and upload/download through a layered async architecture that separates storage abstraction, local temporary file caching, and cloud backend integration, enabling resumable multi-part uploads with real-time progress tracking.

AppFlowy-IO/AppFlowy manages user-generated files—images, attachments, and custom themes—using a modular Rust architecture designed for reliability and cloud agnosticism. The file storage and upload/download implementation leverages trait-based abstraction, temporary file staging, and a queue-driven worker system to handle concurrent transfers while insulating the Flutter UI from backend complexity.

Storage Service Abstraction Layer

At the core of AppFlowy’s file handling lies a dual-trait system that decouples the frontend from concrete storage implementations. This design allows the same UI code to target local mock storage, AF Cloud, or future S3-compatible backends without modification.

The Public API Contract

The primary interface for all storage operations is the StorageService trait defined in frontend/rust-lib/flowy-storage-pub/src/storage.rs. This trait exposes async methods for upload creation, deletion, download, and progress subscription:

#[async_trait]
pub trait StorageService: Send + Sync {
    async fn delete_object(&self, url: String) -> FlowyResult<()>;
    fn download_object(&self, url: String, local_file_path: String) -> FlowyResult<()>;
    
    async fn create_upload(
        &self,
        workspace_id: &str,
        parent_dir: &str,
        local_file_path: &str,
    ) -> Result<(CreatedUpload, Option<FileProgressReceiver>), FlowyError>;

    async fn start_upload(&self, record: &BoxAny) -> Result<(), FlowyError>;
    
    async fn resume_upload(
        &self,
        workspace_id: &str,
        parent_dir: &str,
        file_id: &str,
    ) -> Result<(), FlowyError>;
}

All UI-layer components interact with the storage subsystem through an Arc<dyn StorageService>, ensuring the Flutter frontend remains agnostic of whether files live locally or in the cloud.

Cloud Backend Abstraction

The actual remote operations are governed by the StorageCloudService trait in frontend/rust-lib/flowy-storage-pub/src/cloud.rs. This interface defines the multi-part upload protocol and CRUD operations:

pub trait StorageCloudService: Send + Sync {
    async fn create_upload(
        &self, 
        workspace_id: &Uuid, 
        parent_dir: &str, 
        file_id: &str,
        content_type: &str, 
        file_size: u64
    ) -> Result<CreateUploadResponse, FlowyError>;

    async fn upload_part(
        &self,
        workspace_id: &Uuid,
        parent_dir: &str,
        upload_id: &str,
        file_id: &str,
        part_number: i32,
        body: Vec<u8>,
    ) -> Result<UploadPartResponse, FlowyError>;

    async fn complete_upload(
        &self,
        workspace_id: &Uuid,
        parent_dir: &str,
        upload_id: &str,
        file_id: &str,
        parts: Vec<CompletedPartRequest>,
    ) -> Result<(), FlowyError>;
}

The concrete implementation for AppFlowy’s hosted backend, AFCloudFileStorageServiceImpl, resides in frontend/rust-lib/flowy-server/src/af_cloud/impls/file_storage.rs and maps these methods to the AF Cloud HTTP API.

Upload Pipeline Architecture

When a user initiates a file upload, the system executes a five-stage pipeline that guarantees data integrity and supports resumption after network interruptions.

Temporary File Staging

Before any network activity begins, the FileTempStorage struct (in frontend/rust-lib/flowy-storage/src/file_cache.rs) creates a defensive copy of the user’s file inside a .cache_files directory within the application root. This ensures the original file cannot be mutated during the chunked upload process.

Key methods include:

• create_temp_file_from_existing – Copies the selected file into the temporary cache
• delete_temp_file – Cleans up cached chunks after successful upload

Upload Queue Management

The StorageManager (in frontend/rust-lib/flowy-storage/src/manager.rs) orchestrates the upload flow. When create_upload() is invoked, it:

1. Validates against the global is_exceed_storage_limit flag
2. Stages the file via FileTempStorage
3. Persists metadata to SQLite (UploadFileTable)
4. Enqueues a task in the UploadTaskQueue

The FileUploader worker (defined in frontend/rust-lib/flowy-storage/src/uploader.rs) processes the BinaryHeap of tasks with a concurrency limit of 3. It handles retries with exponential backoff and automatically pauses when storage quotas are exceeded. The FileUploaderRunner operates as an async loop reacting to Signal::Proceed and Signal::Stop events to drive the queue forward.

AF Cloud Backend Implementation

The AFCloudFileStorageServiceImpl<T> executes the actual multi-part upload against the remote API:

1. create_upload – Initializes the upload session via the /create-upload endpoint, returning a pre-signed URL and upload_id
2. upload_part – Transmits individual chunks (as Vec<u8>) to the pre-signed URL, storing returned ETags and part numbers in the local database
3. complete_upload – Finalizes the object by sending the assembled parts list to the cloud, where the service reconstructs the final file

This implementation supports resumable uploads; if a transfer interrupts, the system can invoke resume_upload() using the persisted chunk metadata to restart from the last successful part.

Download Implementation

File retrieval follows a simpler path through StorageServiceImpl::download_object (in frontend/rust-lib/flowy-storage/src/manager.rs). The method spawns a background Tokio task that:

1. Checks if local_file_path already exists (idempotent behavior)
2. Calls cloud_service.get_object(url) to fetch raw bytes
3. Writes the data to disk using tokio::fs::OpenOptions

If the file already exists locally, the download is skipped, preventing redundant network traffic.

Progress Notification System

Throughout the upload lifecycle, the system emits FileProgress events containing completion ratios (0.0 to 1.0) or error states. These events flow through a global broadcast channel (global_notifier) to which the UI can subscribe. When create_upload() is called, it returns an optional FileProgressReceiver that streams updates to the Flutter frontend, enabling real-time progress bars and completion callbacks.

Flutter Frontend Integration

The Dart layer interacts with the Rust backend through the AppFlowy SDK (FIDL bridge). In frontend/appflowy_flutter/lib/plugins/document/presentation/editor_plugins/image/upload_image_menu.dart, the upload flow is triggered by:

await storageManager.createUpload(
  workspaceId,
  parentDir,
  file.path,
);

The widget listens to the returned progress stream to display loading indicators and handles completion states when the receiver emits a progress value of 1.0. Additional upload interfaces, such as the theme customizer in theme_upload.dart, reuse the same storage manager methods, demonstrating the abstraction’s flexibility.

Summary

• Modular trait-based design – StorageService and StorageCloudService traits enable swapping cloud providers without UI changes
• Defensive temporary staging – FileTempStorage copies files to .cache_files before upload to prevent mutation
• Queue-driven concurrency – FileUploader processes up to 3 concurrent uploads with retry logic and backpressure handling
• Resumable multi-part protocol – Chunked uploads with SQLite-backed progress tracking support interruption and recovery
• Real-time progress streaming – FileProgressReceiver channels broadcast updates to Flutter widgets
• Idempotent downloads – Background tasks skip existing files to conserve bandwidth

Frequently Asked Questions

How does AppFlowy handle large file uploads?

AppFlowy handles large files through chunked multi-part uploads. The AFCloudFileStorageServiceImpl splits files into discrete parts and uploads them sequentially via upload_part(). Each part’s ETag and number are stored in SQLite, allowing the system to resume from the last successful chunk if the connection drops. A debug-only size limit (maximum_upload_file_size_in_bytes) is enforced during the create_upload phase.

Can AppFlowy resume interrupted uploads?

Yes. The upload system is fully resumable. When resume_upload() is called on the StorageService implementation, the system queries the UploadFileTable for existing chunk metadata and invokes the cloud backend’s upload_part method only for missing segments. The FileUploader queue automatically handles retry logic with exponential backoff until the complete_upload signal is sent.

What cloud storage backends does AppFlowy support?

Currently, AppFlowy ships with AF Cloud as the primary backend, implemented by AFCloudFileStorageServiceImpl in flowy-server/src/af_cloud/impls/file_storage.rs. However, the architecture supports any backend implementing the StorageCloudService trait. Developers can inject alternative implementations for S3-compatible services or local mock storage by providing a different Arc<dyn StorageCloudService> to the StorageManager.

How does the Flutter UI track upload progress?

The UI receives real-time updates through a FileProgressReceiver channel created during the create_upload() call. This receiver streams FileProgress structs containing a float between 0.0 and 1.0. The Flutter widget (e.g., upload_image_menu.dart) listens to this stream and updates progress indicators accordingly. When the value reaches 1.0, the UI displays a completion state and removes the loading indicator.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}