How File Upload and Batch Operations Work in CasaOS: A Technical Deep Dive

CasaOS handles large files through chunked uploads stored in temporary .tmp files and executes bulk file operations asynchronously via a worker queue system.

CasaOS, an open-source personal cloud platform developed by IceWhaleTech, provides enterprise-grade file management through its Go-based backend architecture. Understanding how file upload and batch operations work in CasaOS reveals a sophisticated design that separates HTTP request handling from filesystem I/O. This implementation ensures the UI remains responsive while processing multi-gigabyte transfers and bulk modifications.

Chunked File Upload Architecture

The chunked upload system enables reliable transfer of large files by breaking them into manageable segments. This approach prevents memory exhaustion and supports resumable uploads through an in-memory state tracking mechanism.

Upload Endpoint and Metadata Handling

The upload process begins at POST /v2/file, handled by the PostUploadFile function in route/v2/file.go. The client sends a multipart form containing chunk metadata including chunkNumber, chunkSize, totalChunks, totalSize, identifier, and relativePath. This metadata allows the server to reconstruct the file in the correct order regardless of arrival sequence.

Temporary File Assembly

The FileUploadService.UploadFile method in service/file_upload.go manages the actual file construction. The service maintains an in-memory map (sync.Map) called uploadStatus to track chunk progress without database overhead. For each chunk received:

1. The system opens a temporary file with the .tmp extension
2. Uses os.OpenFile with Seek to position the write cursor at (chunkNumber-1) × chunkSize
3. Copies the incoming bytes using io.Copy
4. Updates the FileInfo status in the sync.Map, marking uploaded[chunkNumber-1] = true

When uploadedChunkNum equals totalChunks, the service closes the temporary file, renames it to the final filename (removing the .tmp extension), and removes the entry from the upload map. This ensures partially uploaded files never appear as complete in the filesystem.

Chunk Verification Endpoint

Clients can verify which chunks exist using GET /v2/file/check via the CheckUploadChunk handler. The endpoint returns 204 No Content if the chunk is missing, or 200 OK if present, enabling resumable upload implementations in the UI.

Batch File Operations Architecture

CasaOS implements batch operations through a producer-consumer pattern that queues file operations for asynchronous execution.

Queue and Worker Pattern

The batch API endpoint POST /v1/batch/task in route/v1/file.go triggers the PostOperateFileOrDir handler. This handler validates the request against the model.FileOperate struct, checking for empty item lists, identical source/destination paths, and mounted volume restrictions. Upon validation:

1. A UUID is generated for the operation
2. The FileOperate object is stored in service.FileQueue (a sync.Map)
3. The ID is appended to service.OpStrArr
4. If the queue was empty, two goroutines spawn: ExecOpFile for execution and CheckFileStatus for progress monitoring

Operation Execution Flow

The ExecOpFile function in service/file.go processes queued operations sequentially. For each FileOperate item, it iterates through source paths and performs:

• Copy operations: Uses file.CopyDir to duplicate directory structures
• Move operations: Combines file.CopyDir, os.RemoveAll, and file.MoveFile to relocate items
• Overwrite handling: Respects the style parameter (e.g., "overwrite") when destinations exist

The HTTP layer returns immediately with a success response, while these background workers handle the actual filesystem I/O.

Progress Tracking and Notifications

The CheckFileStatus goroutine polls every 3 seconds, calculating completion by comparing ProcessedSize against destination sizes using file.GetFileOrDirSize. When operations complete, the Notify service emits WebSocket broadcasts via SendFileOperateNotify, providing real-time updates to connected clients without requiring polling.

Implementation Examples

Uploading Files via Chunked Transfer

Upload large files by splitting them into chunks and posting to the /v2/file endpoint:

# Example: Uploading a 10MB file in 5MB chunks

FILE=/path/to/large.bin
SIZE=$(stat -c%s "$FILE")
CHUNK=5242880  # 5 MiB

TOTAL=$(( (SIZE + CHUNK - 1) / CHUNK ))
IDENT=upload-$(uuidgen)

for ((i=1; i<=TOTAL; i++)); do
  curl -X POST http://localhost:80/v2/file \
    -F "path=/data" \
    -F "chunkNumber=$i" \
    -F "chunkSize=$CHUNK" \
    -F "currentChunkSize=$(dd if=$FILE bs=$CHUNK skip=$((i-1)) count=1 2>/dev/null | wc -c)" \
    -F "totalChunks=$TOTAL" \
    -F "totalSize=$SIZE" \
    -F "identifier=$IDENT" \
    -F "relativePath=large.bin" \
    -F "filename=large.bin" \
    -F "file=@<(dd if=$FILE bs=$CHUNK skip=$((i-1)) count=1 2>/dev/null)"
done

The PostUploadFile handler receives each segment and writes it to <path>/<relativePath>.tmp, assembling the final file once all chunks arrive.

Executing Batch Copy or Move Operations

Queue bulk operations using the /v1/batch/task endpoint with a JSON payload matching the model.FileOperate structure:

curl -X POST http://localhost:80/v1/batch/task \
  -H "Content-Type: application/json" \
  -d '{
        "type": "move",
        "style": "overwrite",
        "to": "/data/destination",
        "item": [
          { "from": "/data/source1.txt" },
          { "from": "/data/source2.txt" }
        ]
      }'

This returns immediately while ExecOpFile processes the queue in the background. Monitor progress through the WebSocket channel at /ws or by observing the ProcessedSize field in subsequent status checks.

Batch Deletion Requests

Delete multiple paths simultaneously via DELETE /v1/batch:

curl -X DELETE http://localhost:80/v1/batch \
  -H "Content-Type: application/json" \
  -d '["/data/old1.txt","/data/old2.txt"]'

The DeleteFile handler validates paths against mounted volumes before executing os.RemoveAll on each target.

Key Source Files and Components

Understanding the CasaOS file upload and batch operations implementation requires familiarity with these specific files:

• service/file_upload.go: Contains UploadFile and TestChunk methods managing the chunked upload state machine and temporary file assembly
• route/v2/file.go: HTTP handlers PostUploadFile and CheckUploadChunk for the upload API surface
• route/v1/file.go: Implements PostOperateFileOrDir and DeleteFile for batch operation endpoints
• service/file.go: Houses the FileQueue (sync.Map), ExecOpFile worker, and CheckFileStatus monitoring logic
• model/file_operate.go: Defines the FileOperate struct specifying operation types, source items, and destination paths

Summary

• Chunked uploads in CasaOS use temporary .tmp files and in-memory sync.Map tracking to assemble large files without blocking the HTTP layer
• The batch operations API (/v1/batch) implements a non-blocking queue system where PostOperateFileOrDir enqueues tasks and background goroutines execute them
• File upload service (service/file_upload.go) handles chunk positioning via Seek and io.Copy, verifying completion when uploadedChunkNum equals totalChunks
• Worker processes (ExecOpFile and CheckFileStatus) run asynchronously, updating progress every 3 seconds and emitting WebSocket notifications upon completion
• All paths are validated against mounted volumes before deletion or batch operations to prevent system damage

Frequently Asked Questions

How does CasaOS handle interrupted uploads?

CasaOS stores upload progress in an in-memory sync.Map within FileUploadService. If an upload interrupts, the temporary .tmp file persists on disk while the client can query CheckUploadChunk to identify missing chunks. Resuming the upload involves sending only the missing segments, which the UploadFile method writes to the correct offset using Seek before finalizing the file when all chunks arrive.

What is the maximum file size supported for uploads?

The chunked upload architecture theoretically supports files of any size limited only by available disk space and filesystem constraints (such as ext4's 16TB limit or btrfs limits). Since CasaOS streams chunks to disk using io.Copy rather than buffering in memory, individual chunk size and total file size depend on the chunkSize parameter sent by the client, typically configured in the UI settings.

Can batch operations be cancelled once queued?

Once a batch operation enters the service.FileQueue and processing begins via ExecOpFile, the system does not expose a cancellation endpoint in the current implementation. The operation runs to completion unless the CasaOS process restarts, which clears the in-memory queue. Clients should verify operation parameters before submitting to PostOperateFileOrDir.

Why does the batch API return immediately instead of waiting for completion?

The PostOperateFileOrDir handler returns immediately to prevent HTTP timeouts during long-running file operations like copying large directories or moving between different storage devices. This asynchronous design, implemented in route/v1/file.go, allows the UI to remain responsive while background workers in service/file.go handle the actual filesystem I/O and report progress via WebSocket notifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}