CasaOS File Upload Service Architecture: A Deep Dive into the Chunked Upload Pipeline

CasaOS implements a layered, chunk-aware file upload pipeline that separates upload state management, temporary storage handling, and HTTP routing to support resumable, multi-chunk file transfers.

The architecture of the file upload service in CasaOS follows a modular design that enables reliable handling of large files through chunked transfers. This system separates concerns between state management, filesystem operations, and HTTP request handling to ensure uploads can resume after interruptions. Understanding these layers helps developers integrate with or extend CasaOS's storage capabilities.

Core Components of the Upload Architecture

The file upload service is organized into four distinct layers that work together to process multipart uploads.

Service Layer (service/file_upload.go)

The FileUploadService struct serves as the central coordinator for upload state. It maintains a sync.Map of FileInfo objects that track initialization flags, uploaded bitmaps, and chunk counts for each active transfer. The service uses a sync.RWMutex to ensure thread-safe concurrent access to upload metadata.

Key methods include:

• UploadFile – Writes individual chunks to temporary .tmp files and updates the bitmap tracking which segments have been received.
• TestChunk – Provides a lightweight mechanism to probe whether a specific chunk already exists in the temporary storage.

Utility Layer (pkg/utils/file/file.go)

Low-level filesystem operations live in the utility package, decoupled from HTTP concerns. The SpliceFiles function concatenates numbered chunk files into the final destination file by opening each chunk sequentially and writing them via a buffered writer. Supporting functions like MkDir, CheckNotExist, and RMDir handle directory creation and cleanup operations.

HTTP API Layer (route/v1/file.go)

The router exposes two primary endpoints that translate HTTP parameters into service calls:

• GetFileUpload (GET) – Acts as a "chunk exists" probe. It calculates a content hash using file.GetHashByContent, creates the temporary directory structure, and returns HTTP 200 if the chunk exists or 204 if the chunk is ready for upload.
• PostFileUpload (POST) – Receives multipart form data, writes chunks to the temporary folder, and triggers finalization when the last chunk arrives.

Server Wiring (route/v2/route.go)

During application startup, the NewCasaOS function instantiates FileUploadService and injects it into the generated server interface. This dependency injection pattern ensures the upload service is available throughout the request lifecycle.

How the Chunked Upload Pipeline Works

The upload process follows a stateless, file-system-backed protocol where progress is inferred from the presence of chunk files on disk rather than persistent database records.

Step 1: Chunk Probing (Optional)

Before uploading data, clients may probe the server to check which chunks already exist. The client sends a GET request to /file/upload with parameters including path, filename, totalChunks, and chunkNumber. The handler calculates a hash of the filename and checks for the existence of <target>/.temp/<hash><totalChunks>/<chunkNumber>. If found, the server returns HTTP 200; otherwise, it returns HTTP 204.

Step 2: Temporary Directory Creation

For multi-chunk uploads, the system creates a temporary directory under the target path following the pattern .temp/<hash><totalChunks>/. The hash is derived from the filename content, ensuring unique isolation for each upload operation.

Step 3: Chunk Reception and Storage

When receiving a POST request, the handler extracts the multipart file using ctx.Request().FormFile("file"). It writes the chunk data to a file named with its chunk number inside the temporary directory. After each write, the handler reads the directory contents to count existing files, comparing this against totalChunks to determine completion status.

Step 4: Finalization and Splicing

When the number of received chunks equals totalChunks, the handler invokes SpliceFiles(tempDir, path, totalChunks, 1). This function opens the final destination file and sequentially appends each numbered chunk (1 through N) to construct the complete file. The function uses buffered I/O to optimize filesystem performance during concatenation.

Step 5: Cleanup

After successful splicing, the system schedules asynchronous cleanup using time.Sleep(11s) followed by file.RMDir(tempDir). This delay ensures file handles are released and any pending operations complete before removing the temporary structure.

Single-File Optimization

When totalChunks equals 1, the pipeline bypasses the temporary folder entirely. The handler writes directly to <target>/<relativePath>, eliminating overhead for small files.

Implementation Examples

Single-File Upload via cURL

For files that don't require chunking, upload directly to the destination:

curl -X POST http://localhost:80/file/upload \
  -F "path=/share/downloads" \
  -F "filename=myphoto.jpg" \
  -F "relativePath=myphoto.jpg" \
  -F "totalChunks=1" \
  -F "chunkNumber=1" \
  -F "file=@myphoto.jpg"

Chunked Upload Workflow

For large files, implement the probe-then-upload pattern:

# Step 1: Probe existing chunks (optional resumption check)

for i in 1 2 3; do
  curl -G "http://localhost:80/file/upload" \
    -d "path=/share/downloads" \
    -d "filename=video.mp4" \
    -d "relativePath=video.mp4" \
    -d "totalChunks=3" \
    -d "chunkNumber=$i"
done

# Step 2: Upload each chunk

for i in 1 2 3; do
  curl -X POST http://localhost:80/file/upload \
    -F "path=/share/downloads" \
    -F "filename=video.mp4" \
    -F "relativePath=video.mp4" \
    -F "totalChunks=3" \
    -F "chunkNumber=$i" \
    -F "file=@chunk_$i.bin"
done

When the third chunk arrives, CasaOS automatically splices the temporary files into /share/downloads/video.mp4 and schedules the temp folder for deletion.

Programmatic Upload with Go

The repository includes generated client code under codegen/client/ for programmatic integration:

import (
    "context"
    "github.com/IceWhaleTech/CasaOS/codegen"
    "github.com/IceWhaleTech/CasaOS/codegen/client"
)

func uploadChunk(apiClient *client.APIClient, path, filename string, 
                 chunkNum, totalChunks int, data []byte) error {
    opts := codegen.PostFileUploadOpts{
        Path:         path,
        Filename:     filename,
        RelativePath: filename,
        TotalChunks:  strconv.Itoa(totalChunks),
        ChunkNumber:  strconv.Itoa(chunkNum),
    }
    _, err := apiClient.FileApi.PostFileUpload(
        context.Background(), 
        data, 
        &opts,
    )
    return err
}

Summary

• CasaOS file upload service architecture separates concerns across four layers: service state management, filesystem utilities, HTTP routing, and server wiring.
• Chunked uploads use temporary directories under .temp/<hash><totalChunks>/ to store individual segments until finalization.
• State tracking occurs in memory via sync.Map and on disk via file presence, eliminating the need for persistent metadata storage during transfers.
• Finalization uses the SpliceFiles function to concatenate chunks sequentially, followed by asynchronous cleanup after 11 seconds.
• Single-file uploads bypass the temporary storage system entirely when totalChunks equals 1.

Frequently Asked Questions

How does CasaOS handle interrupted uploads?

CasaOS supports resumable uploads through its chunk existence probe mechanism. Clients can query the GetFileUpload endpoint to check which chunks already exist in the temporary directory. Since chunk state persists as physical files in .temp/<hash><totalChunks>/, interrupted uploads can resume by re-sending only missing chunks. The server reconstructs the final file only when all chunks are present.

What is the temporary directory structure for chunked uploads?

The system creates temporary directories under the target path using the pattern .temp/<hash><totalChunks>/, where hash is derived from file.GetHashByContent(filename) and totalChunks represents the total number of expected segments. Individual chunks are stored as numbered files (1, 2, 3, etc.) within this directory until SpliceFiles concatenates them into the final destination.

How does the service track which chunks have been received?

The FileUploadService maintains upload metadata in a sync.Map of FileInfo objects for internal callers, but the HTTP layer primarily tracks progress by inspecting the filesystem. After each chunk write in PostFileUpload, the handler reads the temporary directory using ioutil.ReadDir and compares the file count against totalChunks. This stateless approach ensures durability across process restarts, as progress is stored on disk rather than in volatile memory.

Can I upload files without using the chunked protocol?

Yes. When totalChunks is set to 1, the PostFileUpload handler in route/v1/file.go bypasses the temporary folder creation and writes directly to <target>/<relativePath>. This optimization eliminates the overhead of directory creation, chunk splicing, and cleanup for small files or single-part uploads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}