How to Handle ROM Uploads in RomM's FastAPI: A Complete Guide to Chunked Uploads

RomM implements a robust chunked-upload workflow in FastAPI that splits large ROM files into 64 MiB chunks, tracks them in Redis, and assembles them atomically to prevent corruption.

Handling large ROM files in web applications requires careful memory management and fault tolerance. The rommapp/romm project solves this through a sophisticated chunked-upload API built on FastAPI, allowing you to handle ROM uploads in RomM's FastAPI backend securely and efficiently. This architecture prevents timeouts, enables resume capability, and ensures data integrity through atomic file operations.

The Four-Stage Upload Lifecycle

According to the source code in backend/endpoints/roms/upload.py, RomM's upload process follows a strict state machine with four distinct phases.

1. Initialize the Upload Session (POST /upload/start)

The client initiates the process by calling POST /upload/start with metadata headers. This endpoint, located at lines 99-148 in backend/endpoints/roms/upload.py, performs the following actions:

• Creates a UUIDv4 upload_id to identify the session
• Validates the platform ID, target filename, total file size, and expected chunk count
• Stores session metadata in Redis as a JSON object under the key chunked_upload:<upload_id>
• Creates a temporary directory at ROM_UPLOAD_TMP_BASE/<upload_id> for storing incoming chunks
• Associates the session with the requesting user ID for ownership validation

The endpoint returns the upload_id to the client, which must persist this identifier for subsequent requests.

2. Stream Chunks (PUT /upload/{upload_id})

With an active session, the client uploads individual chunks via PUT /upload/{upload_id}. The implementation at lines 165-244 enforces several critical constraints:

• Authorization: Validates the bearer token and confirms the requesting user owns the session via request.user.id
• Size limits: Rejects any chunk exceeding the hard-coded 64 MiB limit
• Index tracking: Validates the x-chunk-index header against the expected total chunks
• Storage: Writes each chunk to ROM_UPLOAD_TMP_BASE/<upload_id>/<chunk_index>.bin using anyio.open_file for asynchronous I/O
• State management: Records received chunk indexes in a Redis set (chunked_upload:<id>:chunks) for O(1) existence checks

3. Assemble the ROM (POST /upload/{upload_id}/complete)

Once all chunks are transmitted, the client calls POST /upload/{upload_id}/complete to trigger assembly. The handler at lines 250-350 implements an atomic write pattern:

1. Verification: Confirms all expected chunks exist in the Redis set and removes the session key to prevent concurrent assembly attempts
2. Validation: Ensures the destination directory exists and the final path is valid
3. Streaming: Opens each temporary chunk file sequentially and streams contents into a temporary assembly file (.<name>.<uuid>.assembling)
4. Atomic commit: Validates the assembled size matches the original total, then uses replace() to rename the temporary file to its final destination
5. Cleanup: Removes temporary chunk files after successful assembly

This atomic rename guarantees that either a complete, valid file exists at the destination, or no file at all—preventing partial corruption.

4. Cancel the Session (POST /upload/{upload_id}/cancel)

If the upload fails or is abandoned, the client can call POST /upload/{upload_id}/cancel (lines 350-376). This endpoint removes the Redis session entry and deletes the temporary directory, freeing storage immediately.

Security and Permission Controls

The upload endpoints use the @protected_route decorator to enforce Scope.ROMS_WRITE authorization. Each request validates:

• Bearer token presence and validity
• Session ownership against request.user.id (lines 180-188)
• Platform ID existence and user permissions for that platform

This ensures users cannot upload to unauthorized platforms or hijack other users' upload sessions.

Temporary Storage Configuration and Cleanup

RomM manages temporary upload data through two mechanisms:

Redis TTL: Session keys expire automatically after 24 hours (ROM_UPLOAD_TTL), preventing abandoned sessions from consuming memory indefinitely.

Filesystem cleanup: The CleanupUploadTmpTask in backend/tasks/scheduled/cleanup_upload_tmp.py runs hourly to scan ROM_UPLOAD_TMP_BASE (default: <RESOURCES_BASE_PATH>/tmp/uploads). It deletes any directory older than the TTL that lacks a corresponding Redis session, preventing storage bloat from orphaned uploads.

Client Implementation Example

The following cURL commands demonstrate the complete upload lifecycle against RomM's FastAPI backend:

Start the session:

curl -X POST "http://localhost:8000/api/upload/start" \
  -H "Authorization: Bearer <TOKEN>" \
  -H "x-upload-platform: 1" \
  -H "x-upload-filename: mygame.rom" \
  -H "x-upload-total-size: 12345678" \
  -H "x-upload-total-chunks: 5"

Upload individual chunks (repeat for each index):

curl -X PUT "http://localhost:8000/api/upload/${UPLOAD_ID}" \
  -H "Authorization: Bearer <TOKEN>" \
  -H "x-chunk-index: 0" \
  --data-binary "@chunk_0.bin"

Complete the upload:

curl -X POST "http://localhost:8000/api/upload/${UPLOAD_ID}/complete" \
  -H "Authorization: Bearer <TOKEN>"

Cancel if needed:

curl -X POST "http://localhost:8000/api/upload/${UPLOAD_ID}/cancel" \
  -H "Authorization: Bearer <TOKEN>"

Summary

• RomM's FastAPI handles large ROM uploads through a four-stage chunked workflow that prevents memory exhaustion and enables resumability.
• Each upload session receives a UUID and stores metadata in Redis with a 24-hour TTL, while chunks are written to temporary files under ROM_UPLOAD_TMP_BASE.
• The assembly process uses atomic file replacement to guarantee data integrity, ensuring partial files never reach the final library.
• Security controls enforce scoped permissions and session ownership, preventing unauthorized uploads or session hijacking.
• Background tasks automatically clean up orphaned temporary directories older than the configured TTL.

Frequently Asked Questions

What is the maximum chunk size for ROM uploads in RomM?

RomM enforces a hard-coded 64 MiB limit per chunk. Any chunk exceeding this size is rejected by the PUT /upload/{upload_id} endpoint. This limit balances memory usage with network efficiency, preventing excessive RAM consumption while minimizing HTTP request overhead for large ROM files.

How does RomM prevent corrupted ROM files during upload?

The system uses atomic file assembly. During the completion phase, the server writes to a temporary filename with a .assembling suffix and only replaces the final destination file after verifying the assembled size matches the original total. This ensures that either a complete, valid file exists in the library, or no file at all—partial writes are never committed to the final location.

Can I resume an interrupted ROM upload in RomM?

While the API tracks received chunks in a Redis set and validates completeness before assembly, the current implementation does not expose a dedicated endpoint to query missing chunks. A client could theoretically resume by checking which chunk indexes exist in the temporary directory, but the standard workflow expects clients to track upload progress locally and restart the session if the upload fails.

Where does RomM store temporary upload chunks?

Chunks are stored under the directory specified by ROM_UPLOAD_TMP_BASE, which defaults to <RESOURCES_BASE_PATH>/tmp/uploads/<upload_id>/. Each chunk is saved as <chunk_index>.bin. A background task runs hourly to delete directories older than 24 hours that lack active Redis sessions, preventing storage accumulation from abandoned uploads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}