Large File Upload Platform

Overview

The browser splits a file into parts and uploads them concurrently to S3 using backend-issued presigned URLs. The API coordinates the upload without relaying those parts. After completion, the backend reads the stored object, computes its full-file SHA-256 and compares it with the digest calculated in the browser.

Problem

The original approach streamed files through the application server. A large upload held a request thread for minutes, request-size limits capped what users could send, and memory and bandwidth on the API tier scaled with file size rather than user count.

There was also no integrity guarantee. A truncated or corrupted upload could be stored and processed without anyone noticing.

Requirements

• Support files well beyond typical request-size limits
• Use the user's full bandwidth with parallel transfers
• Keep upload-part traffic off the application servers; read the completed object for verification
• Grant storage access that is scoped, temporary and per part
• Guarantee the stored object matches what the user sent
• Report meaningful progress to the user

Architecture

The API creates an S3 multipart upload and records a session. The browser uploads parts directly to S3 with bounded concurrency and presigned UploadPart URLs. It then sends the ordered part numbers, ETags and its full-file SHA-256 digest. After S3 completes the upload, the backend reads the completed object and hashes its bytes. Only a matching digest allows the file to become ready. This final readback is separate from the browser-to-S3 upload path.

{
  "uploadId": "2~x7Qp…",
  "fileSize": 734003200,
  "partSize": 16777216,
  "parts": 44,
  "concurrency": 4,
  "completed": [1, 2, 3, 4, 5],
  "inFlight": [6, 7, 8, 9],
  "sha256": "9f2c41…e71a"
}

• AWS: presigned requests can be reused until expiry (opens in a new tab)
• AWS: full-object and composite checksum types (opens in a new tab)

Engineering decisions

Direct-to-storage uploads

The API does not relay upload parts. Verification still reads the completed object, so its object-read traffic, hashing work and latency scale with file size and must be budgeted separately.

Hash on the client, verify on the server

Independent hashing compares the original browser bytes with the completed object. Copying a client digest into metadata alone would not verify the stored content.

Lifecycle rule for abandoned uploads

Incomplete multipart uploads still cost storage. A bucket lifecycle rule aborts them automatically after a set period.

Challenges

Results

Qualitative outcomes; benchmark measurements are not published.

• Large files upload in parallel, directly to storage, beyond the size limits of a single request.
• Upload parts travel directly to S3; the backend reads the object once afterward for verification.
• Every stored file verified against a client-computed SHA-256 before use.
• Scoped, short-lived storage access instead of broad credentials.

Lessons learned

• Separate the control plane from the data plane early. It simplifies scaling and security at the same time.
• Get the protocol right first: a stable part plan is what retries and resumable uploads would build on later.
• Integrity checks are cheap insurance compared with debugging silently corrupted data.

Technologies used

• Java
• Spring Boot
• AWS S3
• Multipart upload
• Presigned URLs
• TypeScript