Implement Resumable & Large Podcast Upload System

[SB2318]

Overview

Implement a robust, resumable upload system for podcast audio files in the UltimateHealth backend. The new upload pipeline should support large files, interrupted network recovery, upload progress tracking, and direct cloud storage uploads while minimizing backend resource usage.

The implementation should be scalable, fault-tolerant, and suitable for production workloads.

---

Motivation

The current upload flow is suitable for small files but becomes unreliable for large podcast recordings due to:

• Network interruptions
• Request timeouts
• Mobile connectivity changes
• Large memory consumption
• Upload failures requiring users to restart from the beginning

A resumable upload mechanism significantly improves reliability and user experience.

---

Goals

• Support uploads larger than 1 GB
• Resume interrupted uploads
• Upload files in chunks
• Track upload progress
• Verify uploaded chunks
• Prevent duplicate uploads
• Support direct cloud storage uploads
• Minimize backend bandwidth usage

---

Upload Flow

Client
   │
   ▼
Create Upload Session
   │
   ▼
Receive Upload ID
   │
   ▼
Split File into Chunks
   │
   ▼
Upload Chunk #1
Upload Chunk #2
Upload Chunk #3
...
   │
   ▼
Backend Verifies Chunk
   │
   ▼
Store Chunk Metadata
   │
   ▼
Resume if Interrupted
   │
   ▼
Complete Upload
   │
   ▼
Merge/Finalize File
   │
   ▼
Generate Podcast Record

---

API Design

Create Upload Session

POST /podcasts/uploads

Response

{
  "uploadId": "uuid",
  "chunkSize": 5242880,
  "expiresAt": "2026-07-20T10:00:00Z"
}

---

Upload Chunk

PUT /podcasts/uploads/:uploadId/chunks/:chunkNumber

Headers:

Content-Range
Content-Length
Content-Type

---

Check Upload Status

GET /podcasts/uploads/:uploadId

Response

{
  "uploadedChunks": [
    1,
    2,
    3,
    4
  ],
  "progress": 45
}

---

Complete Upload

POST /podcasts/uploads/:uploadId/complete

Backend verifies:

• All chunks uploaded
• File integrity
• Metadata

Then finalizes the upload.

---

Cancel Upload

DELETE /podcasts/uploads/:uploadId

Deletes temporary chunks and marks the upload session as cancelled.

---

---

---

Storage Strategy

Preferred flow:

Client

↓

Presigned Upload URL

↓

Cloud Storage (S3 / Cloud Storage)

↓

Backend receives completion event

↓

Podcast Created

Alternative:

Client

↓

Backend

↓

Cloud Storage

Direct cloud uploads are preferred to reduce backend load.

---

Resume Support

When the client reconnects:

GET /uploads/:uploadId

Backend returns uploaded chunk numbers.

Client uploads only the missing chunks.

---

File Validation

Validate:

• MIME type
• Maximum file size
• Chunk ordering
• Chunk checksum (optional)
• Total file checksum
• Upload ownership

Reject corrupted or duplicate uploads.

---

Background Processing

After upload completion:

• Assemble chunks (if required)
• Verify file integrity
• Generate waveform (optional)
• Extract duration
• Generate audio metadata
• Create podcast record
• Queue transcoding if needed

These operations should run asynchronously.

---

Security

• Authenticate every upload request.
• Ensure upload sessions belong to the authenticated user.
• Use signed upload URLs when uploading directly to cloud storage.
• Expire inactive upload sessions automatically.
• Enforce upload size limits.
• Validate file types before processing.

---

Acceptance Criteria

• Upload session API implemented
• Chunk upload API implemented
• Resume uploads supported
• Upload progress endpoint implemented
• Upload cancellation supported
• Upload completion endpoint implemented
• Large files (>1 GB) supported
• Direct cloud storage uploads supported
• File integrity validation implemented
• Automatic cleanup of expired uploads
• Background processing for completed uploads
• Documentation updated
• Automated tests added

---

Implementation Plan

Phase 1 – Upload Sessions

• Design upload session model.
• Create upload lifecycle APIs.
• Store upload metadata.

---

Phase 2 – Chunk Upload

• Implement chunk upload endpoint.
• Validate chunks.
• Track uploaded chunks.
• Handle retries safely.

---

Phase 3 – Resume Support

• Track completed chunks.
• Expose upload progress endpoint.
• Resume interrupted uploads without re-uploading completed chunks.

---

Phase 4 – Cloud Storage Integration

• Generate presigned upload URLs.
• Upload chunks directly to cloud storage.
• Finalize uploads after completion.

---

Phase 5 – Post-Processing

• Verify uploaded file integrity.
• Extract metadata (duration, size, format).
• Generate waveform and thumbnails (optional).
• Queue transcoding or compression if required.
• Create the final podcast record.

---

Phase 6 – Cleanup & Monitoring

• Remove expired upload sessions and orphaned chunks.
• Add logging, metrics, and monitoring for upload success/failure rates.
• Implement retry policies for transient failures.

---

Estimated Effort

Phase	Estimate
Upload Session APIs	1 day
Chunk Upload System	2 days
Resume Logic	1 day
Cloud Storage Integration	1 day
Background Processing	1 day
Testing & Documentation	1 day

Total Estimated Effort: 7 developer days

---

Notes

• The implementation should be storage-provider agnostic, allowing support for AWS S3, Google Cloud Storage, Azure Blob Storage, or compatible S3 services.
• Upload progress should be resilient to app restarts, allowing users to continue interrupted uploads.
• The architecture should be reusable for future large file uploads, such as medical reports, videos, or diagnostic imaging, not just podcasts.

[github-actions]

Thank you for creating this issue! We'll look into it as soon as possible. Your contributions are highly appreciated! 😊

---

🤖 Assignment Guidelines

To get assigned to this issue, simply leave a comment requesting assignment.
The AI bot will automatically assign you if you meet the following eligibility criteria:

1. You do not currently have any other active assigned issues in this repository.
2. If you were previously assigned an issue, you must have submitted a Pull Request for it before requesting a new one.

[github-actions]

This issue has been triaged as a backend task (level:advanced).

Expected Effect & Scope: The issue is valid, well-detailed, and within the repository's scope for podcast content. It's classified as a backend task due to API design, cloud storage integration, and background processing. The proposal involves significant architectural changes for large file uploads, security considerations, and is a large feature, warranting an 'advanced' difficulty and escalation to a maintainer.

This requires backend investigation and has been routed to @SB2318 for review and prioritization.