tzvetelin88 / very-big-file-upload-download

Uploading and Downloading very big files, by handling reading chunks and interruptions. Resumability.

Solution Design

• Proper way to download or upload very big files with realtime data streaming and information.
  
• Consume less memory and do not break or crash server or client browser.
  
• Control Blob chunks to be R/W and streamed.
  
• Resumability for uploaded files - continue uploading the file from the last written bytes on the server side.
  
• Cancelation for downloading file.
  
• Handle Browser refresh and cancel downloading file.
  

Software set:

• Socket.io: Bi-directional communication between client and server
  
• NodeJS: Backend using native fs to R/W files in proper way by defined chunk size.
  
• Angular: Frontend is using File System Access API to instruct client browser to write Blob chunks directly on client OS.
  

More information:

• Window.showSaveFilePicker: https://developer.mozilla.org/en-US/docs/Web/API/Window/showSaveFilePicker
• Types: https://www.npmjs.com/package/@types/wicg-file-system-access
  

Application UI Example:

Upload

Download

Interuption due browser refresh or cancel button clicked

Run Backend

Install dependencies

yarn install

Run app:

npx ts-node server/src/server.ts

Exposed on port: 3300 (Change LISTEN_PORT in server.ts)

Run Frontend

cd client/

Install dependencies

yarn install

Run app:

yarn start

Application is listening on

http://localhost:4200/

Configurations - global.conf.ts

// Port to be used in the backend to expose the application.
export const BACKEND_LISTEN_PORT = 3300;

// SocketIO origin URL
export const BACKEND_SOCKETIO_ORIGIN = "http://localhost:4200";

// SocketIO URL that also depends on BACKEND_LISTEN_PORT
export const FRONTEND_SOCKETIO_URL = `http://localhost:${BACKEND_LISTEN_PORT}`;

// This is createReadStream highWaterMark size.
export const RW_CHUNK_SIZE = 8 * 1024 * 100; // 500kb;

// How much of the file we want to read, which defines startingRange for R/W.
export const RW_BUFFER_GET_SIZE = 8 * 1024 * 100; // 500kb

// Stream file here, once all content is ready copy to data Folder.
export const STREAM_FOLDER = "./server/data_stream";

// Directory to save files to
export const DATA_FOLDER = "./server/data";