Problem

When requesting to upload a large file (test file size 1GB), an HTTP 500 error response is received. The detailed response header is as follows.

• error-no: 101
• message: The server is not responding.

Cause

After the file upload request, if there is no response for a certain period of time, the file client closes the connection with the server and ends the transaction. This is to prevent the client from waiting infinitely for a response in an abnormal situation where the server cannot respond. If the timeout period is not specified, connection resources may be leaked. However, we have difficulty specifying the timeout time. The time to save a file varies greatly depending on the file size and the performance of the storage device. Therefore, it is difficult to have a consistent standard for exactly how long a client should judge a server to be abnormal. In the problem situation above, the client was set to wait 60 seconds, but when transferring a 1GB file using a slow storage device (USB 3.0), the file upload could not be processed within 60 seconds, causing problems.

Solution

Large file uploads are internally split into 5 MB chunks. And the server has a mechanism to send a response to the client when the last received chunk is uploaded. To solve this problem, whenever the server uploads one 5MB chunk, it can send an additional intermediate response to the client indicating that it is currently processing normally. The client can properly wait for the normal processing time of fixed-size 5MB chunks.

Implement

• Sends an intermediate response (ChunkTransferOk) each time it finishes saving a chunk of the file (5MB).
• The ChunkTransferOk response is sent from the server to the client on file uploads and from the client to the server on downloads. This message does not perform any processing and is only used to inform the IdleStateHandler that the current channel is processing the file normally.

Result

Reliably upload (or download) files without being affected by file size or storage media performance.

Version

0.0.1

Problem

The design of the file upload and download APIs doesn't provide the right concepts. For example,

1. I used the PUT method for idempotent, but the URL actually shows the actions upload and download, not the data related to idempotent.
2. When uploading a file, the idea is that the file is copied from localFile to remoteFile, but the notion of local severely limits the functionality that Fireball provides. From user perspective, the source files you want to upload may not be local, but remote.
3. It does not provide a clear distinction between the concepts of an API server requesting a file transfer and a file server exchanging files.

Solution

1. Make sure that the only thing the URL identifies is an action, not a file. If we have a URL like http://{ip}:{port}/{action}, we can say that it specifies an action for a process with port on a host with ip. Therefore, we consider this action to be more like a POST, where a new action is executed each time, rather than an idempotent.
2. It applies a more general concept rather than one aspect of what the service provides: the concept of source instead of local, and destination instead of remote.
3. Improve a few points in the user experience to make the distinction between the API server and the file server a little clearer.

• Add an api subdomain to our API URL. http://{ip}:{port}/api/{action}
• When users run the service, they can specify the file server port (fireball.server.port). Rename it to fire.server.port so that the name of the setting provides the appropriate meaning.

TcpClient, TcpServer 메서드들의 반환 타입이 비동기적인 처리를 지원하도록 Future를 반환하도록 합니다.

The information below is gathered from the following blog posts. (https://techblog.woowahan.com/11392/)

• Uploading to S3 features
• How to know the progress of your upload
• How to request N files to be uploaded at once
• Minimal memory usage
• Security/authentication features are provided by the default

현재 TcpClient, TcpServer 메서드들의 반환 타입이 중구난방입니다. netty와 java 패키지의 Future를 섞어서 사용해서 사용 상에 혼돈이 생길 수 있다고 판단됩니다.

It would be nice to have a node manager up front that installs and manages fireball instances on the network like a hadoop node manager.

HTTP methods and URIs allow you to identify the behavior of the file server and request actions. This provides a more flexible and extensible interface to the servers. In addition, you can utilize a variety of HTTP-based load testing and monitoring tools to make testing and operations easier.