Low BLOB_MAX_CHUNK_DATA_SIZE leads to performance hits for max_connections=1 about azure-storage-python

[jehine-MSFT]

Hello @Segsfault,

Thanks for the feedback about our download APIs! We are working to improve the perf in all scenarios. For single-threaded downloads we have a design that will greatly improve perf and hope to complete implementation in time for our next major release. Additionally, we are in the process of finding a solution to better handle multi-threaded download.

Thanks again!

[emgerner-msft]

In 0.30.0 single threaded downloads will not be chunked by default whatsoever. This should always improve perf in this case as there is only a single network call made.

Multi threaded downloads (ie max_connections>1) will be chunked by the previous rules. First, a get properties will give us the size of the blob. Then, under 64MB will be done in a single get and over 64MB chunked into 4MB pieces which the various threads can download simulatenously. The variables for these limits have been renamed (MAX_SINGLE_GET_SIZE, MAX_CHUNK_GET_SIZE) but remain changeable.