Exception: Remote server replied with invalid content-length. about trade-dangerous

[Tromador]

Yes. I read that. You will notice that the default DeflateBufferSize is 8KB and we are dealing with files well over 100MB, so I would have to increase the buffer by 6 (?) orders of magnitude.

This is not a good solution for a number of reasons.

Firstly, chunking data and not providing content-length is correct behaviour per HTTP/1.1, thus the server isn't broken or misbehaving. Rather it's the client that is not conforming to spec:
"All HTTP/1.1 applications that receive entities MUST accept the "chunked" transfer-coding (section 3.6), thus allowing this mechanism to be used for messages when the message length cannot be determined in advance." (RFC 2616)
"If a Transfer-Encoding header field is present and the chunked transfer coding (Section 4.1) is the final encoding, the message body length is determined by reading and decoding the chunked data until the transfer coding indicates the data is complete." (RFC 7230)
(and lots of other bits, but you get the idea)

Again 6 orders of magnitude increase to have a 200MB (for headroom) buffer, when the default is 8K. Note that this specifies what fragment size is sent to zlib for compression. Unless anyone can tell me what the effect of sending 100+MB fragment sizes to zlib is, I am very wary of going down that road.

Assuming creating a massive buffer didn't cause zlib to throw a fit or run the server out of memory as it tries to allocate multiple instances of 200MB buffers all over, its still the wrong approach. The correct way to achieve this server side is to maintain pre-gzipped files, then use mod_rewrite, using something similar to https://stackoverflow.com/questions/9076752/how-to-force-apache-to-use-manually-pre-compressed-gz-file-of-css-and-js-files
This is fine, however we still would need to work on the client such that it knew to gzip all files stored on the server.

In short - more correct and robust to make the client compliant with the standard, than set up outlandish server configuration. Yes it could be done. No, I don't think it's wise.

[aadler]

This post on serverfault looks promising. Quote is:

Apache uses chunked encoding only if the compressed file size is larger than the DeflateBufferSize. Increasing this buffer size will therefore prevent the server using chunked encoding also for larger files, causing the Content-Length to be sent even for zipped data. More Information is available here: http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#deflatebuffersize

[eyeonus]

Turns out, TD doesn't look for content-length when the data is chunked:

    encoding = req.headers.get('content-encoding', 'uncompress')
    length = req.headers.get('content-length', None)
    transfer = req.headers.get('transfer-encoding', None)
    if transfer != 'chunked':
        # chunked transfer-encoding doesn't need a content-length
        if length is None:
            raise Exception("Remote server replied with invalid content-length.")
        length = int(length)
        if length <= 0:
            raise TradeException(
                "Remote server gave an empty response. Please try again later."
            )

So the problem is when the data isn't chunked and the content-length header doesn't exist, which from our conversation, is something that isn't supposed to happen.

[Tromador]

Well. I don't think it's supposed to happen, but something is happening and it's apache, on a standard centos 6.9 install and there's no good reason why it should randomly misbehave - it's not like I built it from source with a bunch of funky configure options.

Is there a way to have TD dump out the server headers when this happens, so I can see exactly what was and what was not sent? Already been down one rabbit hole, I prefer not to speculate again.

For example (from the apache manual)
"HTTP/1.1 defines the Transfer-Encoding header as an alternative to Content-Length, allowing the end of the response to be indicated to the client without the client having to know the length beforehand."

So maybe it's sending some other Transfer-Encoding that TD isn't checking for. Which is me speculating wildly and I think I need to know exactly what the client is seeing to progress.

This does resolve one thing in my mind though - given the file sizes, I simply couldn't imagine how they weren't getting chunked. Short answer is they probably are, most (if not all) of the time, but TD actually does handle this correctly.

[eyeonus]

Yes. It's quite easy to have TD do that, actually.

[Tromador]

So - if it happens again, copy the headers below and we'll see what we can see.

[Tromador]

We've had this open for a month now and I've not had any feedback. Closing due to lack of interest.