web.py: Replace `Response.iter_content()` with `Response.save_to_path()`? about alfred-workflow

[fractaledmind]

Since I was the one who first requested iter_content(), I can say that this change makes a lot of sense to me. The use case I had at the time was precisely

retrieving large files in the background without having to load all the data into memory.

Also, I have had one user with two of my workflows have issues, and in both instances the problem was his spotty internet connection. Saving the response data to file ought to allow me to fix this problem. So, I say yes.

[deanishe]

You were indeed the one who requested it, but I've no idea if you're the only one using it…

I don't think it'd help with a spotty Internet connection, however. I have no intention of implementing range headers etc. to enable continuing interrupted downloads.

[owenwater]

Firstly, it's possible to decompress a stream in python which you can find an implementation here.

And here is my sample code. Don't have any web server so test it with a local file stream.

import zlib  
def decompress(stream):
    d = zlib.decompressobj(16 + zlib.MAX_WBITS)
    for chunk in stream:
        rv = d.decompress(chunk)
        if rv:
            yield rv

For save_to_files(), I think it's an interesting feature not just for gzip file. For example, it can be used to download a audio file which will be played later. Always cache large file sounds like a reasonable idea.

However, there are always some scenarios that people don't want cache at all. e.g. A API provider may not allow user to cache anything they got. So my idea is having a save_path=None parameter and save response to file if it's not None. Or we can just leave it to user because save a stream to file isn't something very difficult.

[deanishe]

Good find wrt zlib. I'd only looked at the gzip module, which doesn't work with the response objects from urllib.

I've tested it with a webserver and it works great. So iter_content() doesn't need replacing.

Might still be worth adding a save_to_path() method alongside it, however.

[owenwater]

save_to_path() is a really good idea, I'm 100% support it.

[fractaledmind]

Agreed.

[deanishe]

Gzip support and save_to_path() implemented in v1.9.6