Using streaming with requests-futures about requests-futures

[ross]

Perhaps adding an example of how to do it, or else just stating that it won't work.

I wouldn't expect it to work. The way things are currently written the callback is passed the full response so would wait until everything is back before invoking it. It might be possible to rework things, maybe adding another streaming friendly callback, but I'm not 100% sure what that would look like.

[pkch]

That makes sense.

I think callback can't work with streaming, if requests-futures only uses public API of request: the only way intermediate results are exposed in the public API is through the blocking iterator iter_lines. But I'm really not experienced in this at all, so maybe I just don't see something obvious.

[ross]

Maybe if you describe your use case/reason for needing streaming in an async env I might be able to help. My intuition is that there's not a lot of cases where it'd buy much, but I wouldn't rule out that some could exist.

[pkch]

My use case is not too impressive. Just wanted to use Twitter streaming API to retrieve tweets in real-time, and then display update when something interesting happened - without blocking the main program.

[ross]

My use case is not too impressive. Just wanted to use Twitter streaming API to retrieve tweets in real-time, and then display update when something interesting happened - without blocking the main program.

Ah. In that sort of setup you'd likely be better off directly using threading.Thread to create a (background) thread that makes the requests calls and does the stream processing. The futures paradigm pretty much just gets in the way there. Good luck.