Pickling error when using background_callback with process pool executor about requests-futures

[ross]

AttributeError: Can't pickle local object 'FuturesSession.request..wrap'

Hrm. That looks like it's failing to pickle the wrapper function. I don't imagine it's possible to pickle functions, but can't say that it's something I've directly looked into.

You might look into the built in requests callback stuff, http://docs.python-requests.org/en/master/user/advanced/#event-hooks. To be honest if I had it to do over I wouldn't add the background_callback stuff here as I think those hooks would be a better way to do most of what you would do in them anyway.

[TylerADavis]

Great, thank you for the help! Using the built in hooks system I was able to get what I’d wanted to done (just background parsing of json). The one catch is I ran into issues with actually passing the information back out, I believe due to how the response is pickled and moved between processes. I ended up saying response.elapsed = response.json(). I chose this field as it was actually modifiable, and the other fields were not. Trying to add a new field to the response resulted in the data being lost when using a processpool, although it worked fine with a threadpool.

[ross]

Trying to add a new field to the response resulted in the data being lost when using a processpool, although it worked fine with a threadpool.

Funky & good to know. I think i'm going to go ahead and deprecate background_callback itself and point people towards the callbacks. Sounds like they're workable with processes and they should be a drop-in replacement and better option for threadpools.

[FJunior225]

Hello! I have the latest version and getting a similar error when trying to use your advanced example for the hooks - AttributeError: Can't pickle local object 'ElapsedFuturesSession.request..timing'

class ElapsedFuturesSession(FuturesSession):

    def request(self, method, url, hooks={}, *args, **kwargs):
        start = time()

        def timing(r, *args, **kwargs):
            r.elapsed = time() - start

        try:
            if isinstance(hooks['response'], (list, tuple)):
                # needs to be first so we don't time other hooks execution
                hooks['response'].prepend(timing)
            else:
                hooks['response'] = [timing, hooks['response']]
        except KeyError:
            hooks['response'] = timing

        return super(ElapsedFuturesSession, self) \
            .request(method, url, hooks=hooks, *args, **kwargs)

My main goal here is to track the time it takes to send a request to a downstream. I also seen another attribute error, but I believe the above issue is the main concern.
hooks['response'].prepend(timing)
AttributeError: 'list' object has no attribute 'prepend'

Thoughts? Or suggestions?

Thanks!

[ross]

AttributeError: 'list' object has no attribute 'prepend'

Too many languages with slight diffs and not enough example testing. Looks like I didn't hit that case of the code. That line should be

                    hooks['response'].insert(0, timing)

Will get the readme updated. Thanks.

[FJunior225]

Thank you!

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.