batch operation timeouts not handled gracefully about python

[UmanShahzad]

Hey @klmunday, thanks for reaching out.

This is indeed primarily a backend API issue which'll be fixed very soon.

From the library, you can in the meantime use the async/await handler instead. You would spawn a coroutine for each ASN, and then await them together. Something roughly like this (not tested and incomplete):

import asyncio

async def getAllAsns(asns):
    return await asyncio.gather([
        async_handler.getDetails(asn) for asn in asns
    ])

async def main():
    results = await getAllAsns(["AS1", "AS2"])
    print(results)

You mentioned possibly fixing the code around the requests.post call, but I'm not fully clear on that - do you mean that we should catch the exception and not let it go up if raise_on_fail=False, or something else?

[klmunday]

Hey @UmanShahzad

Thanks for your response and help with emulating the behaviour of the batch operation that is very useful!

My main thought behind capturing the error regarding requests.post was something similar to what you said. Although, when thinking about it, if the lookup errors at this stage then that entire bulk lookup fails, so i'm not actually sure that would be a good route to take since it would result in loss of data if only 1 item in the bulk caused the problem.

Perhaps wait and see until after the /batch endpoint fix? I'm sure that would probably be suffice

[UmanShahzad]

@klmunday Although it wouldn't solve your problem, it would be worth capturing the error from requests.post, because we send multiple requests when the total input size exceeds 1000 (the API limit), and we shouldn't throw an error in that case if raise_on_fail=False, instead returning all successful chunks. It wasn't already being captured mainly because I didn't know requests.posts would throw an exception... one of the issues with exceptions.

But yes, when /batch is fixed to handle non-existent ASNs everything will go back to normal. Apologize for the inconvenience.

[klmunday]

No worries, it does appear I was incorrect about getDetails still working for the problematic ASNs. It seems to throw the same problems. I was about to get around this by directly using requests and checking to see if the resulting status code isn't 200, if it isn't then I just skip that ASN

edit: error occurs on line https://github.com/ipinfo/python/blob/master/ipinfo/handler.py#L90

seems to throw the same exceptions as the getBatchDetails

Thanks again

[UmanShahzad]

@klmunday it should work fine now, with the bad ASN having an error key:

$ curl ipinfo.io/AS1877?token=<token>
{
  "error": "ASN Not Found!"
}

$ echo -e 'AS1877\nAS1878' | curl -XPOST --data-binary @- "https://ipinfo.io/batch?token=<token>"
{
  "AS1878": {
    "asn": "AS1878",
    ...
  },
  "AS1877": {
    "error": "ASN Not Found!"
  }
}

Let me know if it's solved and we can close this issue.

[klmunday]

Can confirm both getBatchDetails and getDetails work with this fix.

Thanks a lot!