Can't load HTML from series of pages about cjworkbench

[adamhooper]

Thank you for reporting this! I'll look into it. It certainly seems like you're doing everything correctly.

[adamhooper]

I duplicated the workflow and clicked "Scrape" three times. The second and third times, there was no data -- most statuses were Can't connect: None. But the first time, half the statuses were 200 OK.

I strongly suspect the IOM servers are treating these requests as nefarious and dropping them.

We at Workbench now have an ethical dilemma: should we help our users circumvent IOM's servers' restrictions? We'll have to discuss this and follow up in this issue.

In the meantime: I infer that this is a low-traffic page (maybe 1 update per day). That being the case, this workaround may be a manageable way of getting this data into Workbench:

1. In a tab, scrape the single URL "https://migration.iom.int/covid19reports" and have it alert you when new data appears. (My tests suggest scraping the single URL should work, if it's scraped rarely enough.)
2. Whenever Workbench emails you that the page has changed, visit it and add the new rows to a Google Sheet.
3. In a separate tab, import from the Google Sheet.

[arky]

Thank you @adamhooper for detailed analysis and workaround.. Am going to close this bug, as it more an issue with filtering on third-party servers.

[adamhooper]

After discussing as a team, we've decided not to wrestle with uncooperative web servers for the time being.

Also, while want to produce a more-helpful error message, that would have a high cost and a high rate of misleading error messages. (This error in particular, None, could be one of a thousand different problems. So if we were to report "you're breaking the site's terms of service," we'd be wrong 99.9% of the time.)

[arky]

Thanks @adamhooper