xicocaio / financial_web_crawler Goto Github PK

Sometimes the WSJ API returns an HTML error, which is mostly related to the number of elements per request. Then, we get the following error:

ERROR: Spider error processing <POST https://api.wsj.net/api/slinger/headlines/806/2250>
KeyError: 'Summary'

However, these errors appear mostly when the system processes all available news and so it should be treated differently. So this error seems to be caused by an empty list os news being retrieved, and so the API returns an HTML instead of an empty json.

The desired behavior is:

• The system should try decreasing the number of elements per request to distinguish from the situation where there is no more available news
• The system should go to the next stock of the list if it is verified there is no more news to retrieve
• The system must not restart when this error happens

It seems scrapy expects all loops on the same website are self-contained. So, instead of handling the stock iteration inside the main, we should change the spider so that it handles the error inside the spider and the moves to the next stock. This solution for this issue is related to #1, and should be executed simultaneously.