The crawlers from pyjobs

crawlers's People

Contributors

Stargazers

Watchers

crawlers's Issues

No logo for LinuxJobs

Hi,

On pyjobs.fr, no logo is displayed for the offers from LinuxJobs. Checking the code it seems linking to https://pbs.twimg.com/profile_images/649599776573403140/vaMrmib1_400x400.png but it does not seem to work.

If you tell me what size you need, I can upload a logo to be used by your crawler.

Regards,
Carl Chenet

Remixjobs crawler failing

Error in log when run pyjobs_web crawl command:

23:01:57,931 ERROR [scrapy.core.scraper] Spider error processing <GET https://remixjobs.com/emploi/DevOps-H-F-/36875> (referer: https://remixjobs.com/rss/python)
Traceback (most recent call last):
  File "/srv/www/pyjobs.fr/venv2.7/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/srv/www/pyjobs.fr/venv2.7/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 28, in process_spider_output
    for x in result:
  File "/srv/www/pyjobs.fr/venv2.7/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/srv/www/pyjobs.fr/venv2.7/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/srv/www/pyjobs.fr/venv2.7/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/srv/www/pyjobs.fr/venv2.7/local/lib/python2.7/site-packages/pyjobs_crawlers/spiders/remixjobs.py", line 69, in parse_job_page
    job_infos = job_node.css('ul.job-infos')[0]
IndexError: list index out of range

Bluecoders.io tags are not very well imported

Tags are imported as-is for example "recrutement développeur docker" instead of "docker". that would be nice to adapt the crawler in order to remove the useless part of tags "recrutement développeur"

Recommend Projects