Coder Social home page Coder Social logo

common-crawl-downloader's People

Contributors

alumik avatar

Stargazers

avatar avatar avatar avatar avatar

Watchers

avatar

common-crawl-downloader's Issues

An error has occurred: HTTP Error 403: Forbidden

D:\anaconda\envs\common_crawl\python.exe D:/code/common-crawl-downloader-main/src/main.py
[2022-07-08 15:50:06,463] [ INFO] Fetching a new job...
[2022-07-08 15:50:06,533] [ INFO] New job fetched: {id=31, uri=crawl-data/CC-MAIN-2021-10/segments/1614178347293.1/wet/CC-MAIN-20210224165708-20210224195708-00000.warc.wet.gz}.
[2022-07-08 15:50:06,533] [ INFO] Download from https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2021-10/segments/1614178347293.1/wet/CC-MAIN-20210224165708-20210224195708-00000.warc.wet.gz
[2022-07-08 15:50:07,774] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:07,774] [ INFO] Retry after 5 seconds (10 left)).
[2022-07-08 15:50:13,987] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:13,987] [ INFO] Retry after 5 seconds (9 left)).
[2022-07-08 15:50:20,207] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:20,207] [ INFO] Retry after 5 seconds (8 left)).
[2022-07-08 15:50:26,405] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:26,405] [ INFO] Retry after 5 seconds (7 left)).
[2022-07-08 15:50:32,617] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:32,617] [ INFO] Retry after 5 seconds (6 left)).
[2022-07-08 15:50:38,813] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:38,813] [ INFO] Retry after 5 seconds (5 left)).
[2022-07-08 15:50:45,029] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:45,029] [ INFO] Retry after 5 seconds (4 left)).
[2022-07-08 15:50:51,311] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:51,311] [ INFO] Retry after 5 seconds (3 left)).
[2022-07-08 15:50:57,536] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:50:57,536] [ INFO] Retry after 5 seconds (2 left)).
[2022-07-08 15:51:03,705] [ ERROR] An error has occurred: HTTP Error 403: Forbidden
[2022-07-08 15:51:03,705] [ INFO] Retry after 5 seconds (1 left)).
[2022-07-08 15:51:10,796] [ ERROR] Job failed.
[2022-07-08 15:51:10,866] [ INFO] Fetching a new job...
[2022-07-08 15:51:10,869] [ INFO] No unclaimed job found. This program is about to exit.

Process finished with exit code 0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.