Coder Social home page Coder Social logo

salamek / blacklist Goto Github PK

View Code? Open in Web Editor NEW
15.0 7.0 20.0 347 KB

Czech web blacklist info page (periodicaly downloads and parses MFCR pdfs and exposes data in REST API and web UI)

Home Page: https://blacklist.salamek.cz/

License: GNU General Public License v3.0

Python 70.39% Mako 0.37% JavaScript 3.85% CSS 0.36% HTML 23.83% Shell 1.19%
blacklist czech-republic pdf api-rest website

blacklist's People

Contributors

mohnish7869 avatar prateek1607 avatar salamek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

blacklist's Issues

Ubuntu 18.04 apt error: No module named 'flask_script'

apt fails to configure python3-blacklist package in Ubuntu 18.04:

Adding blacklist user
Traceback (most recent call last):
  File "/usr/bin/blacklist", line 11, in <module>
    load_entry_point('blacklist==1.0.25', 'console_scripts', 'blacklist')()
  File "/usr/lib/python3/dist-packages/blacklist/__main__.py", line 6, in main
    from blacklist.bin.blacklist import main as _main
  File "/usr/lib/python3/dist-packages/blacklist/bin/blacklist.py", line 70, in <module>
    from flask_script import Shell, Manager
ModuleNotFoundError: No module named 'flask_script'
dpkg: error processing package python3-blacklist (--configure):
 installed python3-blacklist package post-installation script subprocess returned error exit status 1
...
Errors were encountered while processing:
 python3-blacklist
E: Sub-process /usr/bin/dpkg returned an error code (1)

Stats

  • API usage
  • Blocking info
  • Grow of blocked sites in time

crawler error: Permission denied

OS: Ubuntu 18.04 unprivileged LXC container.

May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.130 13925 flask_celery.py:72] Got lock, running.
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.133 13925 blacklist.py:68] Version max: 200
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.133 13925 blacklist.py:72] Trying version 1
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.135 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13909]: D0502 10:00:00.137 13909 beat.py:227] blacklist.tasks.blacklist.crawl_blacklist sent. id->58287994-ab6e-4d5d-9a41-11b05823c83b
May  2 10:00:00 czbl blacklist[13909]: D0502 10:00:00.137 13909 beat.py:561] beat: Waking up in 5.00 minutes.
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.210 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v1.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.213 13925 blacklist.py:72] Trying version 2
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.216 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.292 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v2.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.295 13925 blacklist.py:72] Trying version 3
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.297 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.373 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v3.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.376 13925 blacklist.py:72] Trying version 4
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.378 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.446 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v4.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.450 13925 blacklist.py:72] Trying version 5
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.452 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.535 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v5.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.539 13925 blacklist.py:72] Trying version 6
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.541 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.609 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v6.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.613 13925 blacklist.py:72] Trying version 7
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.615 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.699 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v7.pdf HTTP/1.1" 200 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.704 13925 blacklist.py:72] Trying version 8
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.707 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.770 13925 connectionpool.py:396] https://www.mfcr.cz:443 "HEAD /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v8.pdf HTTP/1.1" 404 0
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.774 13925 blacklist.py:83] Error thrown for version 8 and version 7 was previously found -> using version 7
May  2 10:00:00 czbl blacklist[13906]: I0502 10:00:00.775 13925 blacklist.py:92] Found PDF https://www.mfcr.cz/assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v7.pdf
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.778 13925 connectionpool.py:826] Starting new HTTPS connection (1): www.mfcr.cz
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.873 13925 connectionpool.py:396] https://www.mfcr.cz:443 "GET /assets/cs/media/Zverejnovane-udaje-ze-Seznamu-nepovolenych-internetovych-her_v7.pdf HTTP/1.1" 200 333773
May  2 10:00:00 czbl blacklist[13906]: D0502 10:00:00.903 13925 flask_celery.py:78] Releasing lock.
May  2 10:00:00 czbl blacklist[13906]: E0502 10:00:00.905 13925 trace.py:248] Task blacklist.tasks.blacklist.crawl_blacklist[58287994-ab6e-4d5d-9a41-11b05823c83b] raised unexpected: PermissionError(13, 'Permission denied')
May  2 10:00:00 czbl blacklist[13906]: Traceback (most recent call last):
May  2 10:00:00 czbl blacklist[13906]:   File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 374, in trace_task
May  2 10:00:00 czbl blacklist[13906]:     R = retval = fun(*args, **kwargs)
May  2 10:00:00 czbl blacklist[13906]:   File "/usr/lib/python3/dist-packages/flask_celery.py", line 223, in __call__
May  2 10:00:00 czbl blacklist[13906]:     return task_base.__call__(self, *_args, **_kwargs)
May  2 10:00:00 czbl blacklist[13906]:   File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 629, in __protected_call__
May  2 10:00:00 czbl blacklist[13906]:     return self.run(*args, **kwargs)
May  2 10:00:00 czbl blacklist[13906]:   File "/usr/lib/python3/dist-packages/flask_celery.py", line 267, in wrapped
May  2 10:00:00 czbl blacklist[13906]:     ret_value = func(*args, **kwargs)
May  2 10:00:00 czbl blacklist[13906]:   File "/usr/lib/python3/dist-packages/blacklist/tasks/blacklist.py", line 106, in crawl_blacklist
May  2 10:00:00 czbl blacklist[13906]:     with open(file_path, 'wb') as f:
May  2 10:00:00 czbl blacklist[13906]: PermissionError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/blacklist/static/pdf/ad58d3f193322030f2c5ec8226ab63c417956bdbc3fa75c92dda963bee42b27b.pdf'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.