inejka / crawlviewer

About

The main purpose of this project is to give you a simple app to save a bunch of sites on your hard drive and then view them easily. Maybe later I will add the ability to start crawlers from this app.

Currently supported crawlers

• nudecrawler

Installation

Recommended (and most secure) way is using docker:

docker run --rm -v PAH_TO_YOUR_FOLDER_TO_STORE_DATA:/work -it -p 5173:5173 -p 5000:5000  inejka/crawlviewer:dev       

Then visit http://localhost:5173/

Requirements

• python
• npm

  • link to download windows installer

  • on linux install it with your package manager like this

    sudo apt install npm         
    

• wget

  • on windows easiest way is to install choco and then run

    choco install wget 
    

  • on linux install it with your package manager like this

    sudo apt install wget      
    

Windows

git clone https://github.com/Inejka/crawlviewer
cd crawlviewer
python -m venv .\server\env
.\server\env\Scripts\activate
pip install -r .\server\requirements.txt
cd client
npm install

Linux

git clone https://github.com/Inejka/crawlviewer
cd crawlviewer
python -m venv ./server/env
source server/env/bin/activate
pip install -r server/requirements.txt
cd client
npm install

Run

• windows

  ./run.bat
  

• linux

  chmod +x run.sh
  ./run.sh
  

  Then visit http://localhost:5173/

Docker

Build you own docker image via:

docker build -t user/app:tag -f docker/Dockerfile .      

crawlviewer's Issues

Test environment

Hi!
Please, fix the test environment for test_app.py. Your warning is a piece of ...

I've just erased all data from disk C. My windows system is totally crushed. So, I have to use Arch Linux because of your bad test environment (and BTW I use Arch).