kanetw / kitsune

Next-generation paysite scraper.

Kitsune Archiver

Setup

This repo used mainly as a submodule for the main repo and is pretty hard to work on outside of it. Therefore follow its setup section first to get the files.

Development

While it is mainly used as a submodule, contributing code to this repo is easier with some local setup.

Assuming you are in the archiver folder:

1. Create config.py file:

   cp config.py.example config.py
   cp redis_map.py.example redis_map.py

2. Create a virtual environment:

   pip install virtualenv # install the package if it's not installed
   virtualenv --upgrade-embed-wheels # makes it easier to manage python versions
   virtualenv --python 3.8 venv

3. Activate the virtual environment:

   source venv/bin/activate # venv\Scripts\activate on Windows

4. Install python packages:

   pip install --requirement requirements.txt

5. Install pre-commit hooks:

   pre-commit install --install-hooks

This way you'll have all IDE juice while working on it as a submodule.

IDE-specific steps

VSCode

On windows install libmagic DLL to not crash on debugging:

pip install python-magic-bin

Install the project inside your virtual environment for Run Python File in Terminal command to work.

py -m pip install -e .

Other steps

Updating pre-commit hooks

1. Add an entry to the .pre-commit-config.yaml file
2. Update the hooks:

   pre-commit autoupdate --freeze

3. Install the hooks:

   pre-commit install --install-hooks