sp1thas / scrapy-folder-tree Goto Github PK

View Code? Open in Web Editor NEW

8.0 3.0 2.0 577 KB

A scrapy pipeline which stores files using folder trees.

Home Page: https://scrapy-folder-tree.simakis.me/?utm=gh

License: MIT License

Python 100.00%

scrapy scrapy-extension scrapy-pipeline folder-structure folder-tree python3 hacktoberfest

scrapy-folder-tree's People

Contributors

Stargazers

Watchers

Forkers

simba-97 alexndrmac

scrapy-folder-tree's Issues

Run unit-tests on Python `3.11`

testing job within the build workflow should be executed on python 3.11 too. Feel free to add the extra python to the list.

Migrate documentation from readthedocs to github pages

Introduce `FOLDER_TREE_LENGTH` config parameter on `scrapy_folder_tree.FilesHashTreePipeline`

Given this filename: abcdefg.png and FOLDER_TREE_LENGTH=2

Currently, when using the scrapy_folder_tree.FilesHashTreePipeline pipeline, the file path will look like a/b/abcdefg.png.

An extra config value (FOLDER_TREE_LENGTH) should be added similar to FOLDER_TREE_LENGTH in order to allow the user to specify the folder length. For example, if FOLDER_TREE_LENGTH is equal to 2, the file path should look like ab/cd/abcdefg.png etc.

In order to support this functionality, the extra config should be added here and the filename should be split, based on the FOLDER_TREE_LENGTH here.

Recommend Projects