This DatasetManager
program is designed to handle and manage large datasets in the form of zip files. It includes functionalities to extract specific files based on a given checklist, sort hexadecimal strings, and use multiprocessing to speed up the process. The program also uses logging to keep track of its operations.
- Initialization: Initialize the
DatasetManager
with paths to the dataset and a checklist. - Checklist Parsing: Read and parse the checklist file to extract required file paths.
- Zip File Listing: List all zip files present in the dataset directory.
- Folder Checking: Inspect zip files to extract folder names and sort them using hexadecimal sorting.
- Extract Specific Files: Extract specified files from zip files and save them to an output directory.
- Multiprocessing Support: Use Python's
multiprocessing
to process checklist items concurrently. - Logging: Log operations and errors for monitoring and debugging purposes.
- Environment Variable Support: Configurations can be read from environment variables using
python-dotenv
.
Ensure you have Python installed along with the necessary packages:
pip install python-dotenv
Create a .env file in the project root directory to store the configuration paths:
dataset_path=/path/to/dataset
checklist_path=/path/to/checklist.txt
output_directory=/path/to/output/directory
Initialize the Manager: The DatasetManager class is initialized with the dataset path and the checklist path.
Process the Checklist: The process_checklist function handles the process of extracting files listed in the checklist. It creates the output directory and uses multiprocessing to fasten the extraction process.
Run the Script: Execute the script using the following command:
python dataset_manager.py
Ensure that the necessary paths are correctly set in the .env file.
An example of how to format the checklist to extract a small subset of images found in the pexels-photos-janpf:
./77/77462fcca8bd602e802826eed35ef3c1.jpg
./77/772ebcc38d47a7909f51d26b5e1c4a93.jpg
./c0/c0647d367103e087d73924efafd43714.jpg
./c0/c0f37b1818b5ad6ef0c5c165c556d3fc.jpg
./c0/c0443c1ef5263b98a76fd23e6a00d577.jpg
./c0/c03006a9af22f77533c822097eb9a20e.jpg
All operations are logged in the dataset_mgr.log file, including:
- Initialization of DatasetManager
- Reading the checklist
- Listing zip files in the dataset
- Checking folders within zip files
- Reconstructing zip file names mapping
- Extracting specific files and any errors encountered during extraction
Here's an example of how to run the script programmatically:
if __name__ == "__main__":
checklistConfig = {
"dataset_path": os.getenv("dataset_path"),
"checklist_path": os.getenv("checklist_path"),
"output_directory": os.getenv("output_directory"),
}
process_checklist(checklistConfig)
Ensure that the environment variables are correctly set in your .env file, and run the script to manage your dataset efficiently.
This project is licensed under the AGPL-3.0 license. See the LICENSE file for details.
Contributions are welcome! Please create an issue or submit a pull request.
Created by Louis Del Valle (@gh:nero-dv) Feel free to reach out for any queries or support.