xiexi1990 / setup_imagenet

Set Up ImageNet Dataset

This guide provides step-by-step instructions to set up the ImageNet dataset for your machine learning projects using two different sources: Hugging Face and Kaggle.

Method 1: Download from Hugging Face

1. Clone the ImageNet Dataset Repository

   Use the following git command to clone the ImageNet dataset. You will be prompted to enter a password; use your Hugging Face access token as the password.

   git clone https://[email protected]/datasets/imagenet-1k

   Replace HUGGINGFACE_ACCESS_TOKEN with your actual Hugging Face access token.

2. Install pigz

   sudo apt-get install pigz

3. Decompress the Dataset

   bash ./decompress.bash

4. Organize Dataset Files

   Execute the Python script to create necessary folders and move the dataset files into their respective categories.

   python create_folders_and_mv.py

Method 2: Download from Kaggle

Follow the detailed guide on Using the ImageNet Dataset with PyTorch.

Steps:

1. Just do

   bash setup_kaggle.bash