Coder Social home page Coder Social logo

fractaldb-pretrained-resnet-pytorch's Introduction

FractalDB

Summary

The repository contains a Fractal Category Search, FractalDB Construction, Pre-training, and Fine-tuning in Python/PyTorch.

The repository is based on the paper: Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura and Yutaka Satoh, "Pre-training without Natural Images", International Journal of Computer Vision (IJCV) / ACCV 2020 Best Paper Honorable Mention Award [Project] [PDF (IJCV)] [PDF (ACCV)] [Dataset] [Oral] [Poster]

Updates

Update (Mar 23, 2022)

  • The paper was accepted to International Journal of Computer Vision (IJCV). We updated the scripts and pre-trained models in the extended experiments. [PDF] [Pre-trained Models]

Update (May 22, 2021)

  • Related project "Can Vision Transformers Learn without Natural Images?" was released. We achieved to train vision transformers (ViT) without natural images. [Project] [PDF] [Code]

Update (Jan. 8, 2021)

  • Pre-training & Fine-tuning codes
  • Downloadable pre-training models [Link]
  • Multi-thread preparation with param_search/parallel_dir.py
  • Divide execution files into single-thread processing exe.sh and multi-thread processing exe_parallel.sh for FractalDB rendering.

Citation

If you use this code, please cite the following paper:

@article{KataokaIJCV2022,
  author={Kataoka, Hirokatsu and Okayasu, Kazushige and Matsumoto, Asato and Yamagata, Eisuke and Yamada, Ryosuke and Inoue, Nakamasa and Nakamura, Akio and Satoh, Yutaka},
  title={Pre-training without Natural Images},
  article={International Journal on Computer Vision (IJCV)},
  year={2022},
}

@inproceedings{KataokaACCV2020,
  author={Kataoka, Hirokatsu and Okayasu, Kazushige and Matsumoto, Asato and Yamagata, Eisuke and Yamada, Ryosuke and Inoue, Nakamasa and Nakamura, Akio and Satoh, Yutaka},
  title={Pre-training without Natural Images},
  booktitle={Asian Conference on Computer Vision (ACCV)},
  year={2020},
}

Requirements

  • Python 3.x (worked at 3.7)

  • Pytorch 1.x (worked at 1.4)

  • CUDA (worked at 10.1)

  • CuDNN (worked at 7.6)

  • Graphic board (worked at single/four NVIDIA V100)

  • Fine-tuning datasets If you would like to fine-tune on an image dataset, you must prepare conventional or self-defined datasets. [This repository] includes a downloader as an optional way. To use the following execution files exe.sh and exe_parallel.sh, you should set the downloaded CIFAR-10 dataset in ./data as the following structure.

./
  data/
    CIFAR10/
      train/
        airplane/
          0001.png
          0002.png
          ...
        ...
      val/
        airplane/
          0001.png
          0002.png
          ...
        ...

# Caution! We changed the dir name from 'test' to 'val'

Execution file

We prepared execution files exe.sh and exe_parallel.sh in the top directory. The execution file contains our recommended parameters. Please type the following commands on your environment. You can execute the Fractal Category Search, FractalDB Construction, Pre-training, and Fine-tuning.

chmod +x exe.sh
./exe.sh

For a faster execution, you shuold run the exe_parallel.sh as follows. You must adjust the thread parameter numof_thread=40 in the script depending on your computational resource.

chmod +x exe_parallel.sh
./exe_parallel.sh

Fractal Category Search

Run the code param_search/ifs_search.py to create fractal categories and their representative images. In our work, the basic parameters are --rate 0.2 --category 1000 --numof_point 100000

python param_search/ifs_search.py --rate=0.2 --category=1000 --numof_point=100000  --save_dir='./data'

The structure of directories is constructed as follows.

./
  data/
    csv_rate20_category1000/
      00000.csv
      00001.csv
      ...
    rate20_category1000/
      00000.png
      00001.png
      ...
  param_search/
  ...

FractalDB Construction

Run the code fractal_renderer/make_fractaldb.py to construct FractalDB.

python fractal_renderer/make_fractaldb.py

The code includes the following parameters.

--load_root: Category root with CSV file. You can find in "./data".
--save_root: Create the directory of FractalDB.)
--image_size_x: x-coordinate image size 
--image_size_y: y-coordinate image size
--pad_size_x: x-coordinate padding size
--pad_size_y: y-coordinate padding size
--iteration: #dot/#patch in a fractal image
--draw_type: Rendering type. You can select "{point, patch}_{gray, color}"
--weight_csv: Weight parameter. You can find "./fractal_renderer/weights"
--instance: #instance. 10 -> 1000 instances per category, 100 -> 10,000 instances per category')

The structure of rendered FractalDB is constructed as follows.

./
  data/
    FractalDB-1000/
      00000/
        00000_00_count_0_flip0.png
        00000_00_count_0_flip1.png
        00000_00_count_0_flip2.png
        00000_00_count_0_flip3.png
        ...
      00001/
        00001_00_count_0_flip0.png
        00001_00_count_0_flip1.png
        00001_00_count_0_flip2.png
        00001_00_count_0_flip3.png
        ...
  ...

Pre-training

Run the code pretraining/main.py to create a FractalDB pre-trained model.

python pretraining/main.py

Please confirm a FractalDB is existing in ./data directory. After the pre-training, a trained model is created like FractalDB-1000_resnet50_epoch90.pth and FractalDB-1000_resnet50_checkpoint.pth.tar. Moreover, you can resume the training from a checkpoint by assigning --resume parameter.

These are the important parameters in pre-training.

--dataset: model name
--path2traindb: path to FractalDB
--path2weight: path to trained weight
--resume: path to latest checkpoint
--usenet: CNN architecture
--epochs: end epoch
--numof_classes: number of pre-trained class

Pre-trained models Our pre-trained models are available in this [Link].

We have mainly prepared two different pre-trained models. These pre-trained models are trained on FractalDB in different categories (1k and 10k) and the same number of instances (1k).

FractalDB-1000_resnet50_epoch90.pth: --dataset=FractalDB-1000 --usenet=resnet50 --epochs=90 --numof_classes=1000
FractalDB-10000_resnet50_epoch90.pth: --dataset=FractalDB-10000 --usenet=resnet50 --epochs=90 --numof_classes=10000

If you would like to additionally train from the pre-trained model, you command with the next fine-tuning code as follows.

# FractalDB-1000_resnet50_epoch90.pth
python finetuning/main.py --path2db='/path/to/your/fine-tuning/data' --dataset='FractalDB-1000' --ft_dataset='YourDataset' --numof_pretrained_classes=1000 --usenet=resnet50

# FractalDB-10000_resnet50_epoch90.pth
python finetuning/main.py --path2db='/path/to/your/fine-tuning/data' --dataset='FractalDB-10000' --ft_dataset='YourDataset' --numof_pretrained_classes=10000 --usenet=resnet50

Fine-tuning

Run the code finetuning/main.py to additionally train any image datasets. However, in order to use the fine-tuning code, you must prepare a fine-tuning dataset (e.g., CIFAR-10/100, Pascal VOC 2012). Please look at Requirements for a dataset preparation and download option.

python finetuning/main.py --path2db='/path/to/your/fine-tuning/data' --ft_dataset='YourDataset'

These are the important parameters in fine-tuning.

--dataset: model name (pre-training dataset)
--ft_dataset: model name (fine-tuning dataset)
--path2db: path to fine-tuning dataset
--path2weight: path to trained weight
--resume: path to latest checkpoint
--useepoch: use epoch in pre-training model
--usenet: CNN architecture
--epochs: end epoch
--numof_pretrained_classes: num of pre-training class
--numof_classes: number of pre-trained class

Anyway, you must arrange the directories train and val under the fine-tuning dataset (or rewrite the phase in data loader DBLoader). The following dataset structure is also written in Requirements.

./
  data/
    CIFAR10/
      train/
        airplane/
          0001.png
          0002.png
          ...
        ...
      val/
        airplane/
          0001.png
          0002.png
          ...
        ...

Terms of use

The authors affiliated in National Institute of Advanced Industrial Science and Technology (AIST), Tokyo Denki University (TDU), and Tokyo Institute of Technology (TITech) are not responsible for the reproduction, duplication, copy, sale, trade, resell or exploitation for any commercial purposes, of any portion of the images and any portion of derived the data. In no event will we be also liable for any other damages resulting from this data or any derived data.

fractaldb-pretrained-resnet-pytorch's People

Contributors

hirokatsukataoka16 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

fractaldb-pretrained-resnet-pytorch's Issues

Test accuracy of FractalDB-1k

In your paper, you show the model's transform result after a train in FractalDB-1k. I am so curious about the model's top-1 test accuracy of original data (FractalDB-1k). Can you show this result?

How large is the dataset?

Hello,

How large is the dataset (in GB), and how long does it take for you to generate it? I'm trying to reproduce your results, and generation is taking a super long time, even on a machine with 64 CPUs.

Also, once you generated the dataset, how did you train your classification model?

Best,

OSError: ./weights/weights_0.1.csv not found.

Good Morning,

After finishing the Fractal Category Search I proceeded to run the code fractal_renderer/make_fractaldb.py to construct FractalDB. However, in doing so I receive an error:
$ python fractal_renderer/make_fractaldb.py
Traceback (most recent call last):
File "fractal_renderer/make_fractaldb.py", line 39, in
weights = np.genfromtxt(args.weight_csv,dtype=np.str,delimiter=',')
File "/home/noahm/RSML/lib/python3.6/site-packages/numpy/lib/npyio.py", line 1772, in genfromtxt
fid = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/home/noahm/RSML/lib/python3.6/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/home/noahm/RSML/lib/python3.6/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: ./weights/weights_0.1.csv not found.

I went to FractalDB-trained-ResNet-PyTorch/fractal_renderer/weights and found the file weights_0.1.csv was indeed there.
I am unsure of the solution to this problem.

Are pretrained weights corrupted?

Hi, I tried to load your resnet-50 pretrained weights but displayed error messages like the below.
It seems that the weight is corrupt. Do I need any preparations after downloading the weight? If really corrupted, would you please fix them?

Thanks in advance.

Python 3.8.10 (default, Jun 26 2021, 12:40:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import torch

In [2]: a = torch.load('FractalDB-1000_res50.pth')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 a = torch.load('FractalDB-1000_res50.pth')

File ~/.pyenv/versions/3.8.10/lib/python3.8/site-packages/torch/serialization.py:713, in load(f, map_location, pickle_module, **pickle_load_args)
    711             return torch.jit.load(opened_file)
    712         return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 713 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File ~/.pyenv/versions/3.8.10/lib/python3.8/site-packages/torch/serialization.py:938, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    936 assert key in deserialized_objects
    937 typed_storage = deserialized_objects[key]
--> 938 typed_storage._storage._set_from_file(
    939     f, offset, f_should_read_directly,
    940     torch._utils._element_size(typed_storage.dtype))
    941 if offset is not None:
    942     offset = f.tell()

RuntimeError: unexpected EOF, expected 2813703 more bytes. The file might be corrupted.

In [3]: torch.__version__
Out[3]: '1.11.0'

Default Learning Rate in pretraining/args.py

I am conducting a replication experiment to create a FractalDB pretrained VGG16 with TensorFlow, referencing this repository.
I have a concern regarding a hyperparameter: the default value for learning_rate in pretraining/args.py is set to 0.1, but shouldn't it be 0.01?
I checked the paper, and it mentions that the initial learning rate is 0.01. Just to be sure, I tried training for about 30 epochs with a learning rate of 0.1, but it was too high and the loss did not decrease at all.

https://github.com/hirokatsukataoka16/FractalDB-Pretrained-ResNet-PyTorch/blob/1b8fb9fe7a6f496d4a2acc4ae5e1c1ab1eab0608/pretraining/args.py#L20C5-L20C89

Thanks in advance.

Finetuning on PASCAL and Omniglot

Hello @hirokatsukataoka16,

I left this question on #1 , but since that issue is closed I realized you may have missed it. My question is about finetuning -- how did you prepare the PascalVOC and Omniglot datasets? My understanding is that PascalVOC is usually in a multi-class classification setup and Omniglot is usually a few-shot learning setup. Did you make them into single-class classification tasks, and if so how exactly did you go about it? I don't believe there is any information about it in the paper.

Thank you so much again for the paper and repo! It's a really nice idea.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.