seungwonpark / randwirenn

Implementation of: "Exploring Randomly Wired Neural Networks for Image Recognition"

RandWireNN

Unofficial PyTorch Implementation of: Exploring Randomly Wired Neural Networks for Image Recognition.

Results

Validation result on Imagenet(ILSVRC2012) dataset:

Top 1 accuracy (%)	Paper	Here
RandWire-WS(4, 0.75), C=78	74.7	69.2

• (2019.06.26) 69.2%: 250 epoch with SGD optimizer, lr 0.1, momentum 0.9, weight decay 5e-5, cosine annealing lr schedule (no label smoothing applied, see loss curve below)
• (2019.04.14) 62.6%: 396k steps with SGD optimizer, lr 0.1, momentum 0.9, weigth decay 5e-5, lr decay about 0.1 at 300k
• (2019.04.12) 62.6%: 416k steps with Adabound optimizer, initial lr 0.001(decayed about 0.1 at 300k), final lr 0.1, no weight decay
• (2019.04) JiaminRen's implementation reached accuarcy which is almost close to paper, using identical training strategy with paper.
• (2019.04.10) 63.0%: 450k steps with Adam optimizer, initial lr 0.001, lr decay about 0.1 for every 150k step
• (2019.04.07) 56.8%: Training took about 16 hours on AWS p3.2xlarge(NVIDIA V100). 120k steps were done in total, and Adam optimizer with lr=0.001, batch_size=128 was used with no learning rate decay.

Dependencies

This code was tested on Python 3.6 with PyTorch 1.0.1. Other packages can be installed by:

pip install -r requirements.txt

Generate random DAG

cd model/graphs
python er.py -p 0.2 -o er-02.txt # Erdos-Renyi
python ba.py -m 7 -o ba-7.txt # Barbasi-Albert
python ws.py -k 4 -p 0.75 ws-4-075.txt # Watts-Strogatz
# number of nodes: -n option

All outputs from commands shown above will produce txt file like:

(number of nodes)
(number of edges)
(lines, each line representing edges)

Train RandWireNN

1. Download ImageNet dataset. Train/val folder should contain list of 1,000 directories, each containing list of images for corresponding category. For validation image files, this script can be useful: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh

2. Edit config.yaml

   cd config
   cp default.yaml config.yaml
   vim config.yaml # specify data directory, graph txt files

3. Train

   Note. Validation performed here won't use entire test set, since it will consume much time. (about 3 min.)

   python trainer.py -c [config yaml] -m [name]
   

4. View tensorboardX

   tensorboard --logdir ./logs
   

Validation

Run full validation:

python validation.py -c [config path] -p [checkpoint path]

This will show accuracy and average test loss of the trained model.

Author

Seungwon Park / @seungwonpark

License

Apache License 2.0

randwirenn's Issues

in_degree and out_degree

It is illustrated in figure 2 in origin paper that there are 3 input degrees and 4 output degrees in node operation. In a random graph, we can not guarantee the degree of a node. Is it just a analogy?

accuracy

oh, I try to run your code,but I don't have gpu to run ImageNet,so I runned it using tiny-ImageNet which have 200 classes,but it's accuracy up to 30%, And the test loss raised up. Can you help me

NodeOp should not have agg. weight when input_node

For input nodes of DAGlayer, NodeOp should not multiply input with learnable parameter - it should be passed by itself.
I’ll fix it when I become available. (Currently I’m on vacancy for some reason.)

Experiment setups?

Hi, I would like to ask what kind/how many GPUs you used in the experiment.

Change train/evaluation process & learning rate scheduler

Currently, training is done iteratively in while loop and is evaluated every 1000 steps. I think we'd better perform the evaluation at the end of every epoch. I will fix this.

How to change the output channels?

RandWireNN/model/dag_layer.py

Line 39 in 6bea89c

out_channel=out_channel if x in self.output_nodes else in_channel,

hi, @seungwonpark :
When I read your code, I find that you change(i.e., explode) the output channels in the output node instead of the input node. And I review Kaiming He's paper again, there is an explanation "The channel count in a random graph is increased by 2× when going from one stage to the next stage, following [11]", [11] refers to the Residual Network. So I think we should increase the output channels in the input node. How do you think?

Performance Gap

Hi! Thanks for your code. Did you find the performance between your implementation and paper?

How long to train an epoch?

Hi, I am excited to run your code. I run it on 4 TITAN Xp GPU. And yet, each epoch roughly takes 2 hours. Is that normal? What about your time to train an epoch?
Thanks!

Is it available to initial the weight zero?

RandWireNN/model/node.py

Line 13 in 6bea89c

self.agg_weight = nn.Parameter(torch.zeros(in_degree, requires_grad=True))

how to get the neural network diagram

the teaser.png, could you tell me how to realize it? could you give me the codes to generate images?

Critical: node index assignment of BA, WS should not be completely random

From the appendix of paper,

The node indexing strategies for the models are — ER: indices are assigned in a random order; BA: the initial M nodes are assigned indices 1 to M, and all other nodes are indexed following their order of adding to the graph; WS: indices are assigned sequentially in the clockwise order.

The current implementation is assigning random indices for both ER, BA, WS. Since BA/WS generated graphs should not be assigned with random node indices, this is a critical issue. Must be experimented again.

a better implementation

https://github.com/JiaminRen/RandWireNN

RandWireNN as backbone of Faster RCNN

Hi @seungwonpark Thanks for you work.

Have you realized the faster rcnn + FPN detection training of RandWireNN as backbone?