hunkim / deep_architecture_genealogy

Deep Learning Architecture Genealogy Project

Deep Architecture Genealogy

There are so many new models and architectures. If you find something interesting and worth paying attention to, please send us a pull requests (PR) and write issues. README.md is automatically generated. Please send PRs on the Neural Net Arch Genealogy.txt file.

Mindmap Coggle Link

https://coggle.it/diagram/Wf5mYoJbsgABUF9P

Text Version

This is automatically generated. Please send a PR on the Neural Net Arch Genealogy.txt file.

• Reinforcement Learning Algorithms
  • A3C, '16.02.06
  • DARLA, '17.07.26
  • ACTKR, '17.08.17
  • c51, '17.10.27
• CNN
  • AlexNet, '12.12
  • VggNet, '14.09
  • GoogLeNet, '14.09
  • ResNet, '15.12
  • DenseNet, '16.08
  • SENet: Squeeze-and-Excitation Networks, '17.09
  • Object Detection
    • R-CNN
    • Fast R-CNN
    • Faster R-CNN
    • Mask R-CNN
    • YOLO
    • SSD
    • R-FCN
  • Semantic Segmentation
    • FCN
    • DeconvNet
    • DeepLab
    • U-Net
  • Super-resolution
    • MemNet
    • FSRCNN
    • SRCNN
    • VDSR
    • DRCN
    • LabSRN
    • EDSR
  • TTS
    • Wavenet, '16.09.12
• Generative Models
  • Autoregressive models
    • MADE, '15.02.12
    • PixelRNN, '16.01.25
    • NADE, '16.05.07
    • PixelCNN, '16.06.16
    • PixelCNN++, '17.01.19
  • Latent variable models
    • VAE, '13.12.20
      • CVAE, '14.06.20
      • AAE, '15.11.18
      • AVB, '17.01.17
      • VQ-VAE, '17.11.2
    • GAN, '14.06.10
      • Variants
        • CGAN, '14.11.06
        • DCGAN, '15.11.19
        • infoGAN, '16.06.12
        • EBGAN, '16.09.11
        • ACGAN, '16.10.30
        • WGAN, '17.01.26
        • BEGAN, '17.02.27
        • WGAN-GP, '17.03.31
        • TripleGAN, '17.03.07
      • Applications
        • Pix2Pix, '16.11.21
        • PPGN, '16.11.30
        • StackGAN, '16.12.10
• RNN
  • LSTM, '97.11
  • GRU, 14.11
  • ACT: Adaptive Computation Time, '17.05
  • S2S: RNN Encoder-Decoder, '14.06
    • Attention: Jointly Learning to Align, '14.09
      • Effective Approaches to Attention, Luong et al. '15.08
      • DCN: Dynamic Coattention Networks, '16.08, DCN+, '17.08
      • Transformer: Attention Is All You Need, '17.06
• Capsule Net, '17.10
• Memory Networks
  • Neural Programming
    • Neural Turing Machine,'14.10
    • Neural Random-Access Machines,'16.02
    • Hierarchical Attentive Memory, '16.02
    • Neural GPUs Learn Algorithms, '16.03
    • Neural Programmer,'16.08
    • Neural Module Networks, '16.06
    • Hybrid Computing, '16.10
  • Memory Networks,'14.10
  • End-to-End Memory Network,'15.03
  • DMN: Dynamic Memory Network, '16.03, DMN+, '16.04

Contributions

Your pull requests and issues are always welcome.

deep_architecture_genealogy's Issues

Better Separate Data File and Markdown file

It's better to have a separate data file (preferrably, yaml format)
and let the python script autogenerates the markdown file.

I guess there is a data file(Neural Net Arch Genealogy.txt),
but it's not included in the repository because of the gitignore.

with open('Neural Net Arch Genealogy.txt') as fin, open('README.md', 'w') as fout:
    fout.write(head)
    for line in fin:
        tab_count = line.count('\t')
        if not tab_count:
            continue

        spaces = ['  ' for i in range(tab_count-1)]
        fout.write(''.join(spaces) + '* ' + line.replace('\t', ''))

    fout.write(tail)

Suggesstion

1. Include a data file
2. Change the data file to yaml format or json format such that it is easy to generate a markdown format

For example,

• each paper has name, date, link
• categories can have data
• categories can have another sub categories

It would look like this

---
Memory Networks:
  data:
  - name: Memory Network
    date: '14.10'
    link: https://arxiv.org/abs/1410.3916

  - name: End to End Memory Network
    date: '15.03'
    link: https://arxiv.org/pdf/1503.08895.pdf

  Neural Programming:
    data:
    - name: Neural Turing Machine
      date: '14.10'
      link: https://arxiv.org/pdf/1410.5401.pdf

    - name: Neural Random-Access Machine
      date: '16.02'
      link: https://arxiv.org/pdf/1511.06392.pdf

CNN:
  data:
  - name: Some CNNs
    date: Some Cnn Date
    link: Some CNN Link

Should it be GoogleNet instead of GoggleNet

Under CNN category, should it be GoogleNet instead of GoggleNet?

infoGAN link error

The paper link of infoGAN is linked to Improved Training of Wasserstein GANs.

Addition to Memory Networks

Neural Turing Machine (2014.10) https://arxiv.org/pdf/1410.5401.pdf

Hybrid Computing using a neural network with dynamic external memory (2016.10) https://www.nature.com/nature/journal/v538/n7626/full/nature20101.html

Neural Programmer (2016.08) https://arxiv.org/pdf/1511.04834.pdf

Papers above this sentence are pretty well-known on the category.

Neural Random-Access Machines (2016.02) https://arxiv.org/pdf/1511.06392.pdf

Neural GPUs Learn Algorithms (2016.03) https://arxiv.org/pdf/1511.08228.pdf

Learning Efficient Algorithm with Hierarchical Attentive Memory (2016.02) https://arxiv.org/abs/1602.03218

Reinforcement learning isn't a NN architecture

RL is a method of training. The actual NN used are often pretty standard feed-forward NN.

One counter-example could be value iteration networks: https://arxiv.org/abs/1602.02867

But I would classify these separately.