yeonghyeon / compare_activation_function

Compare vanishing gradient problem case by case.

Compare Activation Function

The log of training process.
Two(Accuracy and Cross entropy loss) from the left.
ReLU, Sigmoid, ReLU with He initialization.

The loss graph of ReLU and Sigmoid.

The loss graph of ReLU with He initialization and without.

compare_activation_function's Issues

Theoretical Reference

This paper here provides a stronger mathematical proof about the case of both Vanishing and Exploding Gradients and their relation with activation functions and the initializers in the form of a phase transition plot called Edge of Chaos. I would highly recommend to give it a read. Link - https://arxiv.org/abs/1711.04735