Using activation function only in conv layer? about tensorflow-deconvnet-segmentation

[hungnguyen0606]

Can you please show me which paper uses the relu in deconvolution layer?

[erko]

This repository by F. Bormann (& mb others) is implementation of this paper: "Learning Deconvolution Network for Semantic Segmentation." https://arxiv.org/abs/1505.04366 (Hyeonwoo Noh et. al). In this paper there is many mentions of "activation of deconv layer", and like this line: "The output of the deconvolutional layer is an enlarged and dense activation map."
Also, I found original implementation in Caffee by authors of the paper:
https://github.com/HyeonwooNoh/DeconvNet/blob/master/model/DeconvNet/DeconvNet_inference_deploy.prototxt
They using relu after batch normalization.

[hungnguyen0606]

Thank you!
I also found the implementation details of Deconvolution Net in "Visualizing and Understanding Convolutional Networks", which states that we should use Relu fucntion after pooling layers.
However, I don't know if we should use Relu after deconvolution layers.

as I understand the paper "Visualizing and Understanding Convolutional Networks"
where is written:
[..] (ii) passing
the responses through a rectified linear function
(relu(x) = max(x, 0)); (iii) [optionally] max pooling[..]
the RELU activation is done on convolution layers; so BEFORE the optional max pooling layer.

On deconvolution first UnPooling then RELU and then deconvolution is done.

[fabianbormann]

fixed by e4d59e9