ReLU activation with IndyLSTMCell about indrnn

[Sunnydreamrain]

Hi,
You need to add batch normalization layer to make IndRNN work. Following are two examples shown in pytorch. But I haven't done the tensorflow version yet. You can wait for Batzner's response.
https://github.com/StefOe/indrnn-pytorch/blob/master/indrnn.py#L256
https://github.com/Sunnydreamrain/IndRNN_pytorch/blob/master/cPTB/language_model.py#L127
Thanks.

[bshao001]

@Sunnydreamrain Thank you for the quick reply. I noticed that the experiments done on your publication were based on RNN (IndRNN). Can LSTM and GRU benefit from this as well?

I am interested in two factors: 1) Multiple layers to increase the model capacity; 2) The ability to identify components in a sequence (maybe).

Concerning the second, I have this observation. In the old RNN models (LSTM and GRU), if you train the following sentences:

1. My name is x1, what is my name? -> x1
2. My name is x2, what is my name? -> x2
   ...
   n. My names is xn, what is my name? -> xn

Now if you ask: My name is abc, what is my name? the model cannot correctly output abc. I have a feeling that your model may be able to do that. Have you ever done similar experiments?

Thank you again.

[Sunnydreamrain]

No. I did not do NMT experiments.
It is interesting to know that the current RNN models cannot work on the above simple problem. If it is because of not able to keep long-term memory, IndRNN can definitely work better as it can keep longer memory.
For now, I don't think it is necessary to use IndLSTM or IndGRU. One of the benefit introduced by IndRNN is the robust use of ReLU. For LSTM and GRU, the gate parameters should use sigmoid to produce an appropriate range to keep information flow. Moreover, there are much more parameters and computations in IndLSTM and IndGRU. I did do one experiment using IndLSTM at the very beginning, but as far as I recall, the performance is not significantly better than IndRNN.

Thanks.

[bshao001]

Thank you again for the reply. I am able to use relu for IndRNN without any problem (two layers). However, when I stack it to 3 layers, 4 layers, or more (I tried 3, 4, 6, 8 layers), I did not see any increase of model capacity. The model capacity is also very important to me, as I need to handle a large amount of data. Do you have any suggestions for that? Thank you again. If possible, please add me to your wechat, my ID is: bshao001_miami

[batzner]

For now, I think we can close this issue. Feel free to re-open it if any questions or issues come up.