hardmaru / supercell

supercell

supercell

License

MIT

supercell's Issues

pytorch code

Hi, are you going to port the hyper lstm to pytorch?

orthogonal_initializer cause trouble in r0.10

First off, thank you so much for releasing the code! It would really save people tons of time!

Now the real issue:

After Tensorflow r0.10 the initializer will be expected to have a keyword argument called partition_info, so the current code would not run on TF version beyond r0.10, but it only takes a small modification to run properly. we can just replace the initializer with this:

def orthogonal_initializer(scale=1.0):
    def _initializer(shape, dtype=tf.float32, partition_info=None):
        return tf.constant(orthogonal(shape) * scale, dtype)
    return _initializer

Though I am not sure, I don't think adding this will break the code for earlier versions of TF.

Thanks and a question about init the hypercell

Thanks for you code and the design of Hypercell is amazing!
I have a question about the initialization of the cell.

self.cell= supercell.HyperLSTMCell(num_units=args.state_size.....)
output, self.last_state = tf.nn.dynamic_rnn(self.cell, self.X, initial_state=??? )

While I use the basic LSTMcell, the tf has a method to init the cell into zero state.

However the zero init perform not very good, do you have any idea of a better way to init the Hypercell?

Thanks a lot again!

Slicing Enquiry

Thanks for open-sourcing your supercell, it's really useful.

Why are you slicing based on 0:self.num_units and for hyper_state based on self.num_units:?

h = total_h[:, 0:self.num_units]
c = total_c[:, 0:self.num_units]
self.hyper_state = tf.concat(1, [total_h[:,self.num_units:], total_c[:,self.num_units:]])

Thanks!

Why adding bias to the forget gate?

supercell/supercell.py

Line 216 in 063b01e

new_c = c*tf.sigmoid(f+self.forget_bias) + tf.sigmoid(i)*g

The code implementation didn't correspond exactly to the equation we have in the layer normalization paper.
I also have doubts about normalizing all the gates, so for example, the forget gate will never be equal to zero du to the shift we add.
Isn't more logic to just keep the gates as they are and then just normalize cell state?

Thank you

what is the difference between lstm_ortho_initializer and the orthogonal_initializer?