Error in weight normalization code? about bnaf

[nicola-decao]

Hi, I think the line is correct. Probably I used an odd name for the variable but self._diag_weight is a vector not a matrix of the norms for the weight normalization. That is why it is multiplied after a couple of lines. The line w = torch.exp(self._weight) * self.mask_d + self._weight * self.mask_o computes the unnormalized weight matrix and then I divide by its norm and I multiply by self._diag_weight. See

BNAF/bnaf.py

Line 195 in 1a629a2

w = self._diag_weight.exp() * w / w_squared_norm.sqrt()

[johnpjust]

Agreed that it is indeed a vector and dimension-wise what i said wouldn't work, but where (prior to that line) do you actually set the diagonals of the weight matrix equal to the values actually contained in self._diag_weight so that your unnormalized weight matrix has the correct diagonal values? You initialize the diagonal weights separately in the masked_weight function (using a random_uniform distribution) from the rest of the off-diagonal weights contained in self._weight.

I may be missing something somewhere, but seems like the w_squared_norm will be wrong if you don't set the diagonals in self._weight to self._diag_weights somewhere...?

I don't program in pytorch, but in Tensorflow it should look something like this...

w = tf.multiply(tf.exp(tf.diag(self._diag_weight)), self.mask_d) + tf.multiply(self._weight, self.mask_o)

And now the w_squared_norm will be correct.