kwangh / semantic-textual-similarity-tensorflow Goto Github PK

Hi Kyle29, thanks for the code. In the computation graph definitions, you have this for the words in sentences:

sentences_A=tf.placeholder(tf.int32,shape=([None,FLAGS.max_length]),name='sentences_A')
sentencesA_length=tf.placeholder(tf.int32,shape=([None]),name='sentencesA_length')

I wonder what would be the definition if you include the character level? Would it be like this?

sentences_words_A=tf.placeholder(tf.int32,shape=([None,FLAGS.max_length, max_word_len,]),name='sentences_A')
sentencesA_words_length=tf.placeholder(tf.int32,shape=([None, FLAGS.max_length]),name='sentencesA_length')

Thanks!

Hi Kyle, I wonder if the prediction calibration step in the original paper (p. 4) has been implemented in your code? I don't seem to find it in your code although you did a conversion of the input values to lie between [0,1]. Particularly, this is their description of the calibration step:

Due to the simple construction of our similarity function, the predictions of our model are constrained to follow the exp(−x) curve and are thus not suited for these evaluation metrics. After training our model, we apply an additional nonparametric regression step to obtain better-calibrated predictions (with respect to MSE). Over the training set, the given labels (under original [1, 5] scale) are regressed against the univariate MaLSTM g-predicted relatedness as the sole covariate, and the fitted regression function is evaluated on the MaLSTM-predicted relatedness of the test pairs to produce adjusted final predictions. We use the classical local-linear estimator discussed in Fan and Gijbels (1992) with bandwidth selected using leave-one-out cross-validation. This calibration step serves as a minor correction for our restrictively simple similarity function (which is necessary to retain interpretability of the sentence representations).

Cheers,
Kurt