Have anyone reproduced the result? about nea

[jkdufair]

I was able to replicate the results with QWKs in the .8 range. You'll need to utilize the embeddings file, as described here. Additionally, when you download the file from the link in the FAQ, the embeddings values are separated by commas but this repo expects them to be separated by spaces. I was able to accomplish this with

sed -ri ':a;s/(\ [^,]*),/\1 /;ta' embeddings.w2v.txt

@kavehtp Perhaps you want to update the FAQ to reflect this?

Thanks for making this repo available!

[DamonCC]

I also got a similar result, with Kappa scores ranging from 0.5 to 0.6. fold == 0, prompt == 1, all other parameters are default values.

How did you get the final result? As I saw in the source code, the author run 50 epochs on one fold and get the final dev score and test score. Should I run 50 epochs individually in each fold, and average their test results as the final experimental result?

[jkdufair]

I am also trying to replicate the results and getting similar outcomes. I've also tried varying the seed as described in the paper, to no avail.

[jkdufair]

Also, my understanding from the paper is that the best results used a combination of CNN & RNN (LSTM). When I was able to replicate, I passed --cnndim 50 as well. I do not believe CNN is defaulted in parameters.

[nahos]

What versions of python,theano,keras and tensorflow did you use? I am facing issues with tensorflow.

[NNNNNaaaaaa]

Also, my understanding from the paper is that the best results used a combination of CNN & RNN (LSTM). When I was able to replicate, I passed --cnndim 50 as well. I do not believe CNN is defaulted in parameters.

Even with --cnndim 50 and the embeddings file, I still get the highest QWK with 0.556 for prompt 1, fold_0. Did you use any parameters? And how did you deal with words tagged "<unk> <num> <pad>"? Thank you so much!