whuir / dazer Goto Github PK

Hello,

Thank you for the great work. Below are the steps, I follow to run the code where I assume task = space.

1. Use https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html to formulate training data and ignore training records corresponding to categories = ['sci.space'] and ['comp.graphics']. This way, training_data_size = 10,134
2. Use https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html to get val/data data. This way, testing_data_size = 7,532
3. Set c.DAZER.train_class_num = 18 in sample.config. Rest of settings remain same.
4. Run sample-train.sh and sample-test.sh
5. Relevance score file is produced.
6. For the testing dataset, ignore document corresponding to ['comp.graphics'], mark the documents = 1 for category ['sci.space'] and mark the documents = 0 for rest of the categories.
7. Use https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html to calculate AP score for task = space where y_true is binary and y_score = relevance scores.

Following above steps, I get MAP ~ 0.050 which is way far from the reported number. Could you please let me know how did you calculate MAP scores? Additionally, please let me know if any of the above steps are incorrect. Thanks.

I just finished reading the paper and it's a great one! Very clearly written with solid experimental results.

It'll help greatly for people to try out your model if you can provide an end-to-end working example starting from publicly available word embeddings and datasets. The current code requires the user to follow a specific data format and it takes time to convert the data before feeding to the model.

Hi, Could you please give a link of the paper? Since I cannot find it any where else. Thanks.

I want to know which public dataset you are using, or can you send me the data you use, I hope to run the program correctly.Thank you.
My email [email protected]

what is positive_document and negative_document in training data format when training with 20 newsgroup dataset.