cocoxu / simplification Goto Github PK
View Code? Open in Web Editor NEWText Simplification System and Dataset
License: GNU General Public License v3.0
Text Simplification System and Dataset
License: GNU General Public License v3.0
Would you be able to publish the sentence-level human judgement data for calculate correlation of Automatic Metrics.
Thank you.
Thank you for making this code available! I was trying to understand how the different components of the SARI score are computed, and I wonder if I've misunderstood something or if there's an inconsistency between the code and the paper. Consider the following example.
Input: "a b"
Output: "b"
Ref-1: "a b"
Ref-2: "a"
Now if I manually compute the recall of kept tokens using Eq. 5 from the paper, I get
r_{keep}(1) = [min(0, 1) + min(1, 1/2)] / [1 + 1/2] = 1/3,
where the first terms of the numerator and denominator correspond to "a" and the second terms to "b". However, the GitHub implementation gives me
r_{keep}(1) = 1/2.
The reason is that in the code the terms of the numerator are divided individually by the corresponding denominator terms on line 58, instead of dividing the sum of the numerator terms by the sum of the denominator terms as done in Eq. 5 in the paper.
Replacing line 58 by:
keeptmpscore2 += keepgramcountergood_rep[keepgram]
and line 65 by:
keepscore_recall = keeptmpscore2 / sum(keepgramcounterall_rep.values())
seems to fix this and yield p_{keep}(1) = 1/3 as I would expect it to yield.
Have I missed something? Thanks in advance!
Hello,
Could you provide details on the pre-processing applied to the dataset? For example, which tokenizer was used (with which options)? Thank you.
The link you offered for downloading is invalid, could you please upload it again, thanks a lot!
I would like to know whether your project is just comparing the paraphrases or is generating the paraphrases?
Kindly, guide me.
Hi, I have some questions about the STAR.java, I found this script use the f_score of the deletion rather than the precision, which is different from the SARI definition.
`
double recall_del_n = 0.0 ;
if (delRefTotalNgram > 0) {
recall_del_n = delCandCorrectNgram / (double) delRefTotalNgram;
}
double f1_del_n = meanHarmonic(prec_del_n, recall_del_n);
sc += weights[n] * f1_del_n;`
Could you tell me the reason?
Also, I tested STAR on the wiki-small corpus (1 reference ) and I found that using the random sentence as the simplified output can get very high STAR (about 33.7). Is this a normal phenomenon?
In Equation 7 of the paper, my understanding is that you need to compute the precision/recall of each ngram order, and then this is averaged over the maximum order of ngrams (which is 4). Only after that, you calculate the F1 score of each operation, and then compute SARI/STAR by averaging them:
add_precision = (add_precision_1 + add_precision_2 + add_precision_3 + add_precision_4) / 4
add_recall = (add_recall_1 + add_recall_2 + add_recall_3 + add_recall_4) / 4
add_f1 = 2 * add_precision * add_recall / (add_precision + add_recall)
keep_precision = (keep_precision_1 + keep_precision_2 + keep_precision_3 + keep_precision_4) / 4
keep_recall = (keep_recall_1 + keep_recall_2 + keep_recall_3 + keep_recall_4) / 4
keep_f1 = 2 * keep_precision * keep_recall / (keep_precision + keep_recall)
del_precision = (del_precision_1 + del_precision_2 + del_precision_3 + del_precision_4) / 4
sari = (add_f1 + keep_f1 + dep_precision) / 3
However, the code follows a different procedure. There, a F1 score (for each operation) is computed for each ngram order. These are accumulated (averaged by the maximum ngram order) and divided by 3 (the number of operations) in the end.
add_f1_1 = 2 * add_precision_1 * add_recall_1 / (add_precision_1 + add_recall_1)
add_f1_2 = 2 * add_precision_2 * add_recall_2 / (add_precision_2 + add_recall_2)
add_f1_3 = 2 * add_precision_3 * add_recall_3 / (add_precision_3 + add_recall_3)
add_f1_4 = 2 * add_precision_4 * add_recall_4 / (add_precision_4 + add_recall_4)
add_1 = (add_f1_1 + add_f1_2 + add_f1_3 + add_f1_4) / 4
keep_f1_1 = 2 * keep_precision_1 * keep_recall_1 / (keep_precision_1 + keep_recall_1)
keep_f1_2 = 2 * keep_precision_2 * keep_recall_2 / (keep_precision_2 + keep_recall_2)
keep_f1_3 = 2 * keep_precision_3 * keep_recall_3 / (keep_precision_3 + keep_recall_3)
keep_f1_4 = 2 * keep_precision_4 * keep_recall_4 / (keep_precision_4 + keep_recall_4)
keep_1 = (keep_f1_1 + keep_f1_2 + keep_f1_3 + keep_f1_4) / 4
del_precision = (del_precision_1 + del_precision_2 + del_precision_3 + del_precision_4) / 4
sari = (add_f1 + keep_f1 + dep_precision) / 3
These are not mathematically equivalent, so the scores produced by both ways of calculating the metric are different. Which is the correct process then? The one in the paper or the one in the code?
Thanks for your help and clarification.
Hello @cocoxu ,
Just out of curiosity, is there a reason that turkcorpus is split in 2000 (valid) / 359 (test) samples and not 50%/50% for example?
Thank you!
Hello,
Would it be possible to have access to the data collected for the human evaluation? This means, the ratings submitted by the Turkers for Grammaticality, Meaning Preservation and Simplicity Gain for all the systems that were evaluated?
Thank you.
In SARI.py, line 107, I think it should be "if len(addgramcounterall) > 0" instead of "if len(addgramcounter) > 0"
In the readme it is mentioned
Note that STAR is corpus-level version of SARI, SARI is sentence-level.
But, I couldn't find information on STAR anywhere. So, how is it calculated or is it simply the mean of all SARI sentence scores?
Hi,
Thanks for the great work!
I have a problem with SARI score. Looks like it does not support sequence with one word for example? See the example below:
test_src = "global"
test_pred = "scenarioPattern"
test_ref = "scenarioPattern"
print(SARIsent(test_src, test_pred, [test_ref]))
the score is 0.17 rather than 1.
Is a bug or as expected?
Thank you.
I tried to use Joshua decoder with the following script:
-cand sari30it.test.output.1best -format plain -ref est.8turkers.tok.turk. -rps 8 -m SARI 4 test.8turkers.tok.norm -v 0
It seems only return SARI = 0.3125145864872819.
I wonder if I configure something wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.