cnap / gec-ranking Goto Github PK

Hello,

I want to get the human ranking and its concrete scores of your work.
So I followed your instruction (2. Run TrueSkill) like this:

#!/bin/bash
set -eu

if [ ! -e gec-ranking ]; then
    git clone https://github.com/cnap/gec-ranking.git
fi

if [ ! -e wmt-trueskill ]; then
    git clone https://github.com/keisks/wmt-trueskill.git
fi

cd wmt-trueskill
if [ ! -e result ]; then
    mkdir result
fi

pip2 install -r requirements.txt
cd src
cat ../../gec-ranking/data/all_judgments.csv | python2 infer_TS.py ../result/gec -n 2 -d 0 -s 2

And got the result as follows (note that I sorted them by its score):

[('CAMB', 0.10534599231875195),
 ('AMU', 0.04306970884724038),
 ('RAC', 0.020777579012258046),
 ('src', 0.002039173907479636),
 ('UFC', -0.010643493294255905),
 ('IITB', -0.06176253371780177),
 ('CUUI', -0.0891020502995365),
 ('POST', -0.10582097317332297),
 ('SJTU', -0.11394275778296624),
 ('UMC', -0.18228045159921868),
 ('PKU', -0.19595280718488897),
 ('IPN', -0.2648197969906597),
 ('NTHU', -0.3377111687837812)]

However, the human ranking mentioned in Table 2 in your paper is not identical to the above ranking.

How do I get the same human ranking as your paper?

Thank you.

The GLEU+ paper (https://arxiv.org/pdf/1605.02592.pdf) shows that the formula of precision $p_k$ is:

$$ p_n = \cfrac{ \displaystyle \sum_i^{|D|} \sum_{g_n \in h_i} \min(\rho_{i,g_n}, \eta_{i, g_n}) - \max(0, \min(\sigma_{i, g_n}, \eta_{i, g_n}) - \min(\rho_{i, g_n}, \eta_{i, g_n}))} {\displaystyle \sum_i^{|D|} \sum_{g_n \in h_i} \eta_{i, g_n} } $$

However, GLEU+ in your source code is:

$$ p_n = \cfrac{ \displaystyle \sum_i^{|D|} \max(0, \sum_{g_n \in h_i} \min(\rho_{i,g_n}, \eta_{i, g_n}) - \min(\sigma_{i, g_n}^{\mathrm{diff}}, \eta_{i, g_n})) } {\displaystyle \sum_i^{|D|} \sum_{g_n \in h_i} \eta_{i, g_n} } $$

(notation: $\sigma_{i,g_n}=\mathrm{count}(g_n \in s_i)$, $\rho_{i,g_n}=\mathrm{count}(g_n \in r_i)$, $\eta_{i,g_n}=\mathrm{count}(g_n \in h_i)$, $\sigma_{i,g_n}^{\mathrm{diff}}=\sigma_{i,g_n}$ if $\rho_{i,g_n}=0$ else $0$)

Could you provide an update on the formula explanation?