yafuly / mgmo-nat

Hello! My name is Yafu Li, and I am currently a fourth-year PhD student under joint training of Zhejiang University and Westlake University, under the supervision of Prof. Yue Zhang. My research focuses on natural language generation, with a recent focus on LLM-related topics.

I am always open to discussing research, potential collaborations, or opportunities. Feel free to reach out to me at [email protected].

mgmo-nat's Issues

How to reproduce the result of the Metric-based Optimization model mentioned in the Paper?

What is your question?

We fail to reproduce the result of the Metric-based Optimization model baseline mentioned in the Paper. The code repository only contains the implement of the MgMO method.

What have you tried

Following the setting in the paper, we use metirc-based object to fine-tune the pre-trained CMLM. We pass the decoder only once to get the output distribution during training process. Then we use the function torch.multinomial() to sample 10 sentences from the output distribution to calculate the reward.

The parameters we use are shown below:

• learning rate = 5e-6
• lr scheduler = inverse_sqrt
• dropout rate = 0.3
• warmup iteration = 5000
• fintune iteration = 30000

We average 5 best checkpoints on validation set and the test command

CUDA_VISIBLE_DEVICES=0 fairseq-generate \
    $DATA  \
    --gen-subset test \
    --noise full_mask \
    --task translation_lev \
    --path $SAVE/averaged_model.pt \
    --iter-decode-with-beam 5 \
    --iter-decode-max-iter 0 \
    --iter-decode-force-max-iter \
    --iter-decode-eos-penalty 0 \
    --beam 1 --remove-bpe \
    --print-step \
    --batch-size 50 | tee $SAVE/generate.out

The final result on WMT14 EN-DE distilled dataset is
Generate test with beam=1: BLEU4 = 21.61, 50.8/27.1/16.1/9.9 (BP=1.000, ratio=1.151, syslen=74216, reflen=64506)
which is lower than the result 24.8 shown in the paper.

I would appreciate it if you can release the implement of the Metric-based Optimization model baseline!