Max lengths for code documentation about codexglue

[guody5]

Hi, thanks a lot for the wonderful repository and paper.

In the paper section B.3, it is mentioned that "We set the max length of input and inference as 256 and 64, respectively". However, on checking the code provided in the repository I find that inputs longer than 256 are truncated to a max length of 256, and output are truncated to a max length of 128.

So I was confused about what maximum length of the output should I use to reproduce the results, as the results provided in the paper and on the repository are the same, despite the difference in output lengths. I am specifically working on JAVA language.

Please follow the setting in the repo, because we have re-produced the performance of CodeBERT paper using the setting of this repo.

[rishab-32]

Great thanks a lot for the prompt reply. So does this mean setting max length 64 and 128 did not make any difference in the results of codebert?

[guody5]

Great thanks a lot for the prompt reply. So does this mean setting max length 64 and 128 did not make any difference in the results of codebert?

I don’t try 64 max length in this repo. Maybe you can try both settings.

[rishab-32]

Sure I would give it a try.

I think I have a confusion now, the results reported here https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text#result are from CodeBert paper or with the current setting in the repository? Because if I just compare the results from CodeBert paper and the results reported here (https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text#result) both are the same, although with different lengths setting for outputs.

[guody5]

Sure I would give it a try.

I think I have a confusion now, the results reported here https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text#result are from CodeBert paper or with the current setting in the repository? Because if I just compare the results from CodeBert paper and the results reported here (https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text#result) both are the same, although with different lengths setting for outputs.

This repo only provides a faster (from 30 hours to 14 hours for JAVA language) and lower GPU cost (from 4xP100 to 2xP100) pipeline compared with the repo (https://github.com/microsoft/CodeBERT) for participants to use. The results we reported in this repo directly comes from CodeBERT paper, not from the pipeline in this repo.

[rishab-32]

Great thanks, now it's clear.