prediction outputs & model classes about dnabert

[nathanhu3]

I'm having similar issues. I am following the readme instructions and getting poor evaluation and prediction results after fine tuning. I downloaded the DNABERT6 pre-trained model and ran the fine-tune command for the prom-core task under section 3.3.

I get similar behaviour as seen in @Z-Abbas' eval_results.txt where the fine-tuning will be running okay until all of a sudden the accuracy drops to 0.507 or 0.492 and is stuck there, sometimes until the end of fine-tuning, causing the final prediction accuracy to be around the same. Sometimes this behaviour will stop after about 1000-2000 steps and if it does the final prediction accuracy might be a bit better (0.7-0.85). After running fine-tuning from a fresh clone several times, the best accuracy I can achieve is ~0.85 but most attempts either finish fine-tuning with accuracies of exactly 0.507/0.492 or around 0.75.

04/06/2021 00:40:20 - INFO - __main__ -   ***** Eval results  *****
04/06/2021 00:40:20 - INFO - __main__ -     acc = 0.49248183814833585
04/06/2021 00:40:20 - INFO - __main__ -     auc = 0.8266465752924059
04/06/2021 00:40:20 - INFO - __main__ -     f1 = 0.3299750962191533
04/06/2021 00:40:20 - INFO - __main__ -     mcc = 0.0
04/06/2021 00:40:20 - INFO - __main__ -     precision = 0.24624091907416792
04/06/2021 00:40:20 - INFO - __main__ -     recall = 0.5

I am using the repo exactly (using the same dev.tsv and train.tsv). I have also noticed that the loss often doesn't improve over the 3 epochs after the first few hundred steps.

TLDR; Results from fine-tuning DNABERT several times yields poor accuracies. Often it is either exactly 0.492 or 0.507, while other times it will be between 0.7-0.85.

Thanks for your help.

[jerryji1993]

Hi,

We have recently updated the test data and performed many bug fixes. Please kindly see if the reported issue still occurs.

Thanks,
Jerry

[Zhihan1996]

Hi,

For the first question, could you please show me the command you are using when running the model.

For the second question, yes, you can use it. But in our case, BERT and Roberta essential the same. GPT is for sequence generation. So I am not sure if it makes sense to use it in the DNA setting.

[Z-Abbas]

Hi
I have a similar issue. I run the example using the commands below:

3.3 Fine-tune with pre-trained model
export KMER=6
export MODEL_PATH='/home/zeeshan/DNABERT/6-new-12w-0/'
export DATA_PATH='/home/zeeshan/DNABERT/examples/sample_data/ft/prom-core/6/'
export OUTPUT_PATH='/home/zeeshan/DNABERT/examples/OUTPUT/'

4. Prediction
   export KMER=6
   export MODEL_PATH='/home/zeeshan/DNABERT/examples/OUTPUT/'
   export DATA_PATH='/home/zeeshan/DNABERT/examples/sample_data/ft/prom-core/6/'
   export PREDICTION_PATH='/home/zeeshan/DNABERT/examples/predout/'

After running the training and prediction codes, the result of the last (100th) one is exactly the same as the prediction result. Am I doing something wrong? or is there any option to choose the best weights for prediction purpose?
Please, find the eval_results.txt file attached with the screenshot of the prediction result for your reference.

eval_results.txt

[Zhihan1996]

Hi,

I think this task is relatively tricky. We found that the model may fail to converge in some cases. So please try different hyperparameter settings and random seed.

[woreom]

Hi,

I think this task is relatively tricky. We found that the model may fail to converge in some cases. So please try different hyperparameter settings and random seed.

Can you give us an example? because personally it a bit hard for me to follow your codes, even for simplest things like loading the model and pridicting labels

[NikitaBhandare]

@jerryji1993 @Zhihan1996 I am still facing the same issue even after cloning the latest repo. I am performing a binary classification task. I downloaded the pretrained model and finetuned it on my own data but I am getting all 0 predictions and mcc value is always 0. With the data available on github initially the model does work and the accuracy is good but after few iterations the accuracy drops to 50% and mcc is 0 and I get all 0 predictions at the end. I have tried changing hyper parameters but I am unable to get good results. Can you please advice as I have my research based on this model?