aub-mind / arabic-empathetic-chatbot Goto Github PK
View Code? Open in Web Editor NEWSeq2Seq-based open domain empathetic conversational model for Arabic: Dataset & Model
Seq2Seq-based open domain empathetic conversational model for Arabic: Dataset & Model
Thanks for your great work!
I have difficulty in reproducing the bleu score on the test set for the BERT2BERT model, which should be 5.58 according to your paper:
what I have done is:
train-bert2bert.ipynb
file, however, the highest bleu score I could get is less than 2.As you can see, it is far from 5.58. So could you please share with us the prediction code, as well as the values of all args, hyper-parameters (e.g. top_k, top_p, temperature, length_penalty...) that could reproduce your test result?
Thanks a lot!
can u please tell me the steps to run the code on colab ?
Hi,
Thanks for sharing the code and models for your great work!!!
I tried to reproduce the BLUE scores using your code but could not get the 5% BLUE score you reported in the paper.
Based on your code and experiments we did internally, it seems that you calculate the bleu score on the segmented text (with farasa segment). This roughly gives a BLUE score which is x2.5 times larger than what it should be. We found that in the best case (after extensive hyperparameters tuning (>40 experiments) the best BLUE score is around 1.6-1.9.
The bug is in the compute_metrics
method:
def compute_metrics(pred):
labels_ids = pred.label_ids
#pred_ids = torch.argmax(pred.predictions,dim=2)
pred_ids = pred.predictions
# all unnecessary tokens are removed
pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
labels_ids[labels_ids == -100] = tokenizer.pad_token_id
label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)
#########################################################
### You should add these lines to calculate the BLUE score correctly ######
#########################################################
response = response.replace("\'", '')
response = response.replace("[[CLS]", '')
response = response.replace("[SEP]]", '')
response = str(arabert_prep.desegment(response)) # this is the most important one
########################################
return {"bleu": round(corpus_bleu(pred_str , [label_str]).score, 4)}
Here is an example of how BLUE scores vary between the same segmented vs. desegmented text:
t1 = "أنا متأكد من أنها ستكون بخير."
ppt1 = arabert_prep.preprocess(t1)
print(ppt1) # أنا متأكد من أن +ها س+ تكون ب+ خير .
t2 = "أنا متأكد ستكون بخير."
ppt2 = arabert_prep.preprocess(t2)
print(ppt2) # أنا متأكد س+ تكون ب+ خير .
print("Wrong BLUE on segmenented text", sacrebleu.sentence_bleu(ppt2, [ppt1]).score) # 51.51425457345961
print("Correct BLUE on unsegmenented text", sacrebleu.sentence_bleu(t2, [t1]).score) # 33.51600230178196
Thanks,
First, I would like to thank you for the dataset. It did help me a lot in my thesis, good work. Second, when I run the model on Colab it crashes with output Your session crashed after using all available RAM
.
I would like to know if the model requires specific computational specs since it is not mentioned.
Best Regards.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.