ZeroDivisionError: division by zero in AccuracyForLanguageGeneration._compute_single_pred_single_ref about jury

[NISH1001]

Re:
I found the issue. It's during the AccuracyForLanguageGeneration._tokenize(...) process which is stripping off some texts. such as when both the predictions and references are just literal '$':

[NISH1001]

Re: I was able to patch it under try/catch block here:
NISH1001@6bdf680

Should I send a PR? I don't know if need to just throw a warning or also show the original <value> for either of the pred/ref.

[devrimcavusoglu]

Hi @NISH1001, thanks for the heads-up, and also thanks for your comments, it is appreciated. I'll look into the PR asap.