error when generating w. nucleus about ctrl

[keskarnitish]

Can you post the exact command and the prompt?

[orangesandcream]

command is

!python2 generation.py --model seqlen256_v1.ckpt/model.ckpt-413000.data-00000-of-00001 --nucleus 0.9 --temperature=0.2

prompt was

Links http://www.flatearthsociety.org/articles/10-09-2004/suppression-of-our-activities-by-government

--nucleus doesn't cause the error, --temperature alone doesn't cause it, it only happens when they're combined.

[keskarnitish]

It seems that the np.exp step is overflowing because the temperature is low and the logits are large. Not sure what a good workaround here is. In this case, we could cast the logits up to fp64 which solves the problem, default to greedy (i.e., pick the highest logit), or clip the logits to artificially prevent overflow, or something else.

Seems like the cleanest solution which does not surrogate the problem is to replace

prompt_probs = np.exp(prompt_logits[_token])

by

prompt_probs = np.exp(np.float64(prompt_logits[_token]))