Cudnn error cudnn_att.cpp on train_gptcu about llm.c

[gordicaleksa]

Hey @maderix !

Couple of notes:

1. Hellaswag eval not found is not really an error, that just means the script couldn't find the binary file that contains Hellaswag eval data. To fix this run the Python script inside dev/data/hellaswag.py. It will download the dataset, tokenize it and save as the bin file. That'll make that "error"/warning go away.

2. Hopefully you're on the lastest head commit since MFU logic recently changed. We use to display it always in comparison to A100's bf16 peak (312 TFlops) which means it wasn't relevant unless you're running on A100. That should be fixed now, you can find here the GPUs that are supported: https://github.com/karpathy/llm.c/blob/master/llmc/mfu.h#L39

If this solves your issue, please close it, if not here to help! :)

[ngc92]

I think 70% is pretty good, considering that the GPU also has to do other stuff except bf16 matrix multiplications

[ngc92]

there is an environment variable that can be set:
https://docs.nvidia.com/deeplearning/cudnn/latest/reference/troubleshooting.html

also, this might be the same problem as #366

[maderix]

hi @gordicaleksa @ngc92
Thanks for your help. On the latest tip I can get the training going at a much better MFU: