Generation Issue (probability tensor contains either `inf`, `nan` or element < 0) of Flash-LLaMA with Model Parallelism about alpaca_farm

[lxuechen]

Thanks for reporting! How was inputs.bin generated?

[Zhiyuan-Zeng]

Thanks for reporting! How was inputs.bin generated?

@lxuechen It is a batch of input when doing PPO. PPO feeds this batch of queries to the model to get the responses.

[Tiiiger]

Hi @Zhiyuan-Zeng ,

I don't think we ever use the huggingface "model parallelism" directly. So, Just chiming in for some thoughts.

According to https://huggingface.co/docs/accelerate/usage_guides/big_modeling,

By passing device_map="auto", we tell 🤗 Accelerate to determine automatically where to put each layer of the model depending on the available resources:

• first we use the maximum space available on the GPU(s)
• if we still need space, we store the remaining weights on the CPU
• if there is not enough RAM, we store the remaining weights on the hard drive as memory-mapped tensors

So in your case (using 80GB A100), your code should be loading the model onto the first GPU? Can you confirm if this is the case?

[Zhiyuan-Zeng]

Hi @Tiiiger,

• Yes I want to use model parallelism for my own use.
• I print the model.hf_device_map, and the result is {'model.embed_tokens': 0, 'model.layers.0': 0, 'model.layers.1': 0, 'model.layers.2': 0, 'model.layers.3': 0, 'model.layers.4': 0, 'model.layers.5': 0, 'model.layers.6': 0, 'model.layers.7': 0, 'model.layers.8': 0, 'model.layers.9': 0, 'model.layers.10': 0, 'model.layers.11': 0, 'model.layers.12': 0, 'model.layers.13': 0, 'model.layers.14': 0, 'model.layers.15': 0, 'model.layers.16': 1, 'model.layers.17': 1, 'model.layers.18': 1, 'model.layers.19': 1, 'model.layers.20': 1, 'model.layers.21': 1, 'model.layers.22': 1, 'model.layers.23': 1, 'model.layers.24': 1, 'model.layers.25': 1, 'model.layers.26': 1, 'model.layers.27': 1, 'model.layers.28': 1, 'model.layers.29': 1, 'model.layers.30': 1, 'model.layers.31': 1, 'model.norm': 1, 'lm_head': 1}. So I think the model is loaded onto the first two GPUs.

[Tiiiger]

Unfortunately we have no experience with this vanilla model parallelism and unsure why it is causing trouble.

For all of this project, we used FSDP, which we find the most useful. I would suggest you look into how we used that and see if it can help.

[Zhiyuan-Zeng]

@Tiiiger Thanks for your help! Let me close this issue.