Load llm into free vram on multiple gpus about comfyui-exllama-nodes

[xandix1939]

Thank you, works again. You did a great job with this node as i use it very often.

[Zuellni]

Sorry for the delay, as far as I can tell auto_split is the default behavior when nothing is specified but I added an explicit gpu_split field. It should function the same way it does in exllama, eg. 8192, 4096 8, 4 will allocate 8gb to the first device and 4gb to the second one. I don't have multiple gpus to test so let me know if it works.

[Zuellni]

It seems like auto_split was using a different loading method, but I added it now. Should work by specifying auto in gpu_split.

[xandix1939]

Thanks for your update, but now the node does not work anymore when updating exllamav2. theres a torch error.

[Zuellni]

What error exactly? I moved some fields in the generator for clarity so that could be the issue if you haven't removed/readded the node. The workflow in readme is also still using the old order for now.

//Looks like there's an issue with autosplit and sequence length being lower than 2048, so I'm going to revert it for now, manual split should still work.