Error: Warmup(Generation("'bool' object has no attribute 'dtype'")) about lorax

[magdyksaleh]

@KrisWongz Can you please share the arguments to the command you used when you ran into this issue?

[KrisWongz]

Sure @magdyksaleh

docker run --gpus all
--shm-size 1g
-p 8081:80
-v /home/unionlab001/Model/qwen-32b:/data ghcr.io/predibase/lorax:latest
--model-id /data/Qwen1_5-32B-Chat
--trust-remote-code
--quantize bitsandbytes-nf4
--max-batch-prefill-tokens 1024
--max-input-length 1024
--max-total-tokens 1512
--num-shard 4 \

[tgaddair]

Looks like this may be an issue specific to Qwen + nf4 quantization. We'll take a look.