2:4 sparsity + PTQ(int8) model's inference about ao

[jcaip]

Hi @RanchiZhao yes, please see #36, which has a benchmark script and the subclasses. It would be a good idea to add a beginner tutorial as well.

[jcaip]

It should work for LLMs but the speedup characteristics depend on the matmul shapes and depending on what you are trying to do you will see more / less speedups. You will also need the model to be torch.compile traceable, as we use the torchao quantization workflow.

[jcaip]

Do you have some reference papers for PEFT + sparsity? I am also interested in that space as well, but have not been following actively.

It's impossible to say for sure without knowing the exact approach, but theoretically I believe some version of this should be possible, although accuracy is likely prohibitive. In terms of implementation though, this is not something directly supported by our APIs. You may be able to hack something together but we do not plan to add this functionality ATM. We may consider it down the line, so for anyone reading who's interested please react / +1 this comment.

[RanchiZhao]

Hi @jcaip Thanks, I saw this before but didn't get a chance to look it over carefully, i'll do it now

[RanchiZhao]

oh, another thing, is this method available in LLM like LLaMA?
And I wanna do this with Hugging Face's transformers, maybe tough to do.

[RanchiZhao]

thanks a lot, another interesting thing is that, i want do PEFT(like LoRA) on the sparse&quant model.
once we get the trained LoRA modules, we can add them into original models(the bf16 one), and do sparse&quant on it again.
now we get a "sparse&quant aware training" model, we can use it to do inference.

Is this possible?
we should make sure:

• we can put LoRA on the sparse&quant model
• merging LoRA modules into the bf16 model(instead of int8 one because of the dtype conflict) works

[RanchiZhao]

no, AFAIK no, I' ll keep finding