mbzuai-oryx / mobillama Goto Github PK

Could you modify the home activity to load the local model instead? The network speed for downloading the model is quite slow. Thank you for sharing this amazing AI project and Android app.

this is more a question than an issue.
Can one extend the context size of the model ?
I am asking because I would like to test finetuning it to longer context to see how far one can get in terms of context size with constrained resources (RTX 4090)

Great project! After I installed the apk on my Android device, I was able to run the 05B model and enter questions to get some feedback.

But I have some questions:

1. The output content seems to be somewhat irrelevant to the problem;
2. Can the code for loading local models in apk be open source;
3. Whether the model supports fine-tuning so that it can handle specific business scenarios;

When i run test code, got the error:
RuntimeError: FlashAttention only supports Ampere GPUs or newer.

Thanks for your great work!
Will you release the code and checkpoints of MobiLlama-V?
This work is very interesting and I hope to develop my work using MobiLlama-V. Thank you so much!

hello @OmkarThawakar , I used the LLM360 Analysis repo to run eval for siqa task:

python Analysis360/eval/harness/main.py --device cuda:0 --model=hf-causal-experimental --batch_size=auto:1 --model_args="pretrained=MBZUAI/MobiLlama-05B,trust_remote_code=True,dtype=bfloat16" --tasks=social_iqa --num_fewshot=0 --output_path=Analysis360-MobiLlama-05B.json

it only gives 0.3327, which is close to random numbers, since there are only three choices.

Could you share how you ran the siqa evaluation? Thanks

Thanks for your great work! In Multimodal MobiLlama of the Results section, you briefly introduce how you developed MobiLlama-V. The model seems to have a LLaVA-like architecture, but is only trained on the visual instruction tuning data, which is the potential reason that MobiLlama-V exhibits mediocre performance. Hence, my questions are the following:

1. Can you release more details about the architecture and training process of MobiLlama-V?
2. Did/Will you perform two-stage training instead of only the second stage?
3. Do you consider using ALLaVA-4V, a high-quality multimodal dataset for vision-language training? This dataset is proposed to improve the performance of small VLMs.

Thanks!