bip-rep / sherpa

A mobile Implementation of llama.cpp

Demo App for llama.cpp Model

This app is a demo of the llama.cpp model that tries to recreate an offline chatbot, working similar to OpenAI's ChatGPT. The source code for this app is available on GitHub.

Now it works with Vicuna !!!

You can use the latest models on the app.

Works on multiple devices :

Windows, mac and android ! Releases page

The app was developed using Flutter and implements ggerganov/llama.cpp, recompiled to work on mobiles. Please note that Meta officially distributes the LLaMA models, and they will not be provided by the app developers.

To run this app, you need to download the 7B llama model from Meta for research purposes. You can choose the target model (should be a xxx.bin) from the app.

Additionally, you can fine-tune the ouput with preprompts to improve its performance.

Working demo

Click on the image to view the video on YouTube. It shows a OnePlus 7 with 8Gb running Sherpa without speed up.

Usage

To use this app, follow these steps:

1. Download the ggml-model.bin from Meta for research purposes.
2. Rename the downloaded file to ggml-model.bin.
3. Place the file in your device's download folder.
4. Run the app on your mobile device.

Disclaimer

Please note that the llama.cpp models are owned and officially distributed by Meta. This app only serves as a demo for the model's capabilities and functionality. The developers of this app do not provide the LLaMA models and are not responsible for any issues related to their usage.

sherpa's Issues

Support for Linux (Debian-class or other) ?

Hello,
is there a way to build sherpa for a Linux system ?

Thanks, Emmanuel.

How does this work?!?!?!?

I mean seriously, how the f$ck did you get LLama on Android to run? :O

I tried to look at the code myself, but I don't fully understand it. I think this function is where the actual magic happens: https://github.com/Bip-Rep/sherpa/blob/main/lib/lib.dart#L337

Could you give me a brief summary on how this works and what are the most important code parts?
This will probably also help other users a lot, which will help you get more people to use and develop your app better :D

Add support for MobileLLaMA/MobileVLM models

LLM model targeted for mobile devices, available in 1.4B and 2.7B
https://github.com/Meituan-AutoML/MobileVLM

Apparently only "MobileLLaMA" is out, and not "MobileVLM"

Models links :
https://huggingface.co/mtgv/MobileLLaMA-1.4B-Base
https://huggingface.co/mtgv/MobileLLaMA-1.4B-Chat

https://huggingface.co/mtgv/MobileLLaMA-2.7B-Base
https://huggingface.co/mtgv/MobileLLaMA-2.7B-Chat

Context Is null CPU tensor

With a Google pixel 6 pro and chip Google tensor:
[isolate 17:11:10] llama loaded
[isolate 17:11:10] main found: true
[isolate 17:11:10] trying main
[isolate 17:11:10] trying main DONE Instance of "Ilama_context_params
[isolate 17:11:10] context error : unknown tensor in model file

Maybe when compiling LLama.cpp the Google tensor Need a specific flag?

Add support for k-quant latest models

libllama.so is 64-bit instead of 32-bit error when trying to load the model on my Samsung Device

Hi,

I tried using Orca mini 3B q4_0 (https://huggingface.co/TheBloke/orca_mini_3B-GGML/tree/main) model with this repo on my Samsung Galaxy M13 Android Device with 6GB Ram.

I get the following error while trying to load the model.

Failed to load dynamic library '/data/user/0/com.biprep.sherpa/app_flutter/libllama.so': dlopen failed: "/data/data/com.biprep.sherpa/app_flutter/libllama.so" is 64-bit instead of 32-bit

Have u faced this error previously? or is there a different .so file I can use on my device?

Does this support GGUF models?

Does this support GGUF models?

Model file not working

Hi,
Can you please send me the link of the model file which works in this project. All the model files I have downloaded are not working in this project.

Thanks

Support for GGMLv3?

The newest llama.cpp provides a new ggml encoding format, and no modern models support this version. Can you update the llama.cpp version? Thank you!

How to run it on Mac?

It will be great if there is a guide to show how , thanks.

Does it support GGUF (instead of GGML)?

New small models (like Mistral) are not available as GGML, but GGUF. Does Sherpa plan to support this?

[Feature Request] Add ability to edit pre-prompt (and save them)

Would be great if wr could get a way to edit and save pre-prompt.

For inspiration, I like how Faraday app does it.
You can create agents, that have specific pre-prompts, model, model settings, etc.

Then you can create a conversation, or multiple ones, with them, like it instanciate them

Useful when you switch between pre-prompts, or just topics in general.

So I can have ine agent with a sprcific pre-prompt, saying he's an IT expert for example, and another saying he's a music theory expert, or whatever.

Useful when you want to delete your history and start a new conversation too, as we cannot delete messages or rollback as of right now

This can work with Vicuna?

Hello! tks a lot for this app.
Can you make this compatible with vicuna model?

what is the folder of 'download folder'?

it asks me to put the ggml-model.bin into the download folder, i wonder what is the path of this 'download folder' in phone

Generation never starts: "context is null"

Load model,
Type hello
Press send
Open log
Log says:
[isolate 08:13:02] llama loaded
[isolate 08:13:02] main found: true
[isolate 08:13:02] trying main
[isolate 08:13:02] trying main DONE Instance of 'llama_context_params'
[isolate 08:13:02] context is null

Generation never starts

Samsung Z Flip4
8gb of ram
Snapdragon 8+ Gen1
The demo shows a oneplus device, probably an issue with Samsung phones, I'll check later on a different device.

Model file too old

I have downloaded multiple ggm-model .bin files from hugging face and they keep returning this message after trying each model...

Share link to `ggml-model.bin`

In the README it says

Download the ggml-model.bin from Meta for research purposes.

And in the Android app it says

Please download the 7B ggml-model-q4 from the official link meta provided you.

Where can we find the official link?

Is this app support Llama 2?

Has anyone tried running the Llama2 (https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML)model? After I load the model, the APP immediately crashes after sending out the prompt.

App crash on mtk 1080+ 8GB

Sherpa crashes after I ask for a question, no error messages are shown in the log
I also tried mlcchat app and it works, very slow but at least it works

It crashes

It Crashes While Generating A propmt

I am using a s21 fe with 8gb of ram and 8gb vram

how to make it faster

i installed the latest version and its a cool app but it so slow I'm running the vicuna 7b is there a way to make faster i have an 8gb ram phone and what other models does support and please link me to them

Support GGML quantitized models

They are smaller and should be faster than the original .bin file from meta/HF

for example this 7B is too huge, and in 2 .bin files ??
https://huggingface.co/psmathur/orca_mini_v3_7b/tree/main

edit : Here you can find GGMLv3 quantitized versions of this model :
https://huggingface.co/TheBloke/orca_mini_v3_7B-GGML/tree/main

Pretty sure it's not supported though, as the readme just talks of .bin files
edit : well these are bin files too, let me try

Building libllama.so file for Android

Hi,
How to build the libllama.so file for Android? The .so file in this repository is for 64 bit machines and my Android phone is 32 bit. I would like to build a so file to work for my phone. Please share the steps to build a libllama.so file.

Thanks