0ssamaak0 / sirillama

Use locally running LLMs directly from Siri 🦙🟣

Siri LLama

Siri LLama is apple shortcut that access locally running LLMs through Siri or the shortcut UI on any apple device connected to the same network of your host machine. It uses Langchain 🦜🔗 and supports open source models from both Ollama 🦙 or Fireworks AI 🎆

Demo Video🎬

🆕 Multimodal support 🎬

Getting Started

Requirements

Ollama Installation🦙

1. Install Ollama for your machine, you have to run ollama serve in the terminal to start the server

2. pull the models you want to use, for example

ollama run llama3 # chat model
ollama run llava # multimodal

3. Install Langchain and Flask

pip install --upgrade --quiet  langchain langchain-community
pip install flask

4. in config.py set OLLAMA_CHAT and OLLAMA_VISUAL_CHAT to the models you pulled from Ollama

Fireworks AI Installation🎆

1.Install Langchain and Flask

pip install --upgrade --quiet  langchain langchain-fireworks
pip install flask

2. get your Fireworks API Key and put it in fireworks_models.py

3. in config.py set FIREWORKS_CHAT and vchat_model to the models you want to use from Fireworks AI. and set your and FIREWORKS_API_KEY

Config

in confing.py set MEMORY_SIZE (How many previous messages to remember) and ANSWER_SIZE_WORDS (How many words to generate in the answer)

Running SiriLlama 🟣🦙

1. Download or clone the repo

2. set the provider (Ollama / Fireworks) in app.py

3. Run the flask app using

>>> python3 app.py

4. On your Apple device, Download the shortcut from here

5. Run the shortcut through Siri or the shortcut UI, in first time you run the shortcut you will be asked to enter your IP address and the port number showing in terminal

>>> python app.py
...
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000
 * Running on http://192.168.1.134:5000
Press CTRL+C to quit

In the example above, the IP address is 192.168.1.134 and the port is 5000

Common Issues 🐞

• Even we access the flask app (not Ollama server directly), Some windows users who have Ollama installed using WSL have to make sure ollama servere is exposed to the network, Check this issue for more details
• Running the shortcut for the first time from Siri, will ask you to allow sending data to the server. it doesn't click in IOS 17.4, try to send any message from the shortcut UI to allow, then try again from Siri

Other LLM Provider 🤖🤖

Supposedly SiriLLama should work with any LLMs that including OpenAI, Claude, etc. but make sure first you installed the corresponding Langchain packages and set the models in config.py

sirillama's Issues

Can't get to work

• Always says something went wrong.
• readme says enter IP and flask port in shortcut area, but shortcut area says enter IP address, no place for port. IP:5000 didn't work.
• Where is the code or explanation for the shortcut itself?
• In the app it always says "connect to chrome, allow?" After clicking it 4-6 times it says something went wrong.

Could this still work if Ollama serve is running on another machine?

I prefere to have frontend stuff on my Macs, but Ollama running on a linux box with a 3090.
Works great for OpenWebUI etc.

interal server error 500 am not sure how to fix this

• Serving Flask app 'app'
• Debug mode: off
  WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
• Running on all addresses (0.0.0.0)
• Running on http://127.0.0.1:5000
• Running on http://192.168.50.15:5000
  Press CTRL+C to quit
  [2024-05-30 15:30:35,019] ERROR in app: Exception on / [POST]
  Traceback (most recent call last):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\flask\app.py", line 1473, in wsgi_app
  response = self.full_dispatch_request()
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\flask\app.py", line 882, in full_dispatch_request
  rv = self.handle_user_exception(e)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\flask\app.py", line 880, in full_dispatch_request
  rv = self.dispatch_request()
  ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\flask\app.py", line 865, in dispatch_request
  return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\Downloads\SiriLLama-main\SiriLLama-main\app.py", line 175, in generate_route
  response = generate(prompt)
  ^^^^^^^^^^^^^^^^
  File "C:\Users\PC\Downloads\SiriLLama-main\SiriLLama-main\app.py", line 145, in generate
  response = main_chain2.invoke({"input": user_input})
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 2399, in invoke
  input = step.invoke(
  ^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\prompts\base.py", line 128, in invoke
  return self._call_with_config(
  ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 1509, in _call_with_config
  context.run(
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\config.py", line 346, in call_func_with_variable_args
  return func(input, **kwargs) # type: ignore[call-arg]
  ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\prompts\base.py", line 111, in _format_prompt_with_error_handling
  _inner_input = self._validate_input(inner_input)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\prompts\base.py", line 103, in _validate_input
  raise KeyError(
  KeyError: "Input to ChatPromptTemplate is missing variables {'ANSWER_SIZE_WORDS'}. Expected: ['ANSWER_SIZE_WORDS', 'history', 'input'] Received: ['input', 'history']"
  192.168.50.73 - - [30/May/2024 15:30:35] "POST / HTTP/1.1" 500 -