googlecloudplatform / localllm Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Lines 19 to 23 in d27376f
And:
Lines 34 to 38 in d27376f
I'm the author of https://pypi.org/project/llm/ which installs a package called llm
and a CLI tool called llm
as well. My llm
tool is similar to localllm in as much as my tool lets you execute prompts in the terminal, against both remote models and local models (using llama-cpp-python
).
As it stands using my tool and this tool in the same environment won't work, because of the namespace clash.
If you pick a different name for this you can also publish it to PyPI, which would make for a more convenient installation experience for end users.
https://llm.datasette.io/ has more about how my tool works and what it does.
Could you please check what could be wrong here?
I am running this on MAC M1
MacBook-M1 % cd localllm
-MacBook-M1 localllm % gcloud artifacts repositories create $LOCALLLM_REGISTRY
--repository-format=docker
--location=$REGION
--description="DESCRIPTION"
ERROR: (gcloud.artifacts.repositories.create) INVALID_ARGUMENT: Maven config is not supported for format "DOCKER"
I was running the standard command as given in the tutorial to this error, i tried googling this and didn't find much. Could anyone help me out?
iampoppyxx@iampoppyxx:~/new/localllm$ gcloud workstations configs create $LOCALLLM_WORKSTATION \
--region=$REGION \
--cluster=$CLUSTER \
--machine-type=e2-standard-32 \
--container-custom-image=us-central1-docker.pkg.dev/${PROJECT_ID}/${LOCALLLM_REGISTRY}/${LOCALLLM_IMAGE_NAME}:latest
When we start the process running llama-cpp-python, we provide a pipe for stderr and then promptly close it. This means if llama-cpp-python tries to write to stderr, a broken pipe exception is thrown, which for example happens if there is a prefix cache hit when processing a prompt.
#19 is a quick fix for this but it's a bit icky b/c we're still breaking stderr.
What we need to do here is:
Some ideas for how to do this:
If you run models directly using llama-cpp-python's webserver, helpful output is written to stdout and stderr about what's going on (and we even capture that output to make sure things start up correctly) but once the model is running, nothing is capturing that output and writing it anywhere.
This makes it difficult to debug when things go wrong (like #3), so we should capture this output as logs and make it available.
update point 6 in documentation to explicitly state cluster creation takes approximately 20 minutes and need to wait for it before moving forward.
pip3 install openai
pip3 install ./llm-tool/.
llm run TheBloke/Llama-2-13B-Ensemble-v5-GGUF 8000
python3 querylocal.py
Actual Result: Works!
Run python3 querylocal.py again
Fails
http://localhost:8000/v1
Traceback (most recent call last):
File "/home/username/localllm/querylocal.py", line 40, in
chat_completion = client.chat.completions.create(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_utils/_utils.py", line 271, in wrapper
return func(*args, **kwargs)
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 659, in create
return self._post(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 1200, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 889, in request
return self._request(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 965, in _request
return self._retry_request(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 1013, in _retry_request
return self._request(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 965, in _request
return self._retry_request(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 1013, in _retry_request
return self._request(
File "/home/username/miniconda3/envs/localllm/lib/python3.10/site-packages/openai/_base_client.py", line 980, in _request
raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Internal Server Error
I'll submit a PR shortly for this trivial fix.
Running llm ps
or llm kill
on my poor, tired development system resulted in:
$ llm ps
Traceback (most recent call last):
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/psutil/_psosx.py", line 352, in wrapper
return fun(self, *args, **kwargs)
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/psutil/_psosx.py", line 413, in environ
return parse_environ_block(cext.proc_environ(self.pid))
ProcessLookupError: [Errno 3] assume no such process (originated from sysctl(KERN_PROCARGS2) -> EINVAL)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jrhusney/.miniforge3/bin/llm", line 8, in <module>
sys.exit(cli())
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/llm.py", line 115, in ps
m = modelserving.running_models()
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/modelserving.py", line 39, in running_models
env = p.environ()
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/psutil/__init__.py", line 889, in environ
return self._proc.environ()
File "/Users/jrhusney/.miniforge3/lib/python3.10/site-packages/psutil/_psosx.py", line 355, in wrapper
raise ZombieProcess(self.pid, self._name, self._ppid)
psutil.ZombieProcess: PID still exists but it's a zombie (pid=2599, ppid=1071, name='launcher')
This exception needs to be caught and ignored.
Offending process looked like:
ps ax | grep 2599
2599 ?? Z 0:00.00 <defunct>
Hi All,
First of all thank you for this excellent tool which makes it very to easy to run the LLM models without any hassle.
I am aware that the main purpose of the localllm is to eliminate the dependency on GPUs and run the models using CPU. However I wanted to know if there is an option to offload the layers to the GPU.
Machine : Compute engine created in GCP
OS : Ubuntu 22.04 LTS
GPU : Tesla T4
Steps I followed thus far is as given below:
nvidia-smi
output as given belown_gpu_layers
to 4 in llama_cpp/server/settings.py
.pip install -e .
, complete step is given hereHowever I still see that the GPUs are not being utilized.
Are the above steps correct or did I miss anything here?
Thank you,
KK
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.