Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.
Home Page: https://dusty-nv.github.io/NanoLLM/
License: MIT License
On a newly flashed Jetson Orin AGX 64GB Developer Kit by the SDK Manager, docker fails to run because iptables is configured for /usr/sbin/iptables-nft resulting in the following error:
$ sudo docker run hello-world
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
Check the status of docker daemon
$ sudo systemctl status docker
× docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2024-04-15 12:07:27 PDT; 1h 52min ago
TriggeredBy: × docker.socket
Docs: https://docs.docker.com
Main PID: 2688 (code=exited, status=1/FAILURE)
CPU: 219ms
Apr 15 12:07:27 mimzy-jetson systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 15 12:07:27 mimzy-jetson systemd[1]: Stopped Docker Application Container Engine.
Apr 15 12:07:27 mimzy-jetson systemd[1]: docker.service: Start request repeated too quickly.
Apr 15 12:07:27 mimzy-jetson systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 15 12:07:27 mimzy-jetson systemd[1]: Failed to start Docker Application Container Engine.
To fix, switch to iptables-legacy
$ sudo update-alternatives --config iptables
There are 2 choices for the alternative iptables (providing /usr/sbin/iptables).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/sbin/iptables-nft 20 auto mode
1 /usr/sbin/iptables-legacy 10 manual mode
2 /usr/sbin/iptables-nft 20 manual mode
Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
Restart docker
$ sudo systemctl restart docker
Docker is now happy:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-04-15 14:02:37 PDT; 4s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 15010 (dockerd)
Tasks: 16
Memory: 38.3M
CPU: 413ms
CGroup: /system.slice/docker.service
└─15010 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.106155008-07:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf:>
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.156396528-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.156732499-07:00" level=info msg="Loading containers: start."
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.238743844-07:00" level=warning msg="Could not load necessary modules for IPSEC rules: protocol not supported"
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.351532492-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon op>
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.407781438-07:00" level=info msg="Loading containers: done."
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.434780554-07:00" level=info msg="Docker daemon" commit=60b9add containerd-snapshotter=false storage-driver=overla>
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.435385169-07:00" level=info msg="Daemon has completed initialization"
Apr 15 14:02:37 mimzy-jetson dockerd[15010]: time="2024-04-15T14:02:37.482331996-07:00" level=info msg="API listen on /run/docker.sock"
Apr 15 14:02:37 mimzy-jetson systemd[1]: Started Docker Application Container Engine.
Run hello world
$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
478afc919002: Pull complete
Digest: sha256:03b30c6a3c320ff172b52bd68eddffde6ded08ce47e650fe52de861c5e9df46d
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(arm64v8)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
Hello all,
I hope whoever reads this is doing well!! :)
So I'm trying to get this going on my Jetson Nano 8GB. I'm getting stuck (maybe?) at Quantization. I run this command, and I get the terminal output that it's quantizing the model and that this will take a while. And it seems to lock up/get stuck there? I've had it going for the past 1-1.5 hours with no further outputs or such, and the entire Jetson Nano is locked up. I can't interact with it, can't SSH into it.
Do you know if this is normal? or is something going wrong? Am I doing something wrong? I'm going to let it run for a few hours to see if it accomplishes anything.
Thanks for everyone's time!! :) . My run command inside the container :
python3 -m nano_llm.chat --api=mlc
--model Efficient-Large-Model/VILA-2.7b
--max-context-len 128
--max-new-tokens 32
Hi @dusty-nv , amazing work. I am the creator of Nous Obsidian 3B and found out that NanoLLM supports it, so thank you very much!
I just dropped nanoLLaVA which is based on Qwen1.5-0.5B and SigLIP-400M (1B params total), which I think would be awesome to have it support by NanoLLM and get it run on Jetson devices. Please let me know if I can help with anything, thank you!
I cloned the repo and created the venv virtual environment using python3.10.12. Got the following error:
(venv) bryan@mimzy-jetson:~/git/NanoLLM$ pip3 install -r requirements.txt
Requirement already satisfied: torch in ./venv/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (2.2.2)
Requirement already satisfied: torchvision in ./venv/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (0.17.2)
Requirement already satisfied: torchaudio in ./venv/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (2.2.2)
Requirement already satisfied: transformers in ./venv/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (4.39.3)
Requirement already satisfied: tvm in ./venv/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.0.0)
ERROR: Could not find a version that satisfies the requirement faiss (from versions: none)
ERROR: No matching distribution found for faiss
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.