Comments (10)
Thanks, we will ensure that all agent options are available in the server
command too. For the short time being you can run the server and agent as separate containers. I'll see if I can cook up a config for RancherOS that does this.
from k3s.
This is a work in progress. The seems to initially work but I need to do more testing. We will make sure we document and have a good approach for running k3s on RancherOS, and address the agent flags.
rancher:
services:
server:
image: rancher/k3s:v0.1.0
command: server --disable-agent
environment:
- K3S_CLUSTER_SECRET=somethingrandom
volumes:
- /etc/rancher/k3s:/etc/rancher/k3s
- /var/lib/rancher/k3s:/var/lib/rancher/k3s
net: host
agent:
image: rancher/k3s:v0.1.0
command: agent --docker -s https://localhost:6443
environment:
- K3S_CLUSTER_SECRET=somethingrandom
volumes:
- /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
- /var/lib/docker:/var/lib/docker:rshared
- /var/run:/var/run:rshared
- /run:/run:rshared
privileged: true
net: host
pid: host
ipc: host
from k3s.
Thanks. I was going in that direction myself. I've tried the config, but it didn't work. With some changes, I have something that looks like a working k3s cluster, but pods go into CrashLoopBackOff with errors related to failure to create sandbox because the ResolvConfPath doesn't exist.
My service definition:
k3s-server:
image: rancher/k3s:v0.1.0
command: server --no-deploy=servicelb --disable-agent
environment:
- K3S_CLUSTER_SECRET=somethingrandom
- K3S_KUBECONFIG_OUTPUT=/var/lib/rancher/k3s/kubeconfig.yml
- K3S_KUBECONFIG_MODE=666
labels:
io.rancher.os.scope: "system"
net: host
volumes:
- /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
k3s-agent:
image: rancher/k3s:v0.1.0
command: agent --docker -s https://localhost:6443
environment:
- K3S_CLUSTER_SECRET=somethingrandom
labels:
io.rancher.os.scope: "system"
volumes:
- /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
- /var/lib/docker:/var/lib/docker:rshared
- /var/run/docker.sock:/var/run/docker.sock
privileged: true
net: host
pid: host
ipc: host
Most notable changes is that mounting /etc/rancher/k3s
in the server didn't work for some reason (I couldn't find any error messages, but it simply wasn't mounted), and mounting /var/run
and /run
fails because the mount is not a shared mount.
from k3s.
The ResolvConfPath problem was caused by me using /mnt/docker
for container storage, so unrelated to the compose file. This setup sort of works as a workaround (I have found another problem that might be a separate issue, I will investigate and possibly create a new issue for that if it really is a bug).
from k3s.
Since you are running in system-docker /etc/rancher/k3s
will not be visible inside the console. When I was testing I was running the services on user docker. I'm putting in a couple fixes to make this easier.
from k3s.
@mortenlj All agent arguments are available on the server command line now in v0.2.0-rc3. Could you please test that this works for you. Also I've tested the following two configuration for k3s in RancherOS
rancher:
services:
k3s:
image: rancher/k3s:v0.2.0-rc2
command: server --docker
volumes:
- /etc/rancher/k3s:/etc/rancher/k3s
- /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
- /var/lib/docker:/var/lib/docker:rshared
- /var/run:/var/run
- /run:/run
privileged: true
net: host
pid: host
ipc: host
And similar to above but separating agent and server
rancher:
services:
k3s-server:
image: ibuildthecloud/k3s:f90cbed-dirty-amd64
command: server --disable-agent
net: host
volumes:
- /etc/rancher/k3s:/etc/rancher/k3s
- /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
k3s-agent:
image: ibuildthecloud/k3s:f90cbed-dirty-amd64
command: agent --docker -s https://localhost:6443 --token-file /var/lib/rancher/k3s/server/node-token
volumes:
- /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
- /var/lib/docker:/var/lib/docker:rshared
- /var/run:/var/run
- /run:/run
privileged: true
net: host
pid: host
ipc: host
from k3s.
Thanks! I'll see if I get a chance to try it out tomorrow.
from k3s.
Verified with using k3s server --docker and --no-flannel that the arguments are passed to the agent.
from k3s.
I got around to testing this now, and it works fine. Thanks!
from k3s.
@mortenlj All agent arguments are available on the server command line now in v0.2.0-rc3. Could you please test that this works for you. Also I've tested the following two configuration for k3s in RancherOS
rancher: services: k3s: image: rancher/k3s:v0.2.0-rc2 command: server --docker volumes: - /etc/rancher/k3s:/etc/rancher/k3s - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared - /var/lib/docker:/var/lib/docker:rshared - /var/run:/var/run - /run:/run privileged: true net: host pid: host ipc: hostAnd similar to above but separating agent and server
rancher: services: k3s-server: image: ibuildthecloud/k3s:f90cbed-dirty-amd64 command: server --disable-agent net: host volumes: - /etc/rancher/k3s:/etc/rancher/k3s - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared k3s-agent: image: ibuildthecloud/k3s:f90cbed-dirty-amd64 command: agent --docker -s https://localhost:6443 --token-file /var/lib/rancher/k3s/server/node-token volumes: - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared - /var/lib/docker:/var/lib/docker:rshared - /var/run:/var/run - /run:/run privileged: true net: host pid: host ipc: host
When trying this and restarting the k3s(-server/-agent) service it seems to close the docker socket. Is this a k3s, crictl or kubelet problem? I've been trying to follow the code to find what would close the docker socket on exit but I can't find it.
from k3s.
Related Issues (20)
- Use private ip with external cloud manager that doesn't know about that ip
- Allow setting `priorityClassName` on ServiceLB daemonset. HOT 5
- k3s cluster should use GPU without mentioning runtimeClassName: "nvidia" in pod specs HOT 3
- Fail to run static pod without master running HOT 2
- --node-external-ip can not add annotate flannel.alpha.coreos.com/public-ip-overwrite HOT 1
- Pods are going into pending state after upgrading from v1.26.12-k3s1 to v1.27.11-k3s1 and v1.28.5-k3s1 (Issue is quite random)
- leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur HOT 4
- Update minio-go to `v7.0.70` or newer HOT 2
- Embedded helm controller does not track owner references properly
- Network policy in dual-stack cluster stops allowing ingress after pod restart HOT 14
- ingresses stop working every now and then HOT 8
- Create a new single node cluster with embedded registry mirror does not work HOT 5
- K3S startup stuck in a deadlock when a KMS provider is configured and the node is rebooted HOT 4
- Expose k3s metrics over http HOT 2
- Capturing K3s metrics over http/https from VM HOT 1
- Agent node IP not replaced with Tailscale VPN IP HOT 7
- Pods not always creating logs HOT 1
- Best practice of setting the cluster name at deployment level HOT 4
- k3s crashes if I do not start the cloud-provider
- sh: 0: cannot open –: No such file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.