rootless-containers / usernetes

Kubernetes without the root privileges

Usernetes: Kubernetes without the root privileges (Generation 2)

Usernetes (Gen2) deploys a Kubernetes cluster inside Rootless Docker, so as to mitigate potential container-breakout vulnerabilities.

Note

Usernetes (Gen2) has significantly diverged from the original Usernetes (Gen1), which did not require Rootless Docker to be installed on hosts.

See the gen1 branch for the original Usernetes (Gen1).

Usernetes (Gen2) is similar to Rootless kind and Rootless minikube, but Usernetes (Gen 2) supports creating a cluster with multiple hosts.

Components

• Cluster configuration: kubeadm
• CRI: containerd
• OCI: runc
• CNI: Flannel

Requirements

• One of the following host operating system:

Host operating system	Minimum version
Ubuntu (recommended)	22.04
Rocky Linux	9
AlmaLinux	9
Fedora	(?)

• One of the following container engines:

Container Engine	Minimum version
Rootless Docker (recommended)	v20.10
Rootless Podman	v4.x
Rootless nerdctl	v1.6

curl -o install.sh -fsSL https://get.docker.com
sudo sh install.sh
dockerd-rootless-setuptool.sh install

• systemd lingering:

sudo loginctl enable-linger $(whoami)

• cgroup v2 delegation:

sudo mkdir -p /etc/systemd/system/[email protected]

sudo tee /etc/systemd/system/[email protected]/delegate.conf <<EOF >/dev/null
[Service]
Delegate=cpu cpuset io memory pids
EOF

sudo systemctl daemon-reload

• Kernel modules:

sudo tee /etc/modules-load.d/usernetes.conf <<EOF >/dev/null
br_netfilter
vxlan
EOF

sudo systemctl restart systemd-modules-load.service

• sysctl:

sudo tee /etc/sysctl.d/99-usernetes.conf <<EOF >/dev/null
net.ipv4.conf.default.rp_filter = 2
EOF

sudo sysctl --system

Use scripts in ./init-host for automating these steps.

Usage

See make help.

# Bootstrap a cluster
make up
make kubeadm-init
make install-flannel

# Enable kubectl
make kubeconfig
export KUBECONFIG=$(pwd)/kubeconfig
kubectl get pods -A

# Multi-host
make join-command
scp join-command another-host:~/usernetes
ssh another-host make -C ~/usernetes up kubeadm-join
make sync-external-ip

# Debug
make logs
make shell
make kubeadm-reset
make down-v
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

The container engine defaults to Docker. To change the container engine, set export CONTAINER_ENGINE=podman or export CONTAINER_ENGINE=nerdctl.

Limitations

• Node ports cannot be exposed automatically. Edit docker-compose.yaml for exposing additional node ports.
• Most of host files are not visible with hostPath mounts. Edit docker-compose.yaml for mounting additional files.
• Some volume drivers such as nfs do not work.

Advanced topics

Network

When CONTAINER_ENGINE is set to nerdctl, bypass4netns can be enabled for accelerating connect(2) syscalls. The acceleration currently does not apply to VXLAN packets.

containerd-rootless-setuptool.sh install-bypass4netnsd
export CONTAINER_ENGINE=nerdctl
make up

Misc

• Although Usernetes (Gen2) is designed to be used with Rootless Docker, it should work with the regular "rootful" Docker too. This might be useful for some people who are looking for "multi-host" version of kind and minikube.

usernetes's Issues

[k8s] expose NodePort to the host network namespace automatically

I.e. equivalent of https://github.com/moby/vpnkit/tree/master/go/cmd/kube-vpnkit-forwarder

Steps

Step 1: Add RootlessKit to NodePort management API

RootlessKit will provide implementation-agnostic REST API for exposing a port in the child netns to the parent netns, via an UNIX socket.

e.g. to expose 80/tcp as 8080/tcp in the host:

curl -X PUT -H Content-Type:application/json -d '{"childPort":80}' --unix-socket /run/user/1001/usernetes/rootlesskit/rootlesskit.sock http://v1/parentPorts/tcp/8080

Planned implementations:

builtin

Built-in implementation based on libnetwork proxy

reexec-with-cgroups

Re-exec builtin with cgroups

vpnkit-native

Uses VPNKit "VMN3T" API

slirp4netns-native

Uses slirp4netns builtin forwarder.
slirp4netns needs to be modified.

exec-privileged-iptables

Executes iptables with some SETUID/SETCAP helper.
Less secure but fast.

Step 2: Fork kube-vpnkit-forwarder into kube-rootlesskit-forwader

Step 3: Add kube-rootlesskit-forwarder to Usernetes

cc @giuseppe

[k8s] build kubelet as a static binary

bin/hyperkube:
        linux-vdso.so.1 (0x00007ffcfd788000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f04cfbc8000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f04cf82a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f04cf60b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f04cf407000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f04cf016000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f04cff56000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f04cedfe000)

[UMBRELLA] patch submission status

Kubernetes (https://github.com/rootless-containers/usernetes/tree/master/src/patches/kubernetes)

• KEP: kubernetes/enhancements#1371 (being updated)
• Patches: kubernetes/kubernetes#92863

CRI-O (https://github.com/rootless-containers/usernetes/tree/master/src/patches/crio)

• rootless: enable resource limit when cgroup v2 controllers are delegated: cri-o/cri-o#4528

[k8s] /etc/docker/certs.d/k8s.gcr.io: permission denied

The pods like pause is failing to be pulled:

tePodSandbox for pod \"pod1_default(afaf1c4e-9a2d-11e8-b119-ae906b331d15)\" failed: rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.1\": Error response from daemon: open /etc/docker/certs.d/k8s.gcr.io: permission denied"

I was able to manually run containers with the dockercli.sh script and using the vfs driver in fedora.

smoke testing with several distros (Ubuntu, Fedora, openSUSE, RHEL/CentOS, SLES...)

Probably using Vagrant

Fake cgroup v1 fs layout using FUSE (or seccomp-trap-to-userspace)

https://twitter.com/_AkihiroSuda_/status/1135154204731772928

possible backends:

• NOP
• cgroup2 unified-mode via systemd API
• pam_cgfs
• SUID helper

kube-proxy proxy-mode

usernetes/boot/kube-proxy.sh
proxy-mode can iptables or ipvs be supported ？

[crio] does not work when /etc/containers/policy.json is unavailable on the host

[kubelet-crio] E0824 07:42:07.848387   13836 pod_workers.go:186] Error syncing pod f2bae4d7-a770-11e8-8182-7a7eddbf465f ("foo-7b675c6d5-trfmv_default(f2bae4d7-a770-11e8-8182-7a7eddbf465f)"), skipping: failed to "CreatePodSandbox" for "foo-7b675c6d5-trfmv_default(f2bae4d7-a770-11e8-8182-7a7eddbf465f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"foo-7b675c6d5-trfmv_default(f2bae4d7-a770-11e8-8182-7a7eddbf465f)\" failed: rpc error: code = Unknown desc = error creating pod sandbox with name \"k8s_foo-7b675c6d5-trfmv_default_f2bae4d7-a770-11e8-8182-7a7eddbf465f_0\": open /etc/containers/policy.json: no such file or directory"

Dockerfile cannot be built without DOCKER_BUILDKIT=1

without DOCKER_BUILDKIT=1, bin/cni/* files are unexpectedly generated under bin

master components do not need to be executed inside RootlessKit

etcd and kube-{apiserver,scheduler,controller-manager} do not require privileges and should be executed as a plain old rootless user without nsentering into RootlessKit's USER+MNT+NET namespaces.

Remove Docker from Usernetes releases

dockershim might not be going to support cgroup v2: kubernetes/enhancements#1370 (comment)

[k8s] `docker run nginx` works while `kubectl run nginx --image nginx` not (port 80 EPERM)

Kubelet fails to start

hyperkube kubelet \
--cert-dir /tmp/usernetes/pki \
--root-dir /tmp/usernetes/kubelet \
--log-dir /tmp/usernetes/kubelet-log \
--volume-plugin-dir /tmp/usernetes/kubelet-plugins-exec \
--docker-endpoint unix://${XDG_RUNTIME_DIR}/docker.sock \
--kubeconfig localhost.kubeconfig \
--anonymous-auth=true \
--authorization-mode=AlwaysAllow \
--fail-swap-on=false \
--feature-gates DevicePlugins=false

[kubelet] Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[kubelet] I0723 14:03:49.502669    4922 server.go:406] Version: v1.12-usernetes
[kubelet] I0723 14:03:49.502912    4922 plugins.go:97] No cloud provider specified.
[kubelet] I0723 14:03:49.537776    4922 server.go:646] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
[kubelet] I0723 14:03:49.538427    4922 container_manager_linux.go:244] container manager verified user specified cgroup-root exists: []
[kubelet] I0723 14:03:49.538452    4922 container_manager_linux.go:249] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/tmp/usernetes/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true}
[kubelet] I0723 14:03:49.538588    4922 container_manager_linux.go:268] Creating device plugin manager: false
[kubelet] I0723 14:03:49.538627    4922 state_mem.go:36] [cpumanager] initializing new in-memory state store
[kubelet] I0723 14:03:49.538796    4922 state_mem.go:84] [cpumanager] updated default cpuset: ""
[kubelet] I0723 14:03:49.538825    4922 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
[kubelet] W0723 14:03:49.739246    4922 server.go:719] write /proc/self/oom_score_adj: permission denied
[kubelet] I0723 14:03:49.739482    4922 kubelet.go:297] Watching apiserver
[kubelet] I0723 14:03:49.749619    4922 client.go:75] Connecting to docker on unix:///run/user/1000/docker.sock
[kubelet] I0723 14:03:49.749723    4922 client.go:104] Start docker client with request timeout=2m0s
[kubelet] W0723 14:03:49.752820    4922 docker_service.go:545] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
[kubelet] I0723 14:03:49.752851    4922 docker_service.go:238] Hairpin mode set to "hairpin-veth"
[kubelet] W0723 14:03:49.752962    4922 cni.go:180] Unable to update cni config: No networks found in /etc/cni/net.d
[kubelet] W0723 14:03:49.756544    4922 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
[kubelet] W0723 14:03:49.758685    4922 plugins.go:186] can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
[kubelet] I0723 14:03:49.758755    4922 docker_service.go:253] Docker cri networking managed by kubernetes.io/no-op
[kubelet] I0723 14:03:49.765941    4922 docker_service.go:258] Docker Info: &{ID:IPII:W3CP:WJVF:WA2C:266N:Y7YJ:FD5N:UJX2:MUHX:2LQO:ZEBF:7XJ5 Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:vfs DriverStatus:[] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:false BridgeNfIP6tables:false Debug:false NFd:22 OomKillDisable:true NGoroutines:43 SystemTime:2018-07-23T14:03:49.760094599+05:30 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.17.6-200.fc28.x86_64 OperatingSystem:Fedora 28 (Workstation Edition) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc420976a80 NCPU:4 MemTotal:16724094976 GenericResources:[] DockerRootDir:/home/frezbo/.local/share/docker HTTPProxy: HTTPSProxy: NoProxy: Name:localhost.localdomain Labels:[] ExperimentalBuild:true ServerVersion:dev ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d64c661f1d51c48782c9cec8fda7604785f93587 Expected:d64c661f1d51c48782c9cec8fda7604785f93587} RuncCommit:{ID:2c632d1a2de0192c3f18a2542ccb6f30a8719b1f Expected:2c632d1a2de0192c3f18a2542ccb6f30a8719b1f} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default name=rootless]}
[kubelet] I0723 14:03:49.766018    4922 docker_service.go:271] Setting cgroupDriver to cgroupfs
[kubelet] I0723 14:03:49.778557    4922 kuberuntime_manager.go:186] Container runtime docker initialized, version: dev, apiVersion: 1.38.0
[kubelet] I0723 14:03:49.779030    4922 csi_plugin.go:110] kubernetes.io/csi: plugin initializing...
[kubelet] I0723 14:03:49.780158    4922 server.go:984] Started kubelet
[kubelet] E0723 14:03:49.781268    4922 kubelet.go:1223] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
[kubelet] I0723 14:03:49.783766    4922 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
[kubelet] I0723 14:03:49.783864    4922 status_manager.go:152] Starting to sync pod status with apiserver
[kubelet] I0723 14:03:49.783924    4922 kubelet.go:1720] Starting kubelet main sync loop.
[kubelet] I0723 14:03:49.783996    4922 kubelet.go:1737] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
[kubelet] I0723 14:03:49.784466    4922 server.go:129] Starting to listen on 0.0.0.0:10250
[kubelet] I0723 14:03:49.785396    4922 server.go:303] Adding debug handlers to kubelet server.
[kubelet] I0723 14:03:49.801132    4922 volume_manager.go:247] Starting Kubelet Volume Manager
[kubelet] I0723 14:03:49.801575    4922 desired_state_of_world_populator.go:130] Desired state populator starts to run
[kubelet] I0723 14:03:49.884386    4922 kubelet.go:1737] skipping pod synchronization - [container runtime is down]
[kubelet] I0723 14:03:49.901712    4922 kubelet_node_status.go:268] Setting node annotation to enable volume controller attach/detach
[kubelet] I0723 14:03:49.904748    4922 kubelet_node_status.go:78] Attempting to register node localhost.localdomain
[kubelet] I0723 14:03:49.910958    4922 kubelet_node_status.go:122] Node localhost.localdomain was previously registered
[kubelet] I0723 14:03:49.910993    4922 kubelet_node_status.go:81] Successfully registered node localhost.localdomain
[kubelet] I0723 14:03:49.919312    4922 kubelet_node_status.go:824] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2018-07-23 14:03:49.919290603 +0530 IST m=+0.606415439 LastTransitionTime:2018-07-23 14:03:49.919290603 +0530 IST m=+0.606415439 Reason:KubeletNotReady Message:container runtime is down}
[kubelet] I0723 14:03:50.090399    4922 kubelet.go:1737] skipping pod synchronization - [container runtime is down]
[kubelet] E0723 14:03:50.170425    4922 container_manager_linux.go:98] Unable to ensure the docker processes run in the desired containers: [errors moving "docker" pid: failed to find pid namespace of init process, errors moving "docker-containerd" pid: failed to find pid namespace of init process, errors moving "docker-containerd" pid: failed to find pid namespace of init process]
[kubelet] I0723 14:03:50.322927    4922 cpu_manager.go:155] [cpumanager] starting with none policy
[kubelet] I0723 14:03:50.322971    4922 cpu_manager.go:156] [cpumanager] reconciling every 10s
[kubelet] I0723 14:03:50.322991    4922 policy_none.go:42] [cpumanager] none policy: Start
[kubelet] F0723 14:03:50.323422    4922 kubelet.go:1295] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/tmp/usernetes/kubelet": could not find device with major: 0, minor: 45 in cached partitions map

Seems these two are not available when running under docker-rootlesskit

• write /proc/self/oom_score_adj: permission denied
• can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory

I think this is causing the kubelet to fail

OS Info

Fedora 28
~ (aws:rean-gov-sd)(kc)$ uname -am
Linux localhost.localdomain 4.17.6-200.fc28.x86_64 #1 SMP Wed Jul 11 20:29:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
~ (aws:rean-gov-sd)(kc)$ 

Use fuse-overlayfs

docs: add cfssl license

https://github.com/cloudflare/cfssl/blob/master/LICENSE

v20190603.0 regression: pod cannot be terminated

the pod status remains Terminating

Reproducible with all CRI runtimes

launch kube-apiserver with --authorization-mode=Node,RBAC

kube-apiserver should be launched with--authorization-mode=Node,RBAC
https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/1.15.3/docs/08-bootstrapping-kubernetes-controllers.md

Move AkihiroSuda/kubernetes to rootless-containers/kubernetes

When we open PRs to the kubernetes/kubernetes upstream, maybe we should squash commits. (Signed-Off-By should be kept)

set up TLS

• https://github.com/OpenVPN/easy-rsa
• https://github.com/FiloSottile/mkcert
• https://github.com/jsha/minica

[k8s] alpine containers are killed per 30 seconds; ubuntu/debian are killed immediately

Remove unused CNI plugins

[k8s] `kubectl exec`: executing setns process caused \"exit status 22\"": unknown command terminated with exit code 126

$ ./kubectl.sh exec -it foo-68d78ff5c9-bdzs5 ls
OCI runtime exec failed: exec failed: container_linux.go:336: starting container process caused "process_linux.go:86: executing setns process caused \"exit status 22\"": unknown
command terminated with exit code 126

Dockerfile.crio is broken (CNI commit is unpinned)

$ task -d build build-crio
...
[build-crio] #7 230.5 ./build.sh: 27: [: windows: unexpected operator                                                                                                        
[build-crio] #7 230.5   windows                                                                                                                                          
[build-crio] #7 230.5 can't load package: package github.com/containernetworking/plugins/plugins/main/windows: no Go files in /go/src/github.com/containernetworking/plugins/
gopath/src/github.com/containernetworking/plugins/plugins/main/windows                               
[build-crio] executor failed running [/bin/sh -c git clone https://github.com/kubernetes-incubator/cri-o.git /go/src/github.com/kubernetes-incubator/cri-o &&   cd /go/src/gi
thub.com/kubernetes-incubator/cri-o && git checkout ${CRIO_COMMIT} &&   make BUILDTAGS="exclude_graphdriver_btrfs exclude_graphdriver_devicemapper containers_image_openpgp"
binaries &&   mkdir -p /crio/cni/plugins/ /crio/cni/conf &&   cp bin/conmon bin/crio /crio &&   cp contrib/cni/* /crio/cni/conf &&   git clone https://github.com/containerne
tworking/plugins /go/src/github.com/containernetworking/plugins &&   cd /go/src/github.com/containernetworking/plugins &&   sh ./build.sh -ldflags "-extldflags -static" &&
 cp bin/* /crio/cni/plugins &&   git clone https://github.com/opencontainers/runc.git /go/src/github.com/opencontainers/runc &&   cd /go/src/github.com/opencontainers/runc &
& git checkout ${RUNC_COMMIT} &&   make BUILDTAGS="" SHELL=/bin/sh static &&   cp runc /crio/runc]: exit code: 1
[build-crio] #7    completed: 2018-10-03 11:31:02.974964005 +0000 UTC                      
[build-crio] #7     duration: 3m50.964761354s                                                                                                
[build-crio] #7        error: "executor failed running [/bin/sh -c git clone https://github.com/kubernetes-incubator/cri-o.git /go/src/github.com/kubernetes-incubator/cri-o 
&&   cd /go/src/github.com/kubernetes-incubator/cri-o && git checkout ${CRIO_COMMIT} &&   make BUILDTAGS=\"exclude_graphdriver_btrfs exclude_graphdriver_devicemapper contain
ers_image_openpgp\" binaries &&   mkdir -p /crio/cni/plugins/ /crio/cni/conf &&   cp bin/conmon bin/crio /crio &&   cp contrib/cni/* /crio/cni/conf &&   git clone https://gi
thub.com/containernetworking/plugins /go/src/github.com/containernetworking/plugins &&   cd /go/src/github.com/containernetworking/plugins &&   sh ./build.sh -ldflags \"-ext
ldflags -static\" &&   cp bin/* /crio/cni/plugins &&   git clone https://github.com/opencontainers/runc.git /go/src/github.com/opencontainers/runc &&   cd /go/src/github.com
/opencontainers/runc && git checkout ${RUNC_COMMIT} &&   make BUILDTAGS=\"\" SHELL=/bin/sh static &&   cp runc /crio/runc]: exit code: 1"
[build-crio] 

@giuseppe Could you please pin CNI commit?

Loss of DNS - because rootlesskit.sh died?

I am observing DNS errors after running docker-compose under usernetes (specifically default-docker-nokube) for "a long time". This affects all services (i.e., containers) in my composition. Bringing the composition down and back up resolves the issue - I do not have to restart dockerd.

• Is this likely an issue with my containers bringing down DNS?
• Or is it more likely a problem with the usernetes docker implementation?

For example, after running

./run.sh default-docker-nokube

for several hours (e.g., 19 hours), my DNS fails inside all docker containers.

# python -c "import socket; python -c "import socket; print(socket.gethostbyname('google-public-dns-a.google.com'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
socket.gaierror: [Errno -3] Temporary failure in name resolution

whereas it works fine outside:

$ python -c "import socket; print(socket.gethostbyname('google-public-dns-a.google.com'))"
8.8.8.8

Here is the context in which I am experiencing this:

$ uname -srvpio
Linux 4.18.0-17-generic #18~18.04.1-Ubuntu SMP Fri Mar 15 15:27:12 UTC 2019 x86_64 x86_64 GNU/Linux
$ docker-compose --version
docker-compose version 1.24.0, build 0aa5906
$ docker --version
Docker version 18.09.3-rc1, build 516e093

RFC: implement virtual kubelet provider?

During SIG-node meeting today, there was a discussion about using some virtual kubelet provider instead of modifying kubelet.
https://github.com/virtual-kubelet/virtual-kubelet

Maybe we can easily implement virtual kubelet provider with rootless CRI runtimes or even rootless Podman (I suppose implementing Podman provider is much easier), but I'm not sure about drawbacks.

Probably it doesn't work well with Services?

cc @mrunalp @giuseppe @rhatdan @ibuildthecloud @PatrickLang

CRIO regression (between May 20 and June 8)

cri-o/cri-o#3856

launch slirp4netns with --disable-host-loopback

requires rootless-containers/rootlesskit#32

set up vagrant for cgroup2 envs (and ditch docker-compose files perhaps)

The problem is that vagrant can't be used on any free CI AFAIK 😢

[k8s] implement rootless pkg/kubelet/cm in the proper way (and propose KEP to the upstream)

The current POC is implemented by just putting a bunch of non-robust rsystem.RunningInUserNS() check to a random set of functions: AkihiroSuda/kubernetes@8318759

We need to reduce these checks and implement pkg/kubelet/cm in the proper way.
Especially, pcm.Exists(existingPodName) needs to be implemented to return true even when cgroups is not available. (#17)

cc @cyphar

add cri-o

ref: cri-o/cri-o#1729

cc @giuseppe

support `docker-compose up --scale node=N`

This would need additional service for determining slirp4netns CIDR per each of node.
(RootlessKit should determine CIDR using etcd?)

lchown error pulling kube-cross

docker pull fails pulling kube-cross image. Most other images work though.

docker pull k8s.gcr.io/kube-cross:v1.11.2-1
v1.11.2-1: Pulling from kube-cross
bc9ab73e5b14: Already exists
193a6306c92a: Already exists
e5c3f8c317dc: Already exists
a587a86c9dcb: Already exists
1bc310ac474b: Already exists
87ab348d90cc: Already exists
786bc4873ebc: Already exists
251302927e9c: Pull complete
5651f8073a75: Pull complete
8a85eca14b20: Pull complete
ac20f6dc4bc0: Pull complete
683f835d2e99: Pull complete
bf4a87ea5477: Pull complete
0954fe016605: Extracting [==================================================>]  10.69MB/10.69MB
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: lchown /usr/local/src/etcd/etcd-v3.2.24-linux-amd64: invalid argument

docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 72
Server Version: dev
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b
runc version: a00bf0190895aa465a5fbed0268888e2c8ddfe85
init version: fec3683
Security Options:
 seccomp
  Profile: default
 rootless
Kernel Version: 4.18.16-300.fc29.x86_64
Operating System: Fedora 29 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.39GiB
Name: host
ID: 4C7X:657N:XBEM:NGW3:76BW:OW55:EEQE:QCZ7:W6XE:FU3Y:Y44M:3WK4
Docker Root Dir: /home/naadir/.local/share/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

sd-bus call: Permission denied

Description
Launching usernetes + enable cgroup V2, and create a pod failed.
kubectl describe the pod, show ：
Warning FailedCreatePodSandBox 9s (x4 over 30s) kubelet, izj6cgx3q19bmgalpqjydtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: sd-bus call: Permission denied: unknown

Steps to reproduce the issue:

Running Ubuntu 20.04
Linux iZj6cgx3q19bmgalpqjydtZ 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Changed grub linux arguments to support the unified hierarchy:
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1 vga=792 console=tty0 console=ttyS0,115200n8 net.ifnames=0 noibrs"

Install u7s:
./install.sh --cni=flannel --cri=containerd --publish=0.0.0.0:8472:8472/udp --publish=0.0.0.0:2379:2379/tcp --publish=0.0.0.0:6443:6443/tcp --cgroup-manager=systemd

kubectl create -f pod.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-master
spec:
  selector:
    matchLabels:
      user: test-master
  replicas: 1
  template:
    metadata:
      labels:
        user: test-master
    spec:
      containers:
      - name: busybox-master
        image: busybox
        command: ['sleep','100000000']

XDG_RUNTIME_DIR is exist：
paas@iZj6cgx3q19bmgalpqjydtZ:~/usernetes$ echo $XDG_RUNTIME_DIR
/run/user/1000

flaky test: [... [... -t fuse-overlayfs]] failed: "": wait: no child processes: unknown

+ kubectl get nodes -o wide
NAME                    STATUS   ROLES    AGE   VERSION             INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION          CONTAINER-RUNTIME
localhost.localdomain   Ready    <none>   13s   v1.19.0-usernetes   10.0.2.100    <none>        Fedora 31 (Cloud Edition)   5.3.7-301.fc31.x86_64   containerd://1.3.0-429-g1c1a08e7
+ timeout 60 kubectl run --rm -i --image busybox --restart=Never hello echo hello
pod "hello" deleted
pod default/hello terminated (StartError)
failed to create containerd task: failed to mount rootfs component &{fuse3.fuse-overlayfs overlay [workdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/work upperdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/fs lowerdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/3/fs writeback=0]}: mount helper [mount.fuse3 [overlay /run/user/1000/usernetes/containerd/io.containerd.runtime.v2.task/k8s.io/dcbe12d6759bd896b9c0a65d6f74224652110671371895a27099a1385f4e74a1/rootfs -o workdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/work -o upperdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/4/fs -o lowerdir=/home/vagrant/.local/share/usernetes/containerd/io.containerd.snapshotter.v1.fuse-overlayfs/snapshots/3/fs -o writeback=0 -t fuse-overlayfs]] failed: "": wait: no child processes: unknown
+ ERROR 'Pod is not ready.'

https://travis-ci.org/github/rootless-containers/usernetes/builds/668025122

CNI: Consider switching from Flannel to Calico (VXLAN mode)

add socat to binary distribution

The following Dockerfile does not work

FROM alpine:3.8
RUN apk add --no-cache git build-base autoconf automake
RUN git clone git://repo.or.cz/socat.git /socat
WORKDIR /socat
ARG SOCAT_COMMIT
RUN git pull && git checkout ${SOCAT_COMMIT}
RUN autoconf && ./configure LDFLAGS="-static" && make && \
  mkdir -p /out && cp -f socat /out

#8 15.77 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o xio-socket.o xio-socket.c
#8 15.82 xio-socket.c: In function 'xiocheckpeer':
#8 15.82 xio-socket.c:1713:8: warning: unused variable 'result' [-Wunused-variable]
#8 15.82     int result;
#8 15.82         ^~~~~~
#8 16.02 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o xio-interface.o xio-interface.c
#8 16.07 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o xio-listen.o xio-listen.c
#8 16.15 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o xio-unix.o xio-unix.c
#8 16.26 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o xio-ip.o xio-ip.c
#8 16.36 gcc -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o xio-ip4.o xio-ip4.c
#8 16.39 In file included from xioopen.h:10:0,
#8 16.39                  from xio-ip4.c:11:
#8 16.39 xio-ip4.c: In function 'xioparsenetwork_ip4':
#8 16.39 xio-ip4.c:49:14: error: 'NETDB_INTERNAL' undeclared (first use in this function)
#8 16.39    h_errno == NETDB_INTERNAL ? strerror(errno) :
#8 16.39               ^
#8 16.39 error.h:47:42: note: in definition of macro 'Error2'
#8 16.39  #define Error2(m,a1,a2) msg(E_ERROR,m,a1,a2)
#8 16.39                                           ^~
#8 16.39 xio-ip4.c:49:14: note: each undeclared identifier is reported only once for each function it appears in
#8 16.39    h_errno == NETDB_INTERNAL ? strerror(errno) :
#8 16.39               ^
#8 16.39 error.h:47:42: note: in definition of macro 'Error2'
#8 16.39  #define Error2(m,a1,a2) msg(E_ERROR,m,a1,a2)
#8 16.39                                           ^~
#8 16.39 make: *** [<builtin>: xio-ip4.o] Error 1
#8    completed: 2018-11-09 10:03:29.262687054 +0000 UTC
#8     duration: 16.602700111s
#8        error: "executor failed running [/bin/sh -c autoconf && ./configure LDFLAGS=\"-static\" && make &&   mkdir -p /out && cp -f socat /out]: exit code: 2"

executor failed running [/bin/sh -c autoconf && ./configure LDFLAGS="-static" && make &&   mkdir -p /out && cp -f socat /out]: exit code: 2

`systemctl --user` (allow to restart a single service)

we are currently running each service (etcd, k8s services, runtime) as part of a Taskfile.yml.

It'd be nice if we support restarting a single service, perhaps not using go-task for handling the bootstrap of Usernetes.

Perhaps systemctl --user?

The biggest problem I see is that all the services must be in the same user namespace

set up CoreDNS

via kubeadm probably #69

crio binary path broken

https://travis-ci.org/rootless-containers/usernetes/builds/444919939?utm_source=github_status&utm_medium=notification

drwxr-xr-x travis/travis     0 2018-10-23 04:18 usernetes/bin/crio/
-rwxr-xr-x travis/travis 7765512 2018-10-23 04:18 usernetes/bin/crio/runc
drwxr-xr-x travis/travis       0 2018-10-23 04:16 usernetes/bin/crio/conf/
-rw-r--r-- travis/travis     708 2018-10-23 04:16 usernetes/bin/crio/conf/README.md
-rw-r--r-- travis/travis      54 2018-10-23 04:16 usernetes/bin/crio/conf/99-loopback.conf
-rw-r--r-- travis/travis     294 2018-10-23 04:16 usernetes/bin/crio/conf/10-crio-bridge.conf
-rwxr-xr-x travis/travis 40267088 2018-10-23 04:16 usernetes/bin/crio/crio
drwxr-xr-x travis/travis        0 2018-10-23 04:17 usernetes/bin/crio/plugins/
-rwxr-xr-x travis/travis 11686112 2018-10-23 04:17 usernetes/bin/crio/plugins/dhcp
-rwxr-xr-x travis/travis  5189984 2018-10-23 04:17 usernetes/bin/crio/plugins/bridge
-rwxr-xr-x travis/travis  4745880 2018-10-23 04:17 usernetes/bin/crio/plugins/macvlan
-rwxr-xr-x travis/travis  4240480 2018-10-23 04:17 usernetes/bin/crio/plugins/host-local
-rwxr-xr-x travis/travis  4702000 2018-10-23 04:17 usernetes/bin/crio/plugins/vlan
-rwxr-xr-x travis/travis  4706552 2018-10-23 04:17 usernetes/bin/crio/plugins/ipvlan
-rwxr-xr-x travis/travis  5152856 2018-10-23 04:17 usernetes/bin/crio/plugins/ptp
-rwxr-xr-x travis/travis  4094744 2018-10-23 04:17 usernetes/bin/crio/plugins/host-device
-rwxr-xr-x travis/travis  2927370 2018-10-23 04:17 usernetes/bin/crio/plugins/flannel
-rwxr-xr-x travis/travis  2746953 2018-10-23 04:17 usernetes/bin/crio/plugins/static
-rwxr-xr-x travis/travis  4599416 2018-10-23 04:17 usernetes/bin/crio/plugins/bandwidth
-rwxr-xr-x travis/travis  4632648 2018-10-23 04:17 usernetes/bin/crio/plugins/portmap
-rwxr-xr-x travis/travis  4091368 2018-10-23 04:17 usernetes/bin/crio/plugins/tuning
-rwxr-xr-x travis/travis  4031872 2018-10-23 04:17 usernetes/bin/crio/plugins/loopback
-rwxr-xr-x travis/travis  2716920 2018-10-23 04:17 usernetes/bin/crio/plugins/sample
-rwxr-xr-x travis/travis    55432 2018-10-23 04:16 usernetes/bin/crio/conmon

plugins and conf were expected to be located under the usernetes/bin/crio/cni

Weirdly, the issue does not happen for my local artifact (built with docker/engine@bc4c1c2 , DOCKER_BUILDKIT=1)

Proxy sd_notify() with MAINPID conversion

MAINPID needs to be converted across the PID namespace boundary

port kubeadm

[crio] build crio as a static binary

UPDATE: now crio is dynamically linked with glibc instead of musl libc

$ ldd bin/crio/crio 
        linux-vdso.so.1 (0x00007ffd1c117000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdf5c866000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdf5c662000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdf5c271000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdf5ca85000)

---

(previously crio was unexpectedly linked with musl libc)

penguin01@suda-ws01:/tmp/usernetes$ find bin -executable -type f | xargs ldd
bin/docker-proxy:
        not a dynamic executable
bin/dockerd:
        not a dynamic executable
bin/docker:
        not a dynamic executable
bin/docker-containerd-shim:
        not a dynamic executable
bin/docker-runc:
        not a dynamic executable
bin/docker-init:
        not a dynamic executable
bin/task:
        not a dynamic executable
bin/docker-containerd-ctr:
        not a dynamic executable
bin/hyperkube:
        linux-vdso.so.1 (0x00007ffcfd788000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f04cfbc8000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f04cf82a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f04cf60b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f04cf407000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f04cf016000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f04cff56000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f04cedfe000)
bin/rootlesskit:
        not a dynamic executable
bin/etcdctl:
        not a dynamic executable
bin/etcd:
        not a dynamic executable
bin/docker-containerd:
        not a dynamic executable
bin/crio/runc:
        not a dynamic executable
bin/crio/conmon:
        statically linked
bin/crio/cni/plugins/static:
        linux-vdso.so.1 (0x00007ffe7d7ca000)
        libc.musl-x86_64.so.1 => not found
bin/crio/cni/plugins/vlan:
        not a dynamic executable
bin/crio/cni/plugins/bandwidth:
        not a dynamic executable
bin/crio/cni/plugins/ptp:
        not a dynamic executable
bin/crio/cni/plugins/loopback:
        not a dynamic executable
bin/crio/cni/plugins/ipvlan:
        not a dynamic executable
bin/crio/cni/plugins/host-local:
        not a dynamic executable
bin/crio/cni/plugins/dhcp:
        not a dynamic executable
bin/crio/cni/plugins/bridge:
        not a dynamic executable
bin/crio/cni/plugins/tuning:
        not a dynamic executable
bin/crio/cni/plugins/host-device:
        not a dynamic executable
bin/crio/cni/plugins/sample:
        linux-vdso.so.1 (0x00007fff39ac6000)
        libc.musl-x86_64.so.1 => not found
bin/crio/cni/plugins/macvlan:
        not a dynamic executable
bin/crio/cni/plugins/flannel:
        linux-vdso.so.1 (0x00007ffe96b86000)
        libc.musl-x86_64.so.1 => not found
bin/crio/cni/plugins/portmap:
        not a dynamic executable
bin/crio/crio:
        linux-vdso.so.1 (0x00007fff5e7f1000)
        libgpgme.so.11 => /usr/lib/x86_64-linux-gnu/libgpgme.so.11 (0x00007f5fffab4000)
        libc.musl-x86_64.so.1 => not found
        libassuan.so.0 => /usr/lib/x86_64-linux-gnu/libassuan.so.0 (0x00007f5fff8a1000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5fff68c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5fff29b000)
        /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f5fffcfb000)
bin/slirp4netns:
        statically linked

@giuseppe PTAL?

Not working on OpenVZ

Following the requirements and download steps, I wasn't able to use even dockerd (but actually the fault of rootlesskit) on a OpenVZ container:

[root@s usernetes]# ./run.sh default-docker-nokube                                                                                                                                                                                  
./boot/dockerd.sh
./boot/rootlesskit.sh
[rootlesskit] [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
task: Failed to run task "rootlesskit": exit status 1

uname:

[root@s usernetes]# uname -a
Linux s 2.6.32-042stab128.2 #1 SMP Thu Mar 22 10:58:36 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

Template: CentOS 7 Minimal

However, cat /proc/self/exe does work, and it shows a bunch of garbled characters (well they're assembly anyway)

`termination-log` is not available when mounting `/dev` in containers

I've found that the kubernetes-sigs/sig-storage-local-static-provisioner attempts to mount /dev from the host in its provisioner container, which causes issues when running the daemonset under u7s. Given that there are no reports of this being broken on their issue tracker, I presumed it was an unexpected failure case for u7s rather than them doing the wrong thing (feel free to correct me on this).

It's fairly trivial to work around the issue by setting the terminationMessagePath for the container, although it's not clear to me if this would result in termination messages being lost. See the badness.yaml snippet below and its failure events in describe pod badness:

$ cat badness.yaml
apiVersion: v1
kind: Pod
metadata:
  name: badness
  labels:
    app: badness
spec:
  containers:
  - image: busybox
    name: badness
    command: ["sleep", "infinity"]
    volumeMounts:
      - name: shared-dev
        mountPath: /dev
    #terminationMessagePath: /var/run/termination-log
  volumes:
    - name: shared-dev
      hostPath:
        path: /dev/
$ kubectl apply -f badness.yaml
pod/badness created
$ kubectl describe pod badness
...
Events:
  Type     Reason     Age              From          Message
  ----     ------     ----             ----          -------
  Normal   Scheduled  <unknown>                      Successfully assigned default/user-storage-provisioner-7sn8s to host
  Normal   Pulled     2s (x2 over 2s)  kubelet, host Container image "quay.io/external_storage/local-volume-provisioner:v2.3.4" already present on machine
  Warning  Failed     2s               kubelet, host Error: container create failed: open `/home/user/.local/share/usernetes/containers/storage/overlay/6d3e5e577d2174a25f2947faa0af0b5e68af13bedb662eaaf59068f173640091/merged/termination-log`: No such file or directory
  Warning  Failed     2s               kubelet, host Error: container create failed: open `/home/user/.local/share/usernetes/containers/storage/overlay/273772f44ad6a6e0d55af8b2d3dadb154c734084d4bc82b7f7e4f0cd7e8bda8e/merged/termination-log`: No such file or directory

Uncommenting the terminationMessagePath in badness.yaml makes the pod start happily.

kubelet: split CgroupManager as a separate gRPC service? or move to CRI

If we could separate out CgroupManager to a separate gRPC service that would listen on an UNIX socket, it might be easier to experiment cgroup2 driver as well as "NOP" driver.

@giuseppe WDYT?

cc @ibuildthecloud @cyphar

Fails on Chrome OS Crostini

$ docker run -d --name usernetes-node -p 127.0.0.1:8080:8080  -e U7S_ROOTLESSKIT_PORTS=0.0.0.0:8080:8080/tcp --privileged rootlesscontainers/usernetes default-docker
03c3f4df528ff65f8bef7733613ce1443424356a7fadd60b74012499a48742df

$ docker container logs usernetes-node 
./boot/kube-proxy.sh
./boot/etcd.sh
./boot/rootlesskit.sh
./boot/dockerd.sh
./boot/kube-scheduler.sh
./boot/kubelet-dockershim.sh
./boot/kube-controller-manager.sh
./boot/kube-apiserver.sh
[rootlesskit] open: No such file or directory
[rootlesskit] [rootlesskit:parent] error: failed to setup network &{binary:slirp4netns mtu:65520 ipnet:<nil> disableHostLoopback:true apiSocketPath: enableSandbox:true enableSeccomp:true}: setting up tap tap0: executing [[nsenter -t 107 -n -m -U --preserve-credentials ip tuntap add name tap0 mode tap] [nsenter -t 107 -n -m -U --preserve-credentials ip link set tap0 up]]: exit status 1
[rootlesskit] [rootlesskit:child ] error: parsing message from fd 3: EOF
[kube-proxy] [INFO] Entering RootlessKit namespaces: ...
[kube-apiserver] [INFO] Entering RootlessKit namespaces: ...
[etcd] [INFO] Entering RootlessKit namespaces: ...
[kube-controller-manager] [INFO] Entering RootlessKit namespaces: ...
[kubelet-dockershim] [INFO] Entering RootlessKit namespaces: ...
[kube-scheduler] [INFO] Entering RootlessKit namespaces: ...
task: Failed to run task "rootlesskit": exit status 1
[dockerd] [INFO] Entering RootlessKit namespaces: ...
$

Flaky test: "Smoke test (containerd, with systemd cgroup manager)"

2020-08-13T10:43:48.2176040Z + INFO 'Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local'
2020-08-13T10:43:48.2176720Z �[104m�[97m[INFO]�[49m�[39m Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local
2020-08-13T10:43:48.2177370Z + echo -e '\e[104m\e[97m[INFO]\e[49m\e[39m Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local'
2020-08-13T10:43:48.2178050Z + kubectl run -i --rm --image=alpine --restart=Never dnstest-shell -- sh -exc 'for f in $(seq 0 2); do wget -O- http://dnstest-${f}.dnstest.default.svc.cluster.local; done'
2020-08-13T10:43:50.2488250Z + seq 0 2
2020-08-13T10:43:50.2490070Z + wget -O- http://dnstest-0.dnstest.default.svc.cluster.local
2020-08-13T10:43:50.2491290Z Connecting to dnstest-0.dnstest.default.svc.cluster.local (10.88.0.15:80)
2020-08-13T10:43:50.2491580Z writing to stdout
2020-08-13T10:43:50.2492520Z -                    100% |********************************|   612  0:00:00 ETA
2020-08-13T10:43:50.2492710Z written to stdout
2020-08-13T10:43:50.2494740Z + wget -O- http://dnstest-1.dnstest.default.svc.cluster.local
2020-08-13T10:43:50.2495120Z <!DOCTYPE html>
2020-08-13T10:43:50.2495370Z <html>
2020-08-13T10:43:50.2495580Z <head>
2020-08-13T10:43:50.2495810Z <title>Welcome to nginx!</title>
2020-08-13T10:43:50.2496050Z <style>
2020-08-13T10:43:50.2496200Z     body {
2020-08-13T10:43:50.2496420Z         width: 35em;
2020-08-13T10:43:50.2496740Z         margin: 0 auto;
2020-08-13T10:43:50.2497740Z         font-family: Tahoma, Verdana, Arial, sans-serif;
2020-08-13T10:43:50.2498130Z     }
2020-08-13T10:43:50.2498370Z </style>
2020-08-13T10:43:50.2498600Z </head>
2020-08-13T10:43:50.2498820Z <body>
2020-08-13T10:43:50.2498990Z <h1>Welcome to nginx!</h1>
2020-08-13T10:43:50.2499250Z <p>If you see this page, the nginx web server is successfully installed and
2020-08-13T10:43:50.2499500Z working. Further configuration is required.</p>
2020-08-13T10:43:50.2499670Z 
2020-08-13T10:43:50.2499860Z <p>For online documentation and support please refer to
2020-08-13T10:43:50.2500170Z <a href="http://nginx.org/">nginx.org</a>.<br/>
2020-08-13T10:43:50.2500560Z Commercial support is available at
2020-08-13T10:43:50.2500850Z <a href="http://nginx.com/">nginx.com</a>.</p>
2020-08-13T10:43:50.2501170Z 
2020-08-13T10:43:50.2501360Z <p><em>Thank you for using nginx.</em></p>
2020-08-13T10:43:50.2501580Z </body>
2020-08-13T10:43:50.2501810Z </html>
2020-08-13T10:43:50.2502700Z wget: bad address 'dnstest-1.dnstest.default.svc.cluster.local'
2020-08-13T10:43:50.2878570Z pod default/dnstest-shell terminated (Error)
2020-08-13T10:43:50.2879110Z pod "dnstest-shell" deleted

https://github.com/rootless-containers/usernetes/runs/979933303 (v20200813.0)

---

The failure isn't reproducible on my laptop

hack: allow building binaries using remote Docker hosts

https://github.com/AkihiroSuda/balancker could be used for load-balancing with multiple remote Docker hosts

support opensuse

Memo

• dockerd requires sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter
• kube-proxy requires sudo prlimit --nofile=:65536 --pid $$
• kubectl.sh needs --preserve-credential for nsenter
• still even docker does not work

$ ./dockercli.sh run -it --rm alpine                     
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
8e3ba11ec2a2: Extracting  2.207MB/2.207MB
docker: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/shadow: invalid argument.

---

usernetes: v20180824.0

vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> id
uid=1000(vagrant) gid=100(users) groups=100(users),1000(vagrant)
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> cat /etc/subuid 
vagrant:231072:65536

vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> cat /etc/subgid
vagrant:231072:65536

vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> cat /etc/os-release 
NAME="openSUSE Tumbleweed"
# VERSION="20180827"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20180827"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20180827"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
vagrant@vagrant-openSUSE-Tumbleweed:~/usernetes> uname -a
Linux vagrant-openSUSE-Tumbleweed 4.18.5-1-default #1 SMP PREEMPT Fri Aug 24 12:38:43 UTC 2018 (9e91e29) x86_64 x86_64 x86_64 GNU/Linux

@cyphar

NFS provisioner fails

Hey, great project! I tried to get the helm nfs server provisioner running and ran into some roadblocks on a GCE centOS 8 machine.

First, just starting the helm chart with this config: (run through envsubst)

replicaCount: 1

image:
  repository: quay.io/kubernetes_incubator/nfs-provisioner
  tag: v2.2.1-k8s1.12
  pullPolicy: IfNotPresent

service:
  type: ClusterIP

  nfsPort: 2049
  mountdPort: 20048
  rpcbindPort: 51413
  externalIPs: []

persistence:
  enabled: true
  storageClass: "-"
  accessMode: ReadWriteOnce
  size: "100Gi"

storageClass:
  create: true
  defaultClass: false
  name: nfs
  allowVolumeExpansion: true
  parameters: {}

  mountOptions:
    - vers=4.1
    - noatime

  reclaimPolicy: Retain

rbac:
  create: true
  serviceAccountName: default

resources:
  {}
nodeSelector:
  kubernetes.io/hostname: ${nodename}
tolerations: []
affinity: {}

Afterwards, the necessary PV is created and a PVC is bound to nfs:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "${nfspvcname}"
spec:
  capacity:
    storage: "100Gi"
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "${mainstoragepath}"
  claimRef:
    namespace: default
    name: "${nfspvcname}"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: default-pvc
spec:
  storageClassName: "nfs"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: "90Gi"

$nfspvcname is set to the PVC created by NFS.

Now the pod for nfs crashes continously:

kubectl describe pod nfs-release-nfs-server-provisioner-0
[INFO] Entering RootlessKit namespaces: OK
Name:         nfs-release-nfs-server-provisioner-0
Namespace:    default
Node:         centos/10.0.2.100
Start Time:   Fri, 03 Jan 2020 17:49:16 +0000
Labels:       app=nfs-server-provisioner
              chart=nfs-server-provisioner-0.3.2
              controller-revision-hash=nfs-release-nfs-server-provisioner-79c9977558
              heritage=Helm
              release=nfs-release
              statefulset.kubernetes.io/pod-name=nfs-release-nfs-server-provisioner-0
Annotations:  <none>
Status:       Running
IP:           10.88.0.4
IPs:
  IP:           10.88.0.4
Controlled By:  StatefulSet/nfs-release-nfs-server-provisioner
Containers:
  nfs-server-provisioner:
    Container ID:  docker://8ce423f7c0df95d08a4c49531b0fd59d6a8e8d97afd9e7756a99a38e51b9736f
    Image:         quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12
    Image ID:      docker-pullable://quay.io/kubernetes_incubator/nfs-provisioner@sha256:f0f0d9d39f8aac4a2f39a1b0b602baa993bca0f22c982f208ca9d7a0d2b2399f
    Ports:         2049/TCP, 20048/TCP, 111/TCP, 111/UDP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/UDP
    Args:
      -provisioner=cluster.local/nfs-release-nfs-server-provisioner
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Fri, 03 Jan 2020 17:56:16 +0000
      Finished:     Fri, 03 Jan 2020 17:56:23 +0000
    Ready:          False
    Restart Count:  6
    Environment:
      POD_IP:          (v1:status.podIP)
      SERVICE_NAME:   nfs-release-nfs-server-provisioner
      POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /export from data (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   data-nfs-release-nfs-server-provisioner-0
    ReadOnly:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=centos
Tolerations:     <none>
Events:
  Type     Reason             Age                  From               Message
  ----     ------             ----                 ----               -------
  Warning  FailedScheduling   <unknown>            default-scheduler  error while running "VolumeBinding" filter plugin for pod "nfs-release-nfs-server-provisioner-0": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling   <unknown>            default-scheduler  error while running "VolumeBinding" filter plugin for pod "nfs-release-nfs-server-provisioner-0": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled          <unknown>            default-scheduler  Successfully assigned default/nfs-release-nfs-server-provisioner-0 to centos
  Normal   Pulling            11m                  kubelet, centos    Pulling image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12"
  Normal   Pulled             10m                  kubelet, centos    Successfully pulled image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12"
  Normal   Pulled             10m (x2 over 10m)    kubelet, centos    Container image "quay.io/kubernetes_incubator/nfs-provisioner:v2.2.1-k8s1.12" already present on machine
  Normal   Created            10m (x3 over 10m)    kubelet, centos    Created container nfs-server-provisioner
  Normal   Started            10m (x3 over 10m)    kubelet, centos    Started container nfs-server-provisioner
  Warning  BackOff            9m47s (x3 over 10m)  kubelet, centos    Back-off restarting failed container
  Warning  MissingClusterDNS  50s (x57 over 11m)   kubelet, centos    pod: "nfs-release-nfs-server-provisioner-0_default(ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.

Since the error seems to be related to missing DNS services I tried to setup kube dns via https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/12-dns-addon.md

I had to change the IP from 10.32.0.10 to 10.0.0.10, but the dns pod also fails:

kubectl describe pod coredns-68567cdb47-78x67 --namespace kube-system
[INFO] Entering RootlessKit namespaces: OK
Name:                 coredns-68567cdb47-78x67
Namespace:            kube-system
Priority Class Name:  system-cluster-critical
Node:                 centos/10.0.2.100
Start Time:           Fri, 03 Jan 2020 17:48:54 +0000
Labels:               k8s-app=kube-dns
                      pod-template-hash=68567cdb47
Annotations:          <none>
Status:               Running
IP:                   10.88.0.3
IPs:
  IP:           10.88.0.3
Controlled By:  ReplicaSet/coredns-68567cdb47
Containers:
  coredns:
    Container ID:  docker://387906805acc0fff0f1bbf1e392e886e77c09363bbcce61720db4f316862aaa7
    Image:         coredns/coredns:1.6.2
    Image ID:      docker-pullable://coredns/coredns@sha256:12eb885b8685b1b13a04ecf5c23bc809c2e57917252fd7b0be9e9c00644e8ee5
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 03 Jan 2020 17:59:48 +0000
      Finished:     Fri, 03 Jan 2020 17:59:48 +0000
    Ready:          False
    Restart Count:  7
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        coredns
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  <unknown>             default-scheduler  Successfully assigned kube-system/coredns-68567cdb47-78x67 to centos
  Normal   Pulling    14m                   kubelet, centos    Pulling image "coredns/coredns:1.6.2"
  Normal   Pulled     14m                   kubelet, centos    Successfully pulled image "coredns/coredns:1.6.2"
  Normal   Created    13m (x5 over 14m)     kubelet, centos    Created container coredns
  Normal   Started    13m (x5 over 14m)     kubelet, centos    Started container coredns
  Normal   Pulled     13m (x4 over 14m)     kubelet, centos    Container image "coredns/coredns:1.6.2" already present on machine
  Warning  BackOff    4m51s (x50 over 14m)  kubelet, centos    Back-off restarting failed container

Stderr of run.sh:

[kubelet-dockershim] E0103 18:03:48.735562     114 container_manager_linux.go:477] cpu and memory cgroup hierarchy not unified.  cpu: /, memory: /user.slice/user-1000.slice/session-1.scope
[kubelet-dockershim] E0103 18:03:48.867398     114 container_manager_linux.go:101] Unable to ensure the docker processes run in the desired containers: errors moving "dockerd" pid: failed to apply oom score -999 to PID 85: write /proc/85/oom_score_adj: permission denied
...
[dockerd] time="2020-01-03T18:04:56.269919903Z" level=info msg="shim containerd-shim started" address=/containerd-shim/6ba264bcbcc738a4686c6b6bbc36cd4c96cbd3a5ff04b2a14b4064f48779d088.sock debug=false pid=7150
[dockerd] time="2020-01-03T18:04:56.903241134Z" level=info msg="shim reaped" id=d023513cf1411c90f83bf1a30d2d55a3e3cde300d59dea05f2ecafea011be2e1
[dockerd] time="2020-01-03T18:04:56.913507529Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[dockerd] time="2020-01-03T18:05:05.238881369Z" level=info msg="shim containerd-shim started" address=/containerd-shim/1a5da1ec06ca697778b35df31d55a6f5befe987f6a82b138b3def79ccab895eb.sock debug=false pid=7269
[dockerd] time="2020-01-03T18:05:05.601143797Z" level=info msg="shim reaped" id=e6eebcd70064eed6d293f505e98c9e6a3d2a682213a281cdfdebecf245b2f3fa
[dockerd] time="2020-01-03T18:05:05.611475434Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
...
[kubelet-dockershim] E0103 18:05:06.218907     114 pod_workers.go:191] Error syncing pod 10011dcf-1fcc-4f90-9692-c27225bfb393 ("coredns-68567cdb47-78x67_kube-system(10011dcf-1fcc-4f90-9692-c27225bfb393)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "back-off 5m0s restarting failed container=coredns pod=coredns-68567cdb47-78x67_kube-system(10011dcf-1fcc-4f90-9692-c27225bfb393)"
[kubelet-dockershim] E0103 18:05:06.595575     114 pod_workers.go:191] Error syncing pod 9ff213d8-36e9-448c-9675-8f344be436fc ("coredns-68567cdb47-xvxnv_kube-system(9ff213d8-36e9-448c-9675-8f344be436fc)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "back-off 5m0s restarting failed container=coredns pod=coredns-68567cdb47-xvxnv_kube-system(9ff213d8-36e9-448c-9675-8f344be436fc)"
[kubelet-dockershim] E0103 18:06:14.151994     114 pod_workers.go:191] Error syncing pod ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9 ("nfs-release-nfs-server-provisioner-0_default(ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9)"), skipping: failed to "StartContainer" for "nfs-server-provisioner" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nfs-server-provisioner pod=nfs-release-nfs-server-provisioner-0_default(ba5bd6f1-c4af-4f78-93f2-c9a48d126ba9)"