kvaps / kubectl-node-shell

Exec into node via kubectl

kubectl node-shell

(formerly known as kubectl-enter)

Start a root shell in the node's host OS running. Uses an alpine pod with nsenter for Linux nodes and a HostProcess pod with PowerShell for Windows nodes.

Installation

using krew:

Plugin can be installed from the official krew repository:

kubectl krew install node-shell

Or from our own krew repository:

kubectl krew index add kvaps https://github.com/kvaps/krew-index
kubectl krew install kvaps/node-shell

or using curl:

curl -LO https://github.com/kvaps/kubectl-node-shell/raw/master/kubectl-node_shell
chmod +x ./kubectl-node_shell
sudo mv ./kubectl-node_shell /usr/local/bin/kubectl-node_shell

Usage

# Get standard bash shell
kubectl node-shell <node>

# Use X-mode (mount /host, and do not enter host namespace)
kubectl node-shell -x <node>

# Execute custom command
kubectl node-shell <node> -- echo 123

# Use stdin
cat /etc/passwd | kubectl node-shell <node> -- sh -c 'cat > /tmp/passwd'

# Run oneliner script
kubectl node-shell <node> -- sh -c 'cat /tmp/passwd; rm -f /tmp/passwd'

X-mode

X-mode can be useful for debugging minimal systems that do not have a built-in shell (eg. Talos).
Here's an example of how you can debug the network for a rootless kube-apiserver container without a filesystem:

kubectl node-shell -x <node>

# Download crictl
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz -O- | \
  tar -xzf- -C /usr/local/bin/

# Setup CRI endpoint
export CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock

# Find your container
crictl ps | grep kube-apiserver
#3ff4626a9f10e       e7972205b6614       6 hours ago         Running             kube-apiserver         0                   215107b47bd7e       kube-apiserver-talos-rzq-nkg

# Find pid of the container
crictl inspect 3ff4626a9f10e | grep pid
#    "pid": 2152,
#            "pid": 1
#            "type": "pid"
#                "getpid",
#                "getppid",
#                "pidfd_open",
#                "pidfd_send_signal",
#                "waitpid",

# Go to network namespace of the pid, but keep mount namespace of the debug container
nsenter -t 2152 -n

You need to be able to start privileged containers for that.

kubectl-node-shell's Issues

KUBECONFIG might overwrite kubectl $KUBECONFIG

I use KUBECONFIG env var in context of kubectl to manage arbitrary k8s config files.
(See https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#config)
This can cause kubectl node-shell to fail under certain conditions.

For example:

frank@wdfl30113561a:~/devel/k8s/cf-for-k8s$ echo $KUBECONFIG /home/frank/.kube/config:/home/frank/devel/k8s/configs/aws-1.yml:/home/frank/devel/k8s/configs/aws-2.yml:/home/frank/devel/k8s/configs/aws-3.yml:/home/frank/devel/k8s/configs/botobot.yml:/home/frank/devel/k8s/configs/gcp-1.yml

frank@wdfl30113561a:~/devel/k8s/cf-for-k8s$ kubectl get no NAME STATUS ROLES AGE VERSION ip-10-250-19-229.eu-central-1.compute.internal Ready 3h38m v1.17.3 ip-10-250-23-109.eu-central-1.compute.internal Ready 3h38m v1.17.3 ip-10-250-3-136.eu-central-1.compute.internal Ready 3h38m v1.17.3 ip-10-250-5-138.eu-central-1.compute.internal Ready 3h38m v1.17.3

frank@wdfl30113561a:~/devel/k8s/cf-for-k8s$ kubectl node-shell ip-10-250-19-229.eu-central-1.compute.internal Error in configuration: context was not found for specified context: shoot--frnksgr--aws-1

The error is caused by KUBECONFIG within kubectl-node_shell overwriting my environment.

Error attaching, falling back to logs: Upgrade request required

I'm getting the following error when attempting to node-shell to a node.

$ kubectl node-shell aks-test11711-31489578-vmss000000
spawning "nsenter-3qn7ml" on "aks-test11711-31489578-vmss000000"
If you don't see a command prompt, try pressing enter.
Error attaching, falling back to logs: Upgrade request required
groups: cannot find name for group ID 11
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

mesg: ttyname failed: Success



error: timed out waiting for the condition
pod "nsenter-3qn7ml" deleted

Distribute via krew?

👋 Hello, maintainer of the kubectl plugin manager krew here.

What do you think about making your plugin available via krew? You would benefit from

• Greater visibility due krew's large user basis.
• Easier installs.
• Seamless upgrades for new plugin versions.

Krew already has over 60 plugins and more are coming in. See our development guide if you're interested. Please also consult our naming guide to see if your plugin already fulfills our naming requirements.

Kubeconfig argument has wrong name "kubconfig"

It's not possible to use --kubeconfig argument due to the following misprint in the script:

--kubeconfig) KUBECONFIG="--**kubconfig** $2"

kubconfig -it's wrong name

A solution to run none-container command on Kubernetes

Hi,

Here is a simple solution to forward container command to outside and run on host OS, so with this you can run real native command by Kubernetes:

https://app-mesh.readthedocs.io/en/latest/success/kubernetes_run_native_application.html

Incorrect Installation Command in README

Title: Incorrect Installation Command in README

Description:

Issue:
The installation command provided in the README file is incorrect, causing confusion for users trying to install the plugin using kubectl krew.

Incorrect Command:

kubectl krew install kvaps/node-shell

Correct Command:

kubectl krew install node-shell

Explanation:
The README suggests installing the node-shell plugin using the command kubectl krew install kvaps/node-shell. However, this command is incorrect as the plugin is available directly as node-shell, not under kvaps/node-shell.

Expected Behavior:
The README should provide the correct installation command to ensure users can easily install the plugin without encountering errors.

Steps to Reproduce:

1. Follow the installation instructions provided in the README.
2. Attempt to install the node-shell plugin using the provided command.
3. Observe that the command fails due to the incorrect plugin name.

Proposed Correction:
Update the README file with the correct installation command:

kubectl krew install node-shell

This issue was raised to improve the usability and accuracy of the installation instructions for the project.
Thank you for your attention to this matter.

Unable to spin shell on a node with taints

First of all - thank you very much for the project. It is elegant and useful!

It saved my day today, but only partially - because it does not support nodes with taints. Would be great to see it being able to spin shells on any type of node.

How to show gif in README.md?

Dear friend:

I saw you show a very good gif in README, where do you store it and how do you record the gif?

Error - nsenter: can't execute 'bash': No such file or directory

spawning "nsenter-sbpw85" on "docker-desktop"




nsenter: can't execute 'bash': No such file or directory
pod default/nsenter-sbpw85 terminated (Error)
pod "nsenter-sbpw85" deleted

not working on ubuntu 20.10

Looks like it doesn't work on Ubuntu 20.10. If I change the script to use #!/bin/bash instead of #!/bin/sh it works. Otherwise it throws the error:
/usr/local/cats/bin/kubectl-node_shell: 66: cmd+="bash", "-l" ]: not found

Kubectl autocompletion

Is there a chance for node name autocompletion? It would cut steps required to jump into a node in half - making the whole experience even nicer in my opinion :)

k node-shell <TAB> and ready to go ;-)

What an elegant project!

Thanks A LOT for this.

Burnt several hours falling into rabbit holes trying to do something similar until I found this. Saved my sanity.

Please feel free to close this issue.

Make used image override available

Hi all

Since we're in a private cluster, we're unable to directly use the docker.io/alpine image. Currently, that image cannot be overridden in any form.

Suggestion: add a command line option to override this image.

Minor version number may contain non numbers

When running kubectl version --client -o yaml I get the following output:

clientVersion:
  buildDate: "2020-01-15T08:26:26Z"
  compiler: gc
  gitCommit: e7f962ba86f4ce7033828210ca3556393c377bcc
  gitTreeState: clean
  gitVersion: v1.16.6-beta.0
  goVersion: go1.13.5
  major: "1"
  minor: 16+
  platform: linux/amd64

The minor: 16+ breaks the check for minor version

I will add a pull request that filters non numerical from the minor version string shortly.

giving error in one one node set but not in other

}
}' --labels= -t -i nsenter-lwoolh
Error from server (BadRequest): container "nsenter" in pod "nsenter-lwoolh" is not available

• EC=1
• kubectl --context=aks-preprod-123-aks-admin --namespace=kube-system delete pod --wait=false nsenter-lwoolh
  pod "nsenter-lwoolh" deleted
• exit 1

Error from server (NotFound): pods "nsenter-qylbtm" not found

new install on a clean AKS cluster

/usr/local/bin/kubectl-node_shell aks-commpool-35862059-vmss000000

spawning "nsenter-qylbtm" on "aks-commpool-35862059-vmss000000"
error: Invalid JSON Patch
Error from server (NotFound): pods "nsenter-qylbtm" not found

Support docker image pull secret for authenticated users

We are getting ImagePullBackOff for rate limiting since the apline image pull is performed from an unauthenticated user. It would be great to support a docker user so we can use an authenticated user to pull the images. Currently it's very hard to use node-shell in high scale envs

resource limits

my cluster is configured with an admission controller that bounces requests without resource limits. The pod spec doesn't have those.

Cannot connect to nodes on VPN

Installed node-shell plugin using krew Trying to connect to a remote node through VPN. The command is
kubectl node-shell test
The command gives no output.
I am using kubernetes version 1.23

Tag releases for new releases?

1.3.0 and 1.4.0 were released as tags here but not actually created as releases. This means if you use kubectl krew upgrade you'll get the new versions, but if you track releases via the upstream repo on github (this repo) you won't. Can we get proper github releases created and update the release process to make sure releases are created for future updates?

violates PodSecurity "baseline:latest" on Talos

Hi,

When I try to use this, I get denied because of PodSecurity violation. I do see the last line in the readme which I assume is about this, but I'm not really clear on what to do about it.

Since Talos is explicitly mentioned I imagine it should work, is there any step I'm missing to get this plugin to work, any namespace I need to switch to warn mode for that policy ?
Thanks

Prompt and terminal text not displayed

kubectl-node_shell -v
kubectl-node-shell 1.10.0

The text of the terminal and the prompt is not displayed, also the whole terminal layout is messed up.

Note: I was typing the uptime wrong cause I could not see anything.

spawning "nsenter-fqgbmy" on "apps-code-monitoring-g22pobfvmj"
If you don't see a command prompt, try pressing enter.

root@apps-code-monitoring-g22pobfvmj:/# 
root@apps-code-monitoring-g22pobfvmj:/# 
root@apps-code-monitoring-g22pobfvmj:/# 
root@apps-code-monitoring-g22pobfvmj:/# 
root@apps-code-monitoring-g22pobfvmj:/# 
root@apps-code-monitoring-g22pobfvmj:/# uptie
Command 'uptie' not found, did you mean:
                                          command 'uptime' from deb procps (2:3.3.17-6ubuntu2)
                                                                                              Try: sudo apt install <deb name>
                                                                                                                              root@apps-code-monitoring-g22pobfvmj:/# uptime
 11:52:30 up 4 days, 15:38,  0 users,  load average: 0.11, 0.24, 0.20
                                                                     root@apps-code-monitoring-g22pobfvmj:/# exit
logout
      pod "nsenter-fqgbmy" deleted

current-context is not set

hi
i get an issue "current-context is not set",

how can i fix it

Don't use deprecated --generator flag

Using plugin on Kubernetes 1.18.0 produces following warning:

Flag --generator has been deprecated, has no effect and will be removed in the future.

Error: Container "nsenter" is not available

When I try to run the command, I get

spawning "nsenter-1iiih5" on "node"
Error from server (BadRequest): container "nsenter" in pod "nsenter-1iiih5" is not available
pod "nsenter-1iiih5" deleted

Here's a shell log:

❯ sh -x /usr/bin/kubectl-node_shell node
+ set -e
+ kubectl=kubectl
+ version=1.7.0
+ generator=
+ node=
+ nodefaultctx=0
+ nodefaultns=0
+ container_cpu=100m
+ container_memory=256Mi
+ labels=
+ '[' -t 0 ']'
+ tty=true
+ '[' 1 -gt 0 ']'
+ key=node
+ case $key in
+ '[' -z '' ']'
+ node=node
+ shift
+ '[' 0 -gt 0 ']'
+ '[' -z node ']'
+ '[' 0 = 1 ']'
++ kubectl config current-context
+ kubectl='kubectl --context=context'
+ '[' 0 = 1 ']'
++ kubectl --context=context config view --minify --output 'jsonpath={.contexts..namespace}'
+ kubectl='kubectl --context=context --namespace=namespace'
++ kubectl --context=context --namespace=namespace get node node -o 'jsonpath={.metadata.labels.kubernetes\.io/os}'
+ os=linux
+ '[' linux = windows ']'
+ image=docker.io/library/alpine
+ name=nsenter
++ env LC_ALL=C tr -dc a-z0-9
++ head -c 6
+ pod=nsenter-iv2at7
+ cmd_start='"nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid"'
+ cmd_arg_prefix=', "--"'
+ cmd_default=', "bash", "-l"'
+ security_context='{"privileged":true}'
+ '[' 0 -gt 0 ']'
+ cmd='[ "nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid" , "bash", "-l" ]'
++ cat
+ overrides='{
  "spec": {
    "nodeName": "node",
    "hostPID": true,
    "hostNetwork": true,
    "containers": [
      {
        "securityContext": {"privileged":true},
        "image": "docker.io/library/alpine",
        "name": "nsenter",
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "command": [ "nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid" , "bash", "-l" ],
        "resources": {
          "limits":   { "cpu": "100m", "memory": "256Mi" },
          "requests": { "cpu": "100m", "memory": "256Mi" }
        }
      }
    ],
    "tolerations": [
      { "key": "CriticalAddonsOnly", "operator": "Exists" },
      { "effect": "NoExecute",       "operator": "Exists" }
    ]
  }
}'
++ kubectl version --client -o yaml
++ awk '-F[ :"]+' '$2 == "minor" {print $3+0}'
+ m=26
+ '[' 26 -lt 18 ']'
+ trap 'EC=$?; kubectl --context=context --namespace=namespace delete pod --wait=false nsenter-iv2at7 >&2 || true; exit $EC' EXIT INT TERM
+ echo 'spawning "nsenter-iv2at7" on "node"'
spawning "nsenter-iv2at7" on "node"
++ '[' true = true ']'
++ echo -t
+ kubectl --context=context --namespace=namespace run --image docker.io/library/alpine --restart=Never '--overrides={
  "spec": {
    "nodeName": "node",
    "hostPID": true,
    "hostNetwork": true,
    "containers": [
      {
        "securityContext": {"privileged":true},
        "image": "docker.io/library/alpine",
        "name": "nsenter",
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "command": [ "nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid" , "bash", "-l" ],
        "resources": {
          "limits":   { "cpu": "100m", "memory": "256Mi" },
          "requests": { "cpu": "100m", "memory": "256Mi" }
        }
      }
    ],
    "tolerations": [
      { "key": "CriticalAddonsOnly", "operator": "Exists" },
      { "effect": "NoExecute",       "operator": "Exists" }
    ]
  }
}' --labels= -t -i nsenter-iv2at7
Error from server (BadRequest): container "nsenter" in pod "nsenter-iv2at7" is not available
+ EC=1
+ kubectl --context=context --namespace=namespace delete pod --wait=false nsenter-iv2at7
pod "nsenter-iv2at7" deleted
+ exit 1

tty flag busted again

Please use [ -t 0 ] instead of whatever you are doing with tty's again. My install is busted.

invalid json patch when running custom command

kubectl node-shell ip-10-240-27-191.eu-west-1.compute.internal -- echo 123
spawning "nsenter-dzp06n" on "ip-10-240-27-191.eu-west-1.compute.internal"
error: Invalid JSON Patch

kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

thoughts?

node-shell pod timing out

I'm seeing the node-shell pod timing out and getting deleted. I do not see a reason why it should timeout, other than perhaps the node is busy. Nothing in kubectl events other than it getting killed after some time. Is it possible to increase the timeout before it gets deleted?

$ kubectl node-shell aks-servpool1-14120502-vmss000017
spawning "nsenter-j7g30x" on "aks-servpool1-14120502-vmss000017"
error: timed out waiting for the condition
pod "nsenter-j7g30x" deleted

$ ke | grep nsenter-j7g30x
19s Normal Pulling pod/nsenter-j7g30x Pulling image "docker.io/library/alpine"
19s Normal Pulled pod/nsenter-j7g30x Successfully pulled image "docker.io/library/alpine" in 264.806927ms (264.815427ms including waiting)
19s Normal Created pod/nsenter-j7g30x Created container nsenter
19s Normal Started pod/nsenter-j7g30x Started container nsenter
18s Normal Killing pod/nsenter-j7g30x Stopping container nsenter

Support space char in context name

I added some spaces character in my Kubernetes contexte name. For example my first cluster.

In such a case, running the plugin give the following result, and returned code is 1.

kubectl node-shell my-node
Error: flags cannot be placed before plugin name: --context=my

A workaround is to generated a (temporary) kube config file without space, like this: kubectl config view --minify --flatten | sed -r 's/: ([[:alnum:]-]+) /: \1-/;s/: ([[:alnum:]-]+) /: \1-/', and use it with this plugin.

node-shell fails to work with kubernetes version 1.18.4

I just get the below message
spawning "nsenter-0wuphu" on "NODENAME"
error: timed out waiting for the condition

kubectl-node-shell 1.7.0 does not support Windows2019

mcr.microsoft.com/powershell only supports Windows2022 and kubectl node-shell will fail with below error with Windows2019.

kubectl-node-shell version: 1.7.0

spawning "pwsh-g9mo7u" on "aksnpwin000000"
error: timed out waiting for the condition
pod "pwsh-g9mo7u" deleted

kubectl describe po pwsh-g9mo7u:

  Warning  Failed   8s    kubelet  Failed to pull image "mcr.microsoft.com/powershell": rpc error: code = NotFound desc = failed to pull and unpack image "mcr.microsoft.com/powershell:latest": no match for platform in manifest: not found
  Normal   BackOff  8s    kubelet  Back-off pulling image "mcr.microsoft.com/powershell"

Support in-cluster auth

Hi,

it would be nice if node-shell would work with in-cluster auth (without having a kubeconfig file)

Allow images from private registries

Hi there,

Before we get started, amazing tool. Love it ❤️

Now for my question, would you be open to adding support for private registries?
Specifically, I would like to conditionally add an image pull secret to the spec based on if an env-var is set or not.

I am basically imagining something similar to the way custom images are handled now.

If you are interested in that, I am willing to build this feature and contribute it to the project. Just wanted to ask before I get into coding :)

Works for ContainerD as container runtime?

Hi, does this tool work for Nodes with ContainerD as the container runtime? Specifically, AKS v1.19+ clusters

I tried to use kubectl-node-shell v1.2.3 and gets the below

tr: write error: Broken pipe
tr: write error
spawning "nsenter-q6n1dt" on "aks-agentpool-000000-0"
error: timed out waiting for the condition
pod "nsenter-q6n1dt" deleted