Failing to kill namespaces in `Terminating` state about knsk

[yamaszone]

Hi @thyarles , that worked! Thanks a lot for debugging this and getting rid of annoying namespace termination failures. Cheers!

[thyarles]

Hi @yamaszone,

Thank you for your comment. By the way, sorry for the delay in analyzing your question, I don't know why, but GitHub did not notify me and just now I saw it. Could you please take a look in /tmp folder from a machine that your run the script? There a file named "yournamespacename.json", could you put here the content of it?

Thanks.

[yamaszone]

Hi @thyarles , thanks a lot for following up on this! Here's an example JSON file for you to review:

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-12-19T01:57:44Z",
        "deletionTimestamp": "2020-04-09T15:25:26Z",
        "name": "exp",
        "resourceVersion": "7492953",
        "selfLink": "/api/v1/namespaces/exp",
        "uid": "f2e8f518-2202-11ea-a046-025000000001"
    },
    "spec": {
        "finalizers": [

        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

Let me know if you need more information. Thanks in advance for your help!

[thyarles]

Please, could you try to run as

./knsk.sh --delete-all --force --dry-run --no-color

And post the output? You can hide sensitive information, if there any.

[yamaszone]

Hi @thyarles , here's the output:

Kubernetes NameSpace Killer

Checking if kubectl is configured... ok

Checking for unavailable apiservices... found
.: Broken -> v1beta1.servicecatalog.k8s.io... dry-run
   timeout 15 kubectl delete apiservice v1beta1.servicecatalog.k8s.io

Checking for stucked namespaces... found
.: Checking resources in namespace a... not found
.: Checking resources in namespace b... not found
.: Checking resources in namespace catalog... not found
.: Checking resources in namespace exp... not found

Checking for stucked resources in the cluster... not found

Checking for orphan resources in the cluster... not found

Forcing deletion of stucked namespaces
.: Checking compliance of --force option... ok
.: Getting the access token to force deletion... ok
.: Starting kubectl proxy... ok
.: Checking for resisted stucked namespaces to force deletion... found
   > Forcing deletion of a... dry-run
     curl -s -o /tmp/a.json.log -X PUT --data-binary @/tmp/a.json http://localhost:8765/api/v1/namespaces/a/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure
   > Forcing deletion of b... dry-run
     curl -s -o /tmp/b.json.log -X PUT --data-binary @/tmp/b.json http://localhost:8765/api/v1/namespaces/b/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure
   > Forcing deletion of catalog... dry-run
     curl -s -o /tmp/catalog.json.log -X PUT --data-binary @/tmp/catalog.json http://localhost:8765/api/v1/namespaces/catalog/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure
   > Forcing deletion of exp... dry-run
     curl -s -o /tmp/exp.json.log -X PUT --data-binary @/tmp/exp.json http://localhost:8765/api/v1/namespaces/exp/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure

.: Stopping kubectl proxy... ok

:: Done in 93 seconds.

Let me know if you need any other info.

[thyarles]

Awesome, @yamaszone!

The first thin is figure out why you v1beta1.servicecatalog.k8s.io is broken. You know from it came? Looking in my full working cluster, I don't have it. If you think it is is importante, don't run the command to delete it bellow.

Let's try the commands manually... if all of it works, the error is something about BASH on MacOS.

1. In one shell, let's start the proxy on port 8765 (or other port you want)

KPORT=8765
kubectl proxy -p $KPORT

2. In another terminal, do the commands

kubectl delete apiservice v1beta1.servicecatalog.k8s.io (do this command only if you sure about it)

KPORT=8765 (it must be the same setted before)
BEARER=xxx (paste here your bearer code)

curl -X PUT --data-binary @/tmp/a.json http://localhost:$KPORT/api/v1/namespaces/a/finalize -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER" --insecure

curl -X PUT --data-binary @/tmp/b.json http://localhost:$KPORT/api/v1/namespaces/b/finalize  -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER" --insecure

curl -X PUT --data-binary @/tmp/catalog.json http://localhost:$KPORT/api/v1/namespaces/catalog/finalize -H "Content-Type: application/json" -H  "Authorization: Bearer $BEARER" --insecure

curl -X PUT --data-binary @/tmp/exp.json http://localhost:$KPORT/api/v1/namespaces/exp/finalize -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER" --insecure

Please, let me know if it works. If not, please, take a close look at all output.