Problem with deploy KIP on AWS EKS cluster about kip

[ldx]

@241423 extraSecurityGroups should go inside cells in provider.yaml, example:

cells:
  defaultInstanceType: t3.nano
  bootImageSpec:
    owners: 689494258501
    filters: name=elotl-kip-*
  itzo:
    url: https://itzo-kip-download.s3.amazonaws.com
    version: latest
  extraSecurityGroups:
  - sg-12345678

As for the nil pointer dereference, I opened a PR that should fix the issue. If you would like to check if the new build fixes the problem in your setup, please use image: elotl/kip:v1.0.0-19-g000f5b4 for the kip container in the statefulset.

[myechuri]

@241423 : thank you for trying kip!

F0825 08:20:15.743785       1 main.go:133] error initializing provider kip: error configuring cloud client: Error setting up cloud client: Could not configure AWS cloud client authorization: Error validationg connection to AWS: AuthFailure: AWS was not able to validate the provided access credentials
        status code: 401, request id: 7205bc35-2b6a-48ff-baeb-0eddfd2d4824

From above log, it looks like the AWS credentials kip received are invalid. Can you please confirm that accessKeyID and secretAccessKey filled in kip/base/provider.yaml file are valid?

Meanwhile, i will reproduce your steps and see i get the same error. Will update the issue with findings. Thanks.

[ldx]

@241423 if you rely on IAM instance profiles for AWS authentication (if you used kustomize build base | kubectl apply -f -, then by default that assumes this), please make sure the permissions documented at https://github.com/elotl/kip/blob/master/docs/kip-iam-permissions.md are present in the IAM instance profile that is attached to the worker nodes.

Another common source of issues is a typo in the access keys as @myechuri pointed out (if you configured access keys via provider.yaml or environment variables), or an inaccurate system clock on the instance where the kip pod lands.

[myechuri]

Thanks for the pointers, @ldx !

Hi guys! Thank You for fast response. The problem has been solved. I prepared the environment from scratch and KIP started. It could be my fault attributing policies to the wrong node role. Now, I have another one problem and it could be related to source code. So, After started kip, I wanted to stop it with this command.

kubectl delete statefulset kip-provider -n kube-system

I could not stop kip with below command, described on main git page

kubectl delete -n kube-system statefulset kip

Next time I wanted run KIP with the same settings I have seen another problem and I can’t resolve it. I tried clone repo again, used kip docker image on the day it worked, tried elder release. Summary, when I use kip container logs command I see below output. Have You ever seen this error? I blacked out some info. Maybe it could be related to code, I found this article with similar error. https://www.joeshaw.org/understanding-go-panic-output/

2020-08-31 11:54:55.222787 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster xxxxxxxxxxxxxxxx
I0831 11:54:55.222809       1 etcd.go:129] Etcd server is ready to serve requests
I0831 11:54:55.222834       1 server.go:100] validating write access to etcd (will block until we can connect)
I0831 11:54:55.223822       1 server.go:110] write to etcd successful
I0831 11:54:55.223859       1 server.go:221] ControllerID: xxxxxxxxxxxxxxxxxxxxxxxxxxx
I0831 11:54:58.384109       1 aws.go:113] detected AWS region: "xxxxxx"
I0831 11:54:58.384129       1 config.go:228] using AWS region "xxxxx"
I0831 11:54:58.384134       1 config.go:248] Validating connection to AWS
I0831 11:54:58.384203       1 aws.go:122] Checking for credential errors
I0831 11:55:01.488246       1 aws.go:127] Using credentials from xxxxxxxxxxx
I0831 11:55:01.488264       1 aws.go:133] Validating read access
I0831 11:55:01.832908       1 config.go:252] Validated access to AWS
I0831 11:55:05.100324       1 network.go:147] Current vpc:  vpc-xxxxxxxxxxxxxxxxx
I0831 11:55:05.101430       1 network.go:154] Getting subnets and availability zones for VPC vpc-xxxxxxxxxxxxxxxxxxxxxx
I0831 11:55:05.281960       1 aws.go:196] cells will run in a private subnet (no route to internet gateway)
I0831 11:55:05.281980       1 config.go:453] controller will connect to nodes via private IPs
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d76cdd]
 
goroutine 1 [running]:
github.com/elotl/kip/pkg/server/cloud/aws.awsSGToMilpa(0xc000*6****, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/travis/gopath/src/github.com/elotl/kip/pkg/server/cloud/aws/security_groups.go:2*6 +0x21d
github.com/elotl/kip/pkg/server/cloud/aws.(*AwsEC2).FindSecurityGroup(0xc000*****0, 0xc001454***, 0x30, 0xc00138****, 0x569***, 0xc00145****)
        /home/travis/gopath/src/github.com/elotl/kip/pkg/server/cloud/aws/security_groups.go:107 +0x572
github.com/elotl/kip/pkg/server/cloud/aws.(*AwsEC2).EnsureSecurityGroup(0xc000c3****, 0xc001454**0, 0x30, 0xc0002*****, 0x4, 0x4, 0xc0000c****, 0x1, 0x1, 0x0, ...)
        /home/travis/gopath/src/github.com/elotl/kip/pkg/server/cloud/aws/security_groups.go:117 +0x5d
github.com/elotl/kip/pkg/server/cloud/aws.(*AwsEC2).EnsureMilpaSecurityGroups(0xc000c3****, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/elotl/kip/pkg/server/cloud/aws/security_groups.go:68 +0x1d3
github.com/elotl/kip/pkg/server.ConfigureCloud(0xc00051****, 0xc0000****, 0x1a, 0xc0009b****, 0x3, 0x1, 0x1, 0x0, 0x0)
        /home/travis/gopath/src/github.com/elotl/kip/pkg/server/config.go:455 +0x18f
github.com/elotl/kip/pkg/server.NewInstanceProvider(0x7ffdb6ef6077, 0x16, 0x7ffdb6****, 0xe, 0xc0002a2***, 0xd, 0x0, 0x0, 0x2c9a3cd, 0xd, ...)
        /home/travis/gopath/src/github.com/elotl/kip/pkg/server/server.go:230 +0x65e
main.main.func1(0x7ffdb6ef****, 0x16, 0x7ffdb6ef****, 0xe, 0x2c8d***, 0x5, 0x0, 0x0, 0x280a, 0x2c***cd, ...)
        /home/travis/gopath/src/github.com/elotl/kip/cmd/kip/main.go:107 +0x205
github.com/elotl/kip/vendor/github.com/elotl/node-cli/internal/commands/root.runRootCommandWithProviderAndClient(0x350e840, 0xc00050****, 0xc0000c****, 0x356****, 0xc0001e****, 0xc0001*****, 0x0, 0x0)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/elotl/node-cli/internal/commands/root/root.go:142 +0x7**
github.com/elotl/kip/vendor/github.com/elotl/node-cli/internal/commands/root.runRootCommand(0x350e840, 0xc00050*****, 0xc0001a****, 0xc00016****, 0x0, 0x0)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/elotl/node-cli/internal/commands/root/root.go:74 +0xfb
github.com/elotl/kip/vendor/github.com/elotl/node-cli/internal/commands/root.NewCommand.func1(0xc00056****, 0xc00054****, 0x0, 0xa, 0x0, 0x0)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/elotl/node-cli/internal/commands/root/root.go:55 +0x50
github.com/elotl/kip/vendor/github.com/spf13/cobra.(*Command).execute(0xc0005****, 0xc0000e****, 0xa, 0xa, 0xc0005****, 0xc0000e****)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/spf13/cobra/command.go:838 +0x460
github.com/elotl/kip/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00056****, 0xc000***0**, 0xc000******, 0xc0005*****)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/spf13/cobra/command.go:943 +0x317
github.com/elotl/kip/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/spf13/cobra/command.go:883
github.com/elotl/kip/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/spf13/cobra/command.go:876
github.com/elotl/kip/vendor/github.com/elotl/node-cli.(*Command).Run(0xc00056****, 0x350***, 0xc00050****, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/elotl/kip/vendor/github.com/elotl/node-cli/cli.go:170 +0x84
main.main()
        /home/travis/gopath/src/github.com/elotl/kip/cmd/kip/main.go:132 +0x6ea

One more question, don’t see where can I attach extra security group id to KIP. Minikube has extraSecurityGroup filed in provider.yaml file. I tried add this piece of code to provider.yaml in base directory and it isn’t working. Could you advice where I can add it?