Can use existing vpc and subnets about eksctl

[archisgore]

I'm working on this issue as we speak (if nobody else already is). This is a scenario I need for myself in prod. :-)

In I only found eksctl because I was about to write a shell script to rinse-repeat launch clusters. So I'll get around to this very soon.

[tigerwings]

@errordeveloper , vpc peering is not a valid solution. The main reason to use DC VPCs is that we need to access our internal service from DC VPCs. On the other hand, we may setup and tear down an EKS cluster often, the peering ops have too much overheads.

[richardcase]

@darrenhaken - if we supported --vpc and --subnets (like kops) on create cluster would that satisfy your requirement?

[errordeveloper]

I think we might want to create a separate issue with a design proposal and discuss details there.

At the moment, I am thinking of the following 3 modes being supported eventually:

1. dedicated VPC (what we currently have already)
2. --subnets specified by the user, VPC ID determined from that (which implies 3 subnets must be given, and routing table as well as any other resources are already configured, subnet size and availably of allocatable IP addresses is up to the user to take care of; should be possible to use default VPC)
3. --vpc specified by the user, all subnets are created (should be possible to use default VPC)

We can start by adding 2 first, as that seems like the least common denominator.

As mentioned earlier, it'd be good also provide a utility command that create pre-requisite VPC stack, so that could be used in the case when VPC management is delegated to a network operations teams who may prefer to use eksctl for convenience, such utility may also write out CloudFormation template that can be customised to enable VPC peering or any other advanced use-cases.

---

I'm happy to use the above as starting point for a proposal, but please let me know if there is anything I've missed, before I go ahead and create one.

[richardcase]

I'll take a stab at this next week if no one picks it up.

[errordeveloper]

I think there will be two major modes here:

1. use a preconfigured VPC, but create new subnets (easiest for the user who is simply short of VPCs)
2. use a preconfigured VPC and subnets (someone who wants certain kind of configures, e.g. as mention above - with direct connect was; or more generally someone who has to control networking separately)

I think it must be possible to go with first option and and instruct the user to configure routes and additional security features separately.

[errordeveloper]

@tigerwings I have a specific question to you personally – can you use VPC peering to connect your EKS VPC with an existing VPC and enable direct connect that way?

[nadenf]

@archisgore .. Any status update on this ? We can look into picking this up also.

If you are wondering of a use case here's one. In many large enterprises e.g. ours we centrally control the core networking e.g. VPCs but then freely allow various teams to pick and choose what services they want to use. We are wanting to use eksctl but are strictly forbidden from creating VPCs.

[errordeveloper]

@baank thanks for describing your use-case, it's very helpful! Could you clarify if you havr to use pre-existing subnets also, or it would be okay for eksctl to create dedicated subnets?

[nadenf]

@errordeveloper .. No we are not allowed to touch the subnets either.

[errordeveloper]

@baank so you would need two subnets. In your case, would you be able to get two subnets that are dedicated for use with EKS or you would have to use subnets that are also used for other things? If there would be other things, are those gonna be just any AWS workloads or another Kubernetes clusters or something?

What size of subnets would you be able to provide? Here is what we currently use with the dedicated VPC (based on standard CloudFormation template provided):

VpcBlock: 192.168.0.0/16
Subnet01Block: 192.168.64.0/18
Subnet02Block: 192.168.128.0/18

I'd have to check if anything smaller would work, I don't know right now.

Also, are you able to create EC2 security groups or it's a similar situation there?

[nadenf]

We have a number of subnets but they are in different VPCs and AZs:

subnet-A -> vpc1-data-a -> ap-southeast-2a
subnet-B -> vpc1-data-b -> ap-southeast-2b
subnet-C -> vpc1-application-a -> ap-southeast-2a
subnet-D -> vpc1-application-b -> ap-southeast-2b

The subnets are:
x.x.x.x/24
x.x.x.x/25
x.x.x.x/26

And the subnets are basically a free for all. Each team can decide what they want and unless we run out of IP addresses (sometimes happens) then no one particular cares. Some have Kube-AWS clusters, EC2 instances etc. Just picture hundreds of developers each treating it as though it was there personal AWS account.

And no one really cares too much about security groups since they only restrict internal traffic within the VPCs and there are probably easier ways for a rogue employee to steal data or hack into systems.

[errordeveloper]

@baank thanks again, this is very helpful for us! Please note that at the moment EKS is not available in ap-southeast-2, I am not sure what is the timeline.

[nukepuppy]

just checking in.. as defining the CIDR range is probably one of the most basic things folks will need to support..

Though this issue is about re-using existing? Is there a way for new cluster creation to use the CIDR of choosing? - if not where is that on roadmap?

this blocks any kind of peering or direct connect setups which allow for restricted resources elsewhere.. and well.. that's pretty essential

[errordeveloper]

@nukepuppy thanks for clarifying the use-case, could you please open another issue, unless using a pre-existing VPC and subnets would be satisfactory (or even somewhat more beneficial) for you? We are planning to start working on this issue soon. I was hoping subnet CIDR params could wait until we have Cluster API support and can expose those and other advanced parameters via a config file and avoid exploding the number of CLI flags, if we can. If this is a burning issue, please consider looking at the code and opening a PR, I am happy to help you along.

[nukepuppy]

@errordeveloper i'll open a new issue.. The use case is net new environments / VPCs which can be used in an enterprise environment where IPAM is enforced etc.. and using the standard 192 or 172 class Bs wont do.. so yes I'll definitely open a new issue.

As for playing with a PR .. if time allows I'm definitely down to dive into it

[darrenhaken]

How's the progress coming with this? I also have the need to use an existing VPC and subnets.

[darrenhaken]

Definitely yeah

…

On Fri, 7 Sep 2018, 12:13 Richard Case, ***@***.***> wrote: @darrenhaken <https://github.com/darrenhaken> - if we supported --vpc and --subnets (like kops) on create cluster would that satisfy your requirement? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6Me1BknYYAavO6Qc7H8aCYoFsIxzcxks5uYlTGgaJpZM4Ubzu6> .

[errordeveloper]

One big confusing question is whether we should support a mode where one only specifies the VPC, and no subnets? Would that imply we have to try finding suitable existing subnet or create new ones? If we create new ones, how should we go about routing tables? Can one have isolated overlapping subnets (with separate routing tables) in one VPC, would that be of any use? Maybe best to avoid diging ourselves into those holes too much. I think it is better to ask user to supply all pre-existing resources, and bail if something is missing (or unsuitable for our use), as opposed to trying to create what is needed. Additionally, it maybe handy to provide a helper that creates a VPC in a separate step (we should handle IAM similarly also). I don't think we have to ask for VPC as well as subnets, as latter can be determined from the former (unless I am missremembering). I think we should draw one line to start with, not many, i.e. give us all pre-existing things and we don't create anything, or nothing at all and we create everything. I can forsee something in the middle emerging eventually, but it's probably best to avoid trying to identify what exactly that is for now.

[darrenhaken]

I'm basing the best option around how Terraform does it:

resource "aws_eks_cluster" "demo" {
  name            = "${var.cluster-name}"
  role_arn        = "${aws_iam_role.demo-cluster.arn}"

  vpc_config {
    security_group_ids = ["${aws_security_group.demo-cluster.id}"]
    subnet_ids         = ["${aws_subnet.demo.*.id}"]
  }

  depends_on = [
    "aws_iam_role_policy_attachment.demo-cluster-AmazonEKSClusterPolicy",
    "aws_iam_role_policy_attachment.demo-cluster-AmazonEKSServicePolicy",
  ]
}

Here Terraform allows passing the VPC and the subnet IDs

[darrenhaken]

I think the security groups being args is also good

[mduarte]

We also have a use case where we would like to have the iam-role as a parameter as well.

[errordeveloper]

Mario, I agree, there is a use-case for that too. We have it documented in another issue.

…

On Fri, 7 Sep 2018, 5:26 pm Mario Duarte, ***@***.***> wrote: We also have a use case where we would like to have the iam-role as a parameter as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWS6gOVObz7md-wdIDr0VQyPsrQVAfks5uYp4sgaJpZM4Ubzu6> .

[darrenhaken]

IAM role would also be great! I often find I want control of the networking and security resources rather than auto generated. On Fri, 7 Sep 2018, 20:49 Ilya Dmitrichenko, <[email protected]> wrote:

…

Mario, I agree, there is a use-case for that too. We have it documented in another issue. On Fri, 7 Sep 2018, 5:26 pm Mario Duarte, ***@***.***> wrote: > We also have a use case where we would like to have the iam-role as a > parameter as well. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#42 (comment)>, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAPWS6gOVObz7md-wdIDr0VQyPsrQVAfks5uYp4sgaJpZM4Ubzu6 > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6Me_BUhqegxCZlkXTcUJ_g3oJzwUcNks5uYs3ggaJpZM4Ubzu6> .

[darrenhaken]

Any update on how this is going?

[richardcase]

@darrenhaken - i didn't get around to starting it. I'm going to try and get scaling finished and then if no one else picks it i will add this as my next task this week.

[errordeveloper]

At this point --subnets flag (as described above) would be very easy to implement, perhaps even --vpc flag. For next week's release I'd like to focus on #287 and #281, but would be keen to review a PR for this. Any takers? It'd mostly a matter of adding a flag and making sure plumbing is right.

[ArseniiPetrovich]

+1 to this issue. At the moment we are in the process of developing a script for internal use that uses eksctl as one of the core features to create EKS cluster. Possibility to specify an existing VPC is a great feature to add.
Is there any chance this one would be finished before the end of november, please?)

[errordeveloper]

Yes, I would hope to add this next week actually, but I would be equally happy to see a PR too (it is rather easy now, we have most hooks in place).

…

On Fri, 2 Nov 2018, 9:26 am Arsenii Petrovich, ***@***.***> wrote: +1 to this issue. At the moment we are in the process of developing a script for internal use that uses eksctl as one of the core features to create EKS cluster. Possibility to specify an existing VPC is a great feature to add. Is there any chance this one would be finished before the end of november, please?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWS8WtbPkmwfow5ePe5L4JrkkzHepXks5urA_hgaJpZM4Ubzu6> .

[errordeveloper]

We have a design proposal now – #303, I'd love to hear feedback from folks here. I am also intending to add --vpc-private-subnets and --vpc-public-subnets in this weeks release, we can discuss and amend the proposal as needed later on also.

[genums]

Hi,

Will the new changes be available somewhere for us to link to or update the EKS quick start? Currently, the setup in the quick start doesn't work because of our need to use our existing VPC and public subnets which routes the path to the internet through our data centers. Your proposal above would cure a lot of headaches.

Thank you

[errordeveloper]

I am intending to land #305 in this week's release, most likely that will happen on Thursday. Please comment on the PR if you have more specific concerns/questions.

…

On Tue, 6 Nov 2018, 6:19 pm genums, ***@***.***> wrote: Hi, Will the new changes be available somewhere for us to link to or update the EKS quick start? Currently, the setup in the quick start doesn't work because of our need to use our existing VPC and public subnets which routes the path to the internet through our data centers. Your proposal above would cure a lot of headaches. Thank you — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWS0DMk5SG76Jdaale8StdDhm-sZPfks5usdLOgaJpZM4Ubzu6> .

[darrenhaken]

Is this done?’

…

On Wed, 7 Nov 2018 at 10:09, Ilya Dmitrichenko ***@***.***> wrote: Closed #42 <#42>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6Me94NBhtY3lXqv4PFP23adO_e6fQXks5usrFqgaJpZM4Ubzu6> .

[errordeveloper]

Broadly, it we have #305 in master now. We still have #308 to address, but after that we are intending to cut a release before end of the week.

…

On Wed, 7 Nov 2018, 2:01 pm Darren Haken, ***@***.***> wrote: Is this done?’ On Wed, 7 Nov 2018 at 10:09, Ilya Dmitrichenko ***@***.***> wrote: > Closed #42 <#42>. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#42 (comment)>, or mute > the thread > < https://github.com/notifications/unsubscribe-auth/AA6Me94NBhtY3lXqv4PFP23adO_e6fQXks5usrFqgaJpZM4Ubzu6 > > . > — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWS-klCCF4lNgX-gCS8_jha3ipeUxpks5usufNgaJpZM4Ubzu6> .

[errordeveloper]

Once #310 is merged, we can get the release out. On Wed, 7 Nov 2018, 9:27 pm Ilya Dmitrichenko, <[email protected]> wrote:

…

Broadly, it we have #305 in master now. We still have #308 to address, but after that we are intending to cut a release before end of the week. On Wed, 7 Nov 2018, 2:01 pm Darren Haken, ***@***.***> wrote: > Is this done?’ > > On Wed, 7 Nov 2018 at 10:09, Ilya Dmitrichenko ***@***.***> > wrote: > > > Closed #42 <#42>. > > > > — > > You are receiving this because you were mentioned. > > Reply to this email directly, view it on GitHub > > <#42 (comment)>, or > mute > > the thread > > < > https://github.com/notifications/unsubscribe-auth/AA6Me94NBhtY3lXqv4PFP23adO_e6fQXks5usrFqgaJpZM4Ubzu6 > > > > . > > > > — > You are receiving this because you modified the open/close state. > > > Reply to this email directly, view it on GitHub > <#42 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAPWS-klCCF4lNgX-gCS8_jha3ipeUxpks5usufNgaJpZM4Ubzu6> > . >

[ORESoftware]

Anybody know how to reuse/reference an existing VPC?

eksctl create cluster --vpc=vpc-0f9e518dea1b2c523 

?

[errordeveloper]

This is well documented, please have a good look at the readme or eksclt.io page. You need to use subnet IDs, VPC ID is auto-detected.

…

On Thu, 13 Jun 2019, 2:32 am Operations Research Engineering Software+, < ***@***.***> wrote: Anybody know how to reuse/reference an existing VPC? eksctl create cluster --vpc=vpc-0f9e518dea1b2c523 ? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#42>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB5MS2P2PAZ56DL6DLJLNLP2GPURANCNFSM4FDPHO5A> .

[ORESoftware]

its really not well documented, and it seems like a popular use case on that same subject, its unclear in the cluster UI which are private and which are public subnets, I assume by default they are all public?

[errordeveloper]

its really not well documented, and it seems like a popular use case

Could you please tell me what is missing here?

on that same subject, its unclear in the cluster UI which are private and which are public subnets, I assume by default they are all public?

Do you mean the EKS console?

[ORESoftware]

yeah in the EKS console, it's unclear whether subnets are public/private (is it more clear somewhere else, idk?)

Anyway, for long readme's, it would be nice to have a table of contents (ToC) at the top and link to the subsections.

[errordeveloper]

yeah in the EKS console, it's unclear whether subnets are public/private (is it more clear somewhere else, idk?)

The subnets that eksctl creates for you will have it in the name, but it's up to you to name the ones that you are managing yourself.

it would be nice to have a table of contents

We are reworking documentation at the moment, new website is coming any day now! :)