Support launching clusters into private VPCs about flintrock

[eshioji]

For people who want to have a quick hacky solution to this. One can create a machine in the VPC and launch a cluster from that machine using private DNS/IP.

However, one needs to do the following:

• Hack flintrock (https://github.com/eshioji/flintrock/pull/1/files)
• Assign additional security group that allows access to necessary ports (the one flintrock assigns didn't work for my VPC setting)
• Make it so that the machines can resolve their own hostname (the default AMI gets hostnames like ip-xx-xx-xx-xx and Spark fails to launch because it can't resolve it).

For the last problem, I did the following:
Create a script with the following content and let it run at startup using rc.local. Not the nicest way but it works.

OWN_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
HOSTNME=`hostname`
echo "$OWN_IP $HOSTNME" | sudo tee --append /etc/hosts

Finally thank you @nchammas , I wanted to use a custom AMI and this project was a life-saver!

[pferrel]

is something like the chaordic fork implemented here? If so are their instructions? I'm trying to run Spark behind the VPC with not outside traffic.

[MuksSharma]

Thanks @nchammas. This use case is becoming more common in enterprise setting. Any thoughts on when this feature might become available?

[nchammas]

As with the typical open source project, there are no promises on when a feature might be delivered. The things that usually get worked on are critical bug fixes, fun stuff, and everything else, in that order. 😄

My current intent is to work next on adding support for resizing existing clusters and improving launch progress reporting. If I get some time at work to devote to Flintrock (e.g. for this issue), I may work on it after I get those 2 features out.

[rmessner]

It is exactly our case ( and we don't even have hostname resolution, and that would need some hack for spark and hadoop ), I will work out something and submit it to you @nchammas.

My use case will be :

• Downloading spark from a custom location ( thanks to #104 )
• Use private DNS names instead of public ones
• Define several origins ( multiple CIDR addresses to access the cluster for the security groups )

[rmessner]

In case of the private VPC, I'm struggling with the ssh access when installing and configuring, because i'm outside the security group and it take my public IP to access an private IP so doesn't work.

I'm thinking about 2 solutions :

• Opening a security breach for the install and configuration step
• Bind the access_origins to the VPC ( means my private IP is within the range of the origins )

What do you think about it @nchammas ?

[nchammas]

Hey @rmessner, I updated the description of the issue and commented on your PRs as well.

[nchammas]

Related PRs from others who have implemented their own fixes for this:

• chaordic#4