cloudposse / geodesic Goto Github PK

what

• reating should be reading

why

• Misspelled

references

• https://github.com/cloudposse/geodesic/blob/master/Dockerfile#L45

what

detailedInstanceMonitoring: {{ getenv "KOPS_CLOUDWATCH_DETAILED_MONITORING" "true" }}

why

• More detailed monitoring

what

• Use shellcheck to lint all bash scripts
• Consider adding target to build-harness

why

• Reduce bugs, improve consistency

what

• add pwgen

why

• Useful for generating dynamic passwords

what

• Move helmfiles to https://github.com/cloudposse/helmfiles

why

• Allow helmfiles to be versioned independently

references

• cloudposse/helmfiles#2

what

 - etcdMembers:
    {{- range (getenv "KOPS_AVAILABILITY_ZONES" | strings.Split ",") }}
    - encryptedVolume: true
 ...
    {{- end }}

what

• Add support for running Geodesic shells on Windows using WSL

why

• We have some early adopters running windows
• Minimal effort seems required to get this working on windows (seems very close!)

references

• cloudposse-archives/root.cloudposse.co#6
• #199

what

• Use inline values

why

• More concise format
• Does not require external file
• Does not require YAML-selectors

example

what

• Every time bash -l is run (e.g. by multiple concurrent sessions in the same container), then a new syslog-ng daemon is spawned

PID   USER     TIME  COMMAND
    1 root      0:00 /bin/bash -l
    7 root      0:00 aws-vault server
   28 root      0:00 ssh-agent
   31 root      0:00 {syslog-ng} supervising syslog-ng
   32 root      0:00 syslog-ng -f /etc/syslog-ng/syslog-ng.conf
   37 root      0:00 aws-vault exec --assume-role-ttl=1h --server cpco-testing-admin -- bash -l
   46 root      0:00 bash -l
   65 root      0:00 {syslog-ng} supervising syslog-ng
   66 root      0:00 syslog-ng -f /etc/syslog-ng/syslog-ng.conf
  796 root      0:01 kops rolling-update cluster --yes
  812 root      0:00 bash -l
  839 root      0:00 {syslog-ng} supervising syslog-ng
  840 root      0:00 syslog-ng -f /etc/syslog-ng/syslog-ng.conf
  870 root      0:00 ps uxaww

why

• /etc/profile.d/syslog-ng.sh doesn't check for already running instance

what

• Use helm from `packages

references

• cloudposse/packages#12

what

• aws-vault: error: Server failed: exit status 2

why

• aws-vault server should only be run once globally across all shell because it binds to 169.254.169.254

references

• cloudposse-archives/root.cloudposse.co#8 (comment)

what

pidof: unrecognized option: x
BusyBox v1.28.4 (2018-05-30 10:45:57 UTC) multi-call binary.

Usage: pidof [OPTIONS] [NAME]...

List PIDs of all processes with names that match NAMEs

	-s	Show only one PID
	-o PID	Omit given pid
		Use %PPID to omit pid of pidof's parent

why

• pidof on busybox/musl supports different arguments

what

• Use shfmt to format all shell scripts
• Consider adding target to build-harness

why

• Improve consistency

what

• https://github.com/wercker/stern

why

• Facilitate debuging/developing workflows

what

• Add script to easily bootstrap .aws/config from run-time environment settings
• IAM username should be prompted for
• Should define some canonical ENVs and refactor our stanard TF_VAR_* envs to use these

why

• Make it easier for first-time users to get up and running quickly

example

#!/usr/bin/env bash

AWS_SOURCE_PROFILE="cpco"
AWS_PROFILE="cpco-root-admin"
AWS_REGION="us-west-2"
AWS_IAM_ROLE_ARN="arn:aws:iam::847230548837:role/${AWS_PROFILE}"
AWS_IAM_MFA_SERIAL="arn:aws:iam::847230548837:mfa/[email protected]"

function configure() {
  # When creating a new/non-existent profile, the `aws configure` command gets confused if `AWS_PROFILE` or `AWS_DEFAULT_PROFILE`
  # are set to something which does not yet exist. Running it in `env` lets us sanify the environment.

  echo "[$AWS_PROFILE] $1=$2"
  env -u AWS_PROFILE -u AWS_DEFAULT_PROFILE aws configure set "profile.${AWS_PROFILE}.$1" "$2"
}

configure "region" "$AWS_REGION"
configure "role_arn" "$AWS_IAM_ROLE_ARN"
configure "mfa_serial" "$AWS_IAM_MFA_SERIAL"
configure "source_profile" "$AWS_SOURCE_PROFILE"

maybe we should also call this command if source profile not yet defined:

aws-vault add ${AWS_SOURCE_PROFILE}

what

• Deprecate the helmfile.yaml

why

• Moved to https://github.com/cloudposse/helmfiles

what

# Allow the manifest to be extended via a datasource
{{if (datasourceExists "extensions")}}
{{include "extensions"}}
{{end}}

why

• Allow users to extend the manifest

use-case

Add custom instance pools.

e.g.

---
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: {{getenv "KOPS_CLUSTER_NAME"}}
  name: example-nodes
spec:
  detailedInstanceMonitoring: {{ getenv "KOPS_CLOUDWATCH_DETAILED_MONITORING" "true" }}
  {{- if getenv "NODE_IG_SG_IDS" }}
  additionalSecurityGroups:
    {{- range (getenv "NODE_IG_SG_IDS" | strings.Split ",") }}
  - {{.}}
    {{- end }}
  {{- end}}
  associatePublicIp: false
  cloudLabels:
    Role: k8s-vortex-node
  image: {{ getenv "KOPS_BASE_IMAGE" }}
  machineType: {{getenv "EXAMPLE_MACHINE_TYPE"}}
  maxSize: {{getenv "EXAMPLE_MAX_SIZE"}}
  minSize: {{getenv "EXAMPLE_MIN_SIZE"}}
  nodeLabels:
    dedicated: {{getenv "EXAMPLE_LABEL" | default "example" }}
  role: Node
  subnets:
  - {{getenv "AWS_REGION"}}a
  taints:
  - dedicated={{getenv "EXAMPLE_LABEL" | default "example" }}:NoSchedule

what

• Install the helm-s3 plugin

why

• Avoid the need to manage chart repository

references

• https://github.com/hypnoglow/helm-s3

what

• helmfile.yaml sets ingress.tls[0].secretName and ingress.tls[0].hosts[0] for the chartmuseum releases
• Make these values optional

why

• Allow chartmuseum charts to be installed without tls

What

• On start got error that ssh agent is down

Steps to reproduce

• Run geodesic

Expect

• No error

Got

# Mounting /home/goruha into container
# Attaching to existing geodesic session
Available commands:
  leave-role      Leave the current role; run this to release your session
  assume-role     Assume a new role; run this to renew your session
  setup-role      Setup a new role; run this to configure your AWS profile
  use-profile     Use a preconfigured profile; run this to use an AWS profile without assumed roles

Error connecting to agent: Connection refused
Error connecting to agent: Connection refused
Makefile:145: recipe for target 'add-ssh-key' failed
make: *** [add-ssh-key] Error 2

what

• Update helmfile.yaml to support cloudposse/charts#149

why

• New portal uses different values

When I run this (in geodesic or the Dockerfile)

s3 fstab TF_BUCKET /dev /secrets/tf/

The fstab contains:

s3fs#TF_BUCKET:/dev /secrets/tf/ fuse _netdev,allow_other,rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions 0 0

what

mount -a should not produce error

why

s3fs script should check if filesystem is already mounted

what

• Provide a standard interface for shell extensions
• Re-use some existing standard

why

• Allow users to define their own aliases, prompts, inputrcs, etc

references

• https://github.com/dotphiles/dotphiles
• https://github.com/TheLocehiliosan/yadm

what

Concerning version 0.12.6. Also reproduced in 0.11.0.

Editing historical command lines using emacs keys is broken in certain circumstances. My guess is that it has to do with a divergence between the actual length of the prompt when output versus the length of the prompt when queried by the command line editor.

Reproducing this bug is a little tricky. This seems to work for me, but you might have to try some variations.

1. Enter a long command at the command line:

   echo top level this is a long command
   

2. Press the up-arrow ↑ to recall the command to the command line.
3. Press ctrl-A to move the cursor to the beginning of the command line.

Expected: cursor hovers over "e" in echo.
Observed: cursor hovers over "c" in echo.

why

Not only does this cause difficulty in editing historical command lines, it results in a dangerous situation where the command visible on the command line is not exactly what will be submitted when you hit return.

what

• Add env flag to enable/disable RBAC (default to off, for backwards compatibility)

why

• RBAC is the new norm (for almost a year now)
• We need to start supporting it

what

• Mounting $HOME to /localhost is not working on WSL (Windows Shell for Linux)

why

• Explain why this is a problem and what is the expected behavior.
• Explain why this feature request or enhancement is beneficial.

examples

This works:

docker run -it --rm -v /C/Users/sebas_000/AppData/Local/lxss/home/martaver:/test alpine sh

This does not:

docker run -it --rm -v /home/martaver:/test alpine sh

Maybe we should do something like in the wrapper script if we detect WSL:

mount --bind /mnt/c /c

or

mount --bind /C/Users/sebas_000/AppData/Local/lxss/home/martaver /home/martaver

references

• cloudposse-archives/root.cloudposse.co#8
• microsoft/WSL#1854
• https://superuser.com/questions/1344407/docker-on-wsl-wont-bind-mount-home
• https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly

what

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

why

• To support scaling cluster for GPU Instances

references

• https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md
• https://github.com/kubernetes/kops/tree/f400a882d7c274a77d95050452d263d0faf42005/addons/cluster-autoscaler

what

• Running cloud config unmount fails

what

• Add support for the IAM authenticator for kubernetes

why

• RBAC for users

references

• https://github.com/kubernetes-sigs/aws-iam-authenticator

depends on

• cloudposse/packages#64

 ⧉  geodesic
❌   (none) ~ ➤  l /etc/bash_completion.d/
total 372
lrwxrwxrwx 1 root root     37 Jun  7 18:18 aws.sh -> /usr/local/aws/bin/aws_bash_completer

but it is now at:

❌   (none) ~ ➤  which aws_bash_completer 
/usr/bin/aws_bash_completer

My environment:

OS:

sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.6
BuildVersion:	16G29

ls -lah ~/.aws/
total 16
drwxr-xr-x   5 sweetops  staff   170B Sep 12 22:50 .
drwxr-xr-x+ 77 sweetops  staff   2.6K Sep 20 14:09 ..
drwxr-xr-x   3 sweetops  staff   102B Sep 12 22:50 cli
-rw-------   1 sweetops  staff   444B Sep 13 18:58 config
-rw-------   1 sweetops  staff   341B Sep 13 19:30 credentials

Steps for reproduce:

docker run --rm -it test.com > ./geodesic
chmod 755 geodesic
./geodesic use --name=test.com --dev

what

• Remove packer from this image

why

• It's not used by any of our projects
• We should provide it as part of cloudposse/packages for optional inclusion

references

• cloudposse/packages#47

what

• Set memory and CPU limits for Kiam

why

• Kiam may have a memory leak

references

• uswitch/kiam#72
• uswitch/kiam#125

what

• Look for a ~/.geodesic/env file and pass it as --env-file, if found

why

• Provide a way to set some defaults for all geodesic shells.

what

• Release https://github.com/cloudposse/geodesic/tree/0.16.1 broke shell

How to reproduce

• make build - Build geodesic module from geodesic:0.16.1
• make install - Install geodesic module
• $CLUSTER_NAME - run into shell

Expect

• Got shell command line

Exists

goruha@goruha-laptop ~/projects/cloudposse/example.com (feature-collect-logs●●)$ example.com                                                  [ruby-2.5.1p57]
# Mounting /home/goruha into container
# Starting new example.com session from r.cfcr.io/example.com:latest
# Exposing port 39257

• When I click Ctlr+D I got

goruha@goruha-laptop ~/projects/cloudposse/example.com (feature-collect-logs●●)$ example.com                                                  [ruby-2.5.1p57]
# Mounting /home/goruha into container
# Starting new example.com session from r.cfcr.io/example.com:latest
# Exposing port 39257
^C04da1f47d29c:~#

what

• Add support for Docker for Mac (DFM) Kubernetes or Minikube

why

• Faster LDE, protyping
• Testing Helm Charts, Helmfiles

howto

I got it working very easily.

Here's what I did (manually):

1. Enable Kubnernetes mode in DFM.

2. Disable the DOCKER_DNS stuff that points to 8.8.8.8 in the wrapper script as it breaks DFM DNS resolution of docker.for.mac.localhost
3. Update /localhost/.kube/config to use FQHN for Docker host docker.for.mac.localhost
4. export KUBECONFIG=/localhost/.kube/config

sed -i 's,https://localhost:6443,https://docker.for.mac.localhost:6443,g' /localhost/.kube/config

I'd like us to simplify this process so it basically works out-of-the-box.

what

• Do not require bash4
• Would be great if we can support zsh as well.

why

• The default script requires bash4 which enforce to have separate bash installation (e.g. via brew)

what

• Add support for PROMPT_STYLE environment variable that alters the behavior of the shell prompt

why

• The current utf8 heavy prompt breaks some terminals
• Users have mentioned they'd prefer a plain prompt

what

• Add helm-push plugin (e.g. helm plugin install https://github.com/chartmuseum/helm-push)

why

• Support codefresh chartmuseum repo which uses custom auth scheme

references

• https://codefresh.io/docs/docs/new-helm/managed-helm-repository/

what

• Fix this warning

[2018-08-01T23:13:04.271134] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 3.13 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.;

why

We recently upgraded to alpine:3.8 which bumped the syslog-ng version.

I was hoping to watch your demo but after 2 minutes of waiting and staring at rotating backslash I gave up.
Why don't you use asciinema for this kind of thing. It can cut long pauses or provides a slide to skip certain parts of a video.

what

• Move all python requirements to requirements.txt
• Use multi-stage build pattern as described here: https://blog.realkinetic.com/building-minimal-docker-containers-for-python-applications-37d0272c52f3

FROM python:3.6-alpine as base
FROM base as builder
RUN mkdir /install
WORKDIR /install
COPY requirements.txt /requirements.txt
RUN pip install --install-option="--prefix=/install" -r /requirements.txt
FROM base
COPY --from=builder /install /usr/local

why

• Rely on python package management
• Automatic package updates by Dependabot

what

• Add bats-core

why

• Automated tests

what

• https://github.com/cloudposse/geodesic/blob/master/rootfs/conf/kops/helmfile.yaml

why

• More maintainable

related

• roboll/helmfile#160

• roboll/helmfile#137

• roboll/helmfile#151

• Update documentation for for helmfile.d

If you start a new project following the Quick Start guide, would the expectation be that the example.foo.bar website matches root.cloudposse.co Dockerfile?

This isn't a bug-report per-se but a new-user feedback list and items that would be needed to complete the documentation:

• README needs installation section.
• Install script needs to detect and use unix line-endings where appropriate
• aws-vault needs to be fully functional prior to installation
• New project setup guidance would be helpful

• I'm still struggling with how to use geodesic with an existing infrastructure. For instance, I have a full terraform project. How would I start using geodesic with that project? I'm sure you mount it somehow when you start geodesic, but I don't really see that option.
• The cloudeposse/geodesic output script (which you pipe into bash to install, which isn't documented) is in windows line-endings. I had to execute docker run --rm -it $IMAGE | tr -d "\r" > $TMP && bash $TMP 0.9.17
• aws-vault linux support is half-baked. I'm still struggling to get it fully working. It seems doing a backend=file helps, but there's still some weirdness I'm trying to understand
• Why does geodesic need to run with --privileged? If it's a must, then this complicates using aws-vault in both geodesic and native (geodesic will create vault keys as root in your home directory)

what

• aws configure does not respect AWS_DATA_PATH or AWS_CONFIG_FILE

why

• When running aws configure it always writes to ~/.aws regardless of the official ENVs.

what

• Use TLS for apk repositories
• Use the sed hack to use fastly upstream CDN

why

• More secure
• Mitigate MITM attacks

references

• gliderlabs/docker-alpine#184

what

• https://github.com/uswitch/kiam

why

• Numerous stability issues with kube2iam
• Kiam adds caching support, better security, better overall architecture
• Highly maintained