uninett / research-lab

Repo contains code for setting up Kubernetes Federation in various cloud infrastructure

Research Lab Platform

This repository contains scripts and ansible playbooks for managing the Research lab platform.

Bringing up a test cluster in Safespring (IPnett)

In the terraform/deployments/safespring directory, make a local.tfvars based on the example, check your setup with terraform plan --var-file=local.tfvars, then bring up the bare machines with terraform apply --var-file=local.tfvars. The job prints out an ansible inventory at the end.

One can also use the ./run_dev.sh safespring command from the top level directory to do everything.

Bringing up a test cluster in uh-iaas

Prerequisites

• A personal project at uh-iaas, see: http://docs.uh-iaas.no/en/latest/login.html#first-time-login (After you have created your personal project, you can for access to the shared project)
• The API-password for uh-iaas. You can test it with: http://docs.uh-iaas.no/en/latest/api.html
• Terraform: https://www.terraform.io/
• Access to the project

local.tfvars

In the terraform/deployments/uh-iaas directory, make a local.tfvars based on the example.

Download modules

With terraform get, the necessary modules are downloaded to a subfolder .terraform of the working directory. We have used terraform/deployments/uh-iaas as a working directory.

Test and run

Check your setup with terraform plan --var-file=local.tfvars, then bring up the bare machines with terraform apply --var-file=local.tfvars. The job prints out an ansible inventory at the end.

The run_dev.sh script (under development) will in addition create securitygroups and apply the ansible setup.

Debugging / verbose output

For debugging, use the environment variables TF_LOG=DEBUG and OS_DEBUG:

TF_LOG=DEBUG OS_DEBUG=1 terraform apply --var-file=local.tfvars

research-lab's Issues

Problem med security group 'kubetest-ssh_access'

Bruker Federation branch.

Det virker som det blir blir problem når terraform skal opprette security group 'kubetest-ssh_access'.
Forsøker den å opprette den 2 eller flere ganger??

Stikk ord: openstack_networking_secgroup_v2.grp_ssh_access
Slik ser den ut i min state fil.

"openstack_networking_secgroup_v2.grp_ssh_access": { "type": "openstack_networking_secgroup_v2", "depends_on": [], "primary": { "id": "366a774d-01fd-4ffc-8e0e-fa594dc7f9b4", "attributes": { "description": "Security groups for allowing SSH access", "id": "366a774d-01fd-4ffc-8e0e-fa594dc7f9b4", "name": "kubetest-ssh_access", "region": "osl", "tenant_id": "e4f777d28d9f43b3907683892da06cf9" },

Logg

First run

Error applying plan:

4 error(s) occurred:

• module.masters.openstack_compute_instance_v2.master[0]: 1 error(s) occurred:

• openstack_compute_instance_v2.master.0: Error waiting for instance (9f8cdf88-ca44-491c-b19e-20571b7c4f51) to become ready: unexpected state 'ERROR', wanted target 'ACTIVE'. last error: %!s()

• module.masters.openstack_compute_instance_v2.master[1]: 1 error(s) occurred:

• openstack_compute_instance_v2.master.1: Error waiting for instance (7db182a1-1f46-42c6-a020-37dcad104de9) to become ready: unexpected state 'ERROR', wanted target 'ACTIVE'. last error: %!s()

• module.workers.openstack_compute_instance_v2.worker[2]: 1 error(s) occurred:

• openstack_compute_instance_v2.worker.2: Error waiting for instance (3df7890d-bd4d-420c-a7d5-4befbf8fe969) to become ready: unexpected state 'ERROR', wanted target 'ACTIVE'. last error: %!s()

• module.securitygroups.openstack_networking_secgroup_rule_v2.rule_ssh_access_ipv4[0]: 1 error(s) occurred:

• openstack_networking_secgroup_rule_v2.rule_ssh_access_ipv4.0: Expected HTTP response code [201 202] when accessing [POST https://network.api.osl.uh-iaas.no:9696/v2.0/security-group-rules], but got 409 instead
  {"NeutronError": {"message": "Security group rule already exists. Rule id is 2a22016f-62b3-469e-ab6b-5445b13fd522.", "type": "SecurityGroupRuleExists", "detail": ""}}

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Second run

Error applying plan:

4 error(s) occurred:

• module.securitygroups.openstack_networking_secgroup_rule_v2.rule_ssh_access_ipv4[0]: 1 error(s) occurred:

• openstack_networking_secgroup_rule_v2.rule_ssh_access_ipv4.0: Expected HTTP response code [201 202] when accessing [POST https://network.api.osl.uh-iaas.no:9696/v2.0/security-group-rules], but got 409 instead
  {"NeutronError": {"message": "Security group rule already exists. Rule id is 2a22016f-62b3-469e-ab6b-5445b13fd522.", "type": "SecurityGroupRuleExists", "detail": ""}}

• module.masters.openstack_compute_instance_v2.master[1]: 1 error(s) occurred:

• openstack_compute_instance_v2.master.1: Error adding security group (kubetest-ssh_access) to OpenStack server (7db182a1-1f46-42c6-a020-37dcad104de9): Resource not found

• module.masters.openstack_compute_instance_v2.master[0]: 1 error(s) occurred:

• openstack_compute_instance_v2.master.0: Error adding security group (kubetest-ssh_access) to OpenStack server (9f8cdf88-ca44-491c-b19e-20571b7c4f51): Resource not found

• module.workers.openstack_compute_instance_v2.worker[2]: 1 error(s) occurred:

• openstack_compute_instance_v2.worker.2: Error adding security group (kubetest-ssh_access) to OpenStack server (3df7890d-bd4d-420c-a7d5-4befbf8fe969): Resource not found

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Ansible script should have different workflow for windows and linux workers

When the ansible script reads the inventory

[akube01-workers]
akube01-worker-0 ansible_host=xx.yy.ss.ii lb=true
akube01-worker-1 ansible_host=xx.yy.ss.ii lb=true
akube01-worker-2 ansible_host=xx.yy.ss.ii lb=true
akube01-worker-3 ansible_host=xx.yy.ss.ii lb=false

The ansible script expect that all workers are Linux.
To enable "support" for windows. The Ansible script needs to have different workflow for linux and windows workers.

Suggestion:

[akube01-workers]
akube01-worker-0 ansible_host=xx.yy.ss.ii lb=true ostype=windows
akube01-worker-1 ansible_host=xx.yy.ss.ii lb=true  ostype=linux
akube01-worker-2 ansible_host=xx.yy.ss.ii lb=true ostype=linux
akube01-worker-3 ansible_host=xx.yy.ss.ii lb=false ostype=linux

Could ostype as part of the inventory be used to have different handling of linux & windows workers?