fgci-org / fgci-ansible Goto Github PK

With many interfaces that are up - provisioning becomes slow because of DHCP timeouts.

Reported by Ulf

Make a role that sends an annotation to cassini indicating that a configuration change was made with ansible. See https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/master/tools/grafana_annotation.sh and https://github.com/CSC-IT-Center-for-Science/ansible-role-elasticsearch-demo-event/blob/master/tasks/main.yml for examples of how it can be done.

First time it fails/hangs and one must CTRL+C and start the provisioning.yml playbook again.

make the ansible-pull command in ansible-pull-script.sh use a configurable branch.

Reported by Ulf

To make sure the RPMs are cached at the proxy server before continuing.

If your mail-server (for the adminMailAddr e-mail address) has an IPv6 address and IPv6 is kind of disabled (the default) - e-mails to adminMailaddr will not work.

• name: Install FGCI repo
  yum: pkg=http://idris.fgi.csc.fi/fgci7/x86_64/fgci/rpms/fgci-release7-1-1.el7.noarch.rpm
  when: ansible_distribution_major_version == "7"

It would be nice to get it installed by KS

It would be quite useful to have some module usage statistics. First part of this is to write to syslog whenever a module is loaded.

The lua script we use on Taito is available here:
/homeappl/appl_taito/opt/lmod/apps/SitePackage.lua

• rrsync and ssh public key
  • not on by default
  • one must be able to select what data to ship

TASK: [ansible-role-yum | install software that do not need extra configuration] ***
failed: [io-install.fgci.csc.fi] => (item=git,nfs-utils,bash-completion,wget,pdsh) => {"changed": false, "failed": true, "item": "git,nfs-utils,bash-completion,wget,pdsh", "rc": 0, "results": []}
msg: No Package matching 'pdsh' found available, installed or updated

FATAL: all hosts have already failed -- aborting

A MOTD updater would be nice.

Some feature requests:

• pull in data from RSS feed
• different MOTD on different machines in the cluster
• idid / lastdone commands to easily update motd with a small notification?

Reported by Ulf

install node
So it seems that eth1 doesn't come up as it should

https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/master/tools/reinstall_node.yml

• mysqldump
• munge key
• more?

The dns role sets up /etc/resolv.conf so that install node queries itself first. This is OK, after dnsmasq is configured and installed.
Need to run the dns role later.

Since commit https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/b200e101cab6faae9d1f8f17b6b9f36aa8bebe16/tests/Dockerfile or so we run the travis-ci test inside a CentOS7 Docker container. Have not been able to get mariadb to start at all inside this container.

The kickstart_partitions variable in group_vars/login,grid,compute/ is used by ansbile-role-pxe_config and it is not ideal.

On our test compute nodes it creates a /home partition which is not used because they get /home from NFS.

Skip creating a /home and create a /tmp perhaps?

When I run the prep_local.yml playbook, the system did not rename the example files into yml files. But that I could do manually too. The other thing was that tar was in /bin/tar and I had to make a softlink /usr/bin/tar for the script (this way I did get the backup in fgci-ansible/backup).

user's using ubuntu

Blocked by authors of these roles?
The roles:

• Dns
• Ip_forwarder: https://github.com/CSC-IT-Center-for-Science/ansible-role-ip-forwarder
• Network_interface: https://github.com/CSC-IT-Center-for-Science/network_interface
• Provision_vm: https://github.com/CSC-IT-Center-for-Science/ansible-role-provision-vm

Generate SSH host keys and install them on compute nodes.
Currently they must be deleted from all the right places or there will be errors on node reinstall.

The fgci repo's EL7 build of cvmfs-repofiles-fgi does not contain /etc/cvmfs/default.local

These are the file in the FGI EL6 rpm:

CVMFS_CACHE_BASE=/var/cache/cvmfs2
CVMFS_REPOSITORIES=fgi
CVMFS_DEFAULT_DOMAIN=csc.fi
CVMFS_HTTP_PROXY=DIRECT
CVMFS_CACHE_DIR=/var/cache/cvmfs2
CVMFS_QUOTA_LIMIT=20480 

Right now it's:

• reinstall node: https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/master/tools/reinstall_node.yml
• reboot node or set it to state=reboot in slurm and wait for NHC to reboot it
• wait for it to come up and then run the compute.yml playbook

Could we perhaps use ansible-pull on the compute nodes?

http://docs.ansible.com/ansible/playbooks_intro.html

in order to have PXE boot working dnsmasq needs to be installed on install node.
A role have to be built to handle this.
Afterwards /etc/resolv.conf from nodes have to point to install node

https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/master/tools/shutdown_all.yml

Means setting

 adminremove_passwords: True

https://github.com/CSC-IT-Center-for-Science/ansible-role-users/blob/master/defaults/main.yml#L8

We should build a "acceptance test" role that assesses the performance of Infiniband and Disks/RAID.

Interconnect tests
Connectivity tests on each InfiniBand network fabric installed (connected to a switch) on
a cluster.
The FDR Infiniband interconnect tests will consist of Bandwidth tests and Latency tests
on all ports. These will be tested with ib_write_bw and ib_send_lat and to pass the
bandwidth must be better than 5000 MB/s and latency less than 2 microseconds. These
criteria must be met by all point-to-point links between any two nodes.
The tests will be performed with RHEL/CentOS package perftest-2.2-1.el6.x86_64 or
equivalent.

Disk tests
All Disk drives must work without errors, all RAID systems must work without errors.
Disk drives will be tested with smartctl, hdparm, dd to check bandwidth. Output of
dmesg and/or system logs will be checked for seek errors.

Related document: https://confluence.csc.fi/download/attachments/51890567/Liite_3_Hyvaksymismenettely.pdf?api=v2

https://github.com/willshersystems/ansible-sshd

PermitRootLogin without-password

Probably some more safety checks can be added here to make sure all the requirements are set somewhat sanely before continuing.

https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/master/roles/network_interface/templates/ethernet_RedHat.j2#L6

If ext_ip_addr variable is not set on the host - then we'll get

IPADDR={#ext_ip_addr#}

in ifcfg-eth* .

check other nodes as well

A new role or task in an existing role should copy in the x509 certificate to the grid node.

In order to access iDRACs Virtual console, admin node needs javaws

TASK: [ansible-role-fgci-install | synchronize group_vars/ to /var/www/html/group_vars/ - for ansible-pull - secrets and firewalls are not needed] ***
failed: [io-install.fgci.csc.fi -> 127.0.0.1] => {"cmd": "rsync --delay-updates -F --compress --recursive --rsh 'ssh -S none -o StrictHostKeyChecking=no' --exclude=secrets.yml --exclude=trusted_networks.yml --out-format='<>%i %n%L' "/home/lalves/code/fgci-ansible/group_vars/" "[email protected]:/var/www/html/group_vars"", "failed": true, "rc": 127}
msg: X11 forwarding request failed on channel 0
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.1]

FATAL: all hosts have already failed -- aborting

For delivering IP addresses to iDRACs

• a way to include roles
  • could be done similarly to pull_variables.yml
  • also needs a way to ansible-galaxy install these extra roles

Right now the kickstart sets up ansible-pull cronjob that runs every 15 minutes.
Then the ansible-pull itself changes that to every 120 minutes.
This means on a reinstall there's quite a long downtime before the the node is configured.

Possible solution:

• make kickstart run the ansible-pull-script.sh in /etc/rc.local
• have ansible-pull remove it from there and add it as a cronjob

Use {{ ansible_hostname }} instead of {{ inventory_hostname }}

https://github.com/CSC-IT-Center-for-Science/fgci-ansible/blob/master/roles/provision_vm/templates/virt-install.sh

Only have

• nameserver1
• nameserver3

There now, should have namserver2 too.

PXE boot process requires that a file named after the name of the node being installed is created on install-node:/var/www/provision/reinstall/

This needs to be ensured for first installation and possibly a playbook could be run whenever a node needs a reinstall

kickstart files matching kickstart_profile shall be placed on the install node.
boot.py from ansible-role-pxe_bootstrap will try to fetch those from there.

Example:
http://1.1.1.2/ks/FGCI-compute-node

VMs confs need to be changed to survive an hypervisor reboot

[root@io-install ~]# ls -la /var/www/provision/reinstall/
total 8
drwxr-xr-x 2 apache apache 4096 Dec 9 15:55 .
drwxr-xr-x 4 root root 4096 Nov 26 14:06 ..
-rw-r--r-- 1 root root 0 Dec 9 15:55 io1,gpu
-rw-r--r-- 1 root root 0 Dec 9 15:55 io2,gpu
-rw-r--r-- 1 root root 0 Dec 9 15:55 io3,gpu
-rw-r--r-- 1 root root 0 Dec 9 15:55 io4,gpu

Enable systemctl httpd.service through ansible

figure out a way in which the same munge.key file is provided from ansible-pull and admin's workstation

Add an option to enable IPv6:

initially these are required:

• update the network_interfaces role to also configure IPv6
• enable firewalls on compute nodes