clusterinthecloud / docs Goto Github PK

I'm attempting to set up a small cluster-in-the-cloud on my OCI tenancy. The only changes I've made to the original are to use a couple of VM.GPU2.1 nodes as the compute nodes. It all appeared to be going well, until the finish script failed at:

rc = subprocess.call(['ansible-playbook', '--inventory=/home/opc/hosts', '--extra-vars=@/home/opc/users.yml', 'finalise.yml'], cwd='/home/opc/slurm-ansible-playbook')

with the following output:

[opc@mgmt ~]$ ./finish

PLAY [finalise] *********************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************************************************************************************************************************************************************************************
fatal: [mgmt]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"mgmt\". Make sure this host can be reached over ssh", "unreachable": true}
fatal: [compute002]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"compute002\". Make sure this host can be reached over ssh", "unreachable": true}
fatal: [compute001]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"compute001\". Make sure this host can be reached over ssh", "unreachable": true}
        to retry, use: --limit @/home/opc/slurm-ansible-playbook/finalise.retry

PLAY RECAP **************************************************************************************************************************************************************************************************************************************************************************************************************
compute001                 : ok=0    changed=0    unreachable=1    failed=0   
compute002                 : ok=0    changed=0    unreachable=1    failed=0   
mgmt                       : ok=0    changed=0    unreachable=1    failed=0   


#############################################
Error: Ansible run did not complete correctly
#############################################

I'm able to ssh into the instances, so I'm not sure what's going wrong here? Any help would be much appreciated!

Blindly following the docs I added clinfo to my compute_image_extra.sh but the package wasn't found.

    amazon-ebs.aws: Last metadata expiration check: 0:01:40 ago on Thu 10 Dec 2020 02:25:36 PM UTC.
    amazon-ebs.aws: No match for argument: clinfo
==> amazon-ebs.aws: Error: Unable to find a match: clinfo

https://cluster-in-the-cloud.readthedocs.io/en/latest/running.html#configuring-node-images
https://github.com/clusterinthecloud/docs/blob/master/source/running.rst

Now that clusterinthecloud/terraform#42 is fixed, we should update the documentation to use it. It should simplify matters a tad.

Hi,

I just updated to the newest version of CitC and tried to start a cluster. When the ansible notebook is played, I get the following error:

Starting Ansible Pull at 2019-04-15 06:12:47
/usr/bin/ansible-pull --url=https://github.com/ACRC/slurm-ansible-playbook.git --checkout=master --inventory=/root/hosts management.yml
[WARNING]: Could not match supplied host pattern, ignoring: mgmt
[WARNING]: Your git version is too old to fully support the depth argument.
Falling back to full checkouts.
mgmt.subnetad1.clustervcn.oraclevcn.com | CHANGED => {
"after": "ece81babcd538714aa1a61a57c7eee75a51a79c6",
"before": null,
"changed": true
}
[WARNING]: Could not match supplied host pattern, ignoring: mgmt

PLAY [finisher script] *********************************************************
...
TASK [slurm : install Slurm] ***************************************************
Monday 15 April 2019 06:14:55 +0000 (0:00:00.221) 0:02:04.207 **********
failed: [mgmt.subnetad1.clustervcn.oraclevcn.com] (item=slurmctld) => changed=false
item: slurmctld
msg: No package matching 'slurm-slurmctld-17.11*' found available, installed or updated
rc: 126
results:

• No package matching 'slurm-slurmctld-17.11*' found available, installed or updated
  failed: [mgmt.subnetad1.clustervcn.oraclevcn.com] (item=slurmdbd) => changed=false
  item: slurmdbd
  msg: No package matching 'slurm-slurmdbd-17.11*' found available, installed or updated
  rc: 126
  results:
• No package matching 'slurm-slurmdbd-17.11*' found available, installed or updated
  ....
  PLAY RECAP *********************************************************************
  mgmt.subnetad1.clustervcn.oraclevcn.com : ok=37 changed=35 unreachable=0 failed=1

Sincerely,
Joe

Since it is recommended to compile applicaitons on a compute node (rather than management node) I think it might be helpful to give a couple of general pointers to how to edit /home/citc/compute_image_extra.sh to get the node images ready for general compilation, e.g. uncomment

sudo yum -y groupinstall "Development Tools"

and install Python with unversioned Python

sudo yum -y install python3
sudo alternatives --set python /usr/bin/python3

$ vim terraform.tfvars
There's a few variables which can be changed in here, but by default you should not have to change anything.

I found the above confusing, as I did have to change that file to include the public keys I want to use to log into the management machine.

Without adding them in, I couldn't log into the mgmt node.

Following https://diataxis.fr we could reorganise the docs to split them into tutorials, how-tos and explanation. There's not much of a place for "reference" in CitC I think though.

The reorganisation could look like:

• Tutorials
  • Installing a cluster
  • Running your first job
  • Deleting a cluster
• How-tos
  • Setting up ARM nodes on AWS
  • Setting up GPU nodes
  • Installing and managing software
  • Updating CitC
• Explanation
  • How the elastic scaling works
  • Why are nodes managed by name in the way they are

etc.