jrmxn / habaneromsubmit

For submission of Matlab functions/scripts (with Matlab functions/scripts) to Columbia University's Habanero HPC.

Introduction

Author:	James McIntosh
Version:	0.0.3

Purpose

I was getting tired of writing custom submission scripts for every project I was working on. These are basic scripts to automatically submit multiple parallel jobs to Columbia's Habanero HPC. Includes various temporary directory settings to try to make the parallelisation more stable. (Each folder has submission scripts for a different project). Should work with other SLURM systems but paths may require tweaking. Tested mainly on Ubuntu 18.04, although should now work on Windows systems too.

Other info

https://wikis.cuit.columbia.edu/confluence/display/rcs/Habanero+-+Job+Examples https://github.com/gaoyuanjun/hpc_tutorial

Quick Submit

Archives the folder you specify, sends it to the cluster and runs the function you specify within that folder.

Makes use of SSH/SFTP/SCP For Matlab (v2) to connect to the cluster. This should be downloaded and placed in the auxf directory. Also uses (already in auxf) findobj so that the passwords that get used don't stay in Matlab history (definitely not guaranteed). It's also possible to load the password from a .mat file stored on disk (which would usually be stored encrypted).

Example use:

matdir = '/hdd/Cloud/project1';
matname = 'project1';%project name
d.n_par = 24;
d.walltime = '03:00:00';
matfunc = "function_name_to_run";%runs function_name_to_run.m
hab_submit(columbia_id,account_type,matfunc,matdir,matname,'n_par',d.n_par,'walltime',d.walltime);

Will move matdir to the cluster after archiving into a folder on Habanero named project1. Will then run "function_name_to_run" from within project after starting the parallel pool.

If project1 results are being written (for example) to folder 'sim' in 'project1', then sim folder can be retrieved like so:

hab_recover(columbia_id,account_type,'sim','project1');