Note
TLDR: Slurm, but with job extension capabilities!
Melon
is a lightweight distributed job scheduler written in Rust. This is a minimal MVP I wrote for fun while complaining about slurm. The one and only reason I worked on this is that slurm doesn't support extending the allocated time for a job. Melon does! No more timeout errors!! 🥳🥳
Components that have been implemented so far:
-
melond
the melon daemon master node -
mbatch
the submit command -
mworker
compute node -
mqueue
see pending and running jobs -
mcancel
the cancel command -
mextend
to extend jobs -
mshow
a better job info command -
mstat
the job status commands
Stuff that's missing:
- master node recovery
- compute node recovery
- job info persistence
- cgroups management
Install all tools using ./install-all.sh
Run the melond
service. This is the master node that the other tools communicate with. You can override the local port melond --port 8081
. All tools use IPv6 by default. So if you're behind a docker instance, make sure to follow the guide at the end.
melond --port 8081
# for pretty-print
cargo install bunyan
melond --port 8081 | bunyan
Then spin up a couple of mworker
compute nodes. Currently the available resources are read from the environment but that will change in the future (MVP, remember?). On Linux, we use cgroups
to limit the available resources per job, based on the memory and cpu requirements that were requested. On Mac/Windows we just yolo. In order to use cgroups
make sure to run the mworker
command with sudo. Almost all cool features from slurm are of course not implemented. But the basics will get you going. You can override the api endpoint and it's own port.
# on linux
sudo mworker --api_endpoint "http://[::1]:8080" --port 8082
# on mac / windows => no resource limits besides timeout though!
mworker --api_endpoint "http://[::1]:8080" --port 8082
Then to submit a job, first create a simple batch script my_job.sh
. We use drop-in replacements for SBATCH
instructions. Creatively as I am, we will use MBATCH
.
#!/bin/bash
#MBATCH -c 4
#MBATCH -t 0-06:00
#MBATCH -m 1G
for i in $(seq 2 100); do
echo "$i melons are better than $((i-1))"
sleep 1
done
And submit the job using mbatch
mbatch my_job.sh
You can also pass parameters to your job script my_job.sh
:
n_parts=10
for i in $(seq 0 $n_parts)
do
mbatch my_job.sh $n_parts $i
done
You can override the api endpoint if needed. Just make sure to override the endpoint before you pass the script and it's arguments.
mbatch --api_endpoint "http://[::1]:8080" my_job.sh arg1 arg2
$ mqueue # again, override using --api_endpoint "http://[::1]:8080"
# Output:
JOBID NAME USER ST TIME NODES
12 test_data/ chris PD 0-00-00 (PD)
13 test_data/ chris PD 0-00-00 (PD)
10 test_data/ chris R 0-00-01 RwcSZDtzXnHm55PBcY6xS
11 test_data/ chris R 0-00-01 RwcSZDtzXnHm55PBcY6xS
To extend the requested time for a pending or running job (the reason for the existence of this whole project...). Again, the --api_endpoint
override argument is available.
# extend by one hour
mextend -j <job_id> -t 0-01-00
# extend by one day
mextend -j <job_id> -t 1-00-00
# extend with override
mextend -j <job_id> -t 1-00-00 --api_endpoint "http://[::1]:8080"
To cancel a pending or running job:
mcancel job_id
# or
mcancel --api_endpoint "http://[::1]:8080" <job_id>
If you're using the devcontainer, make sure to enable ipv6
in your docker engine.
{
"ipv6": true,
"fixed-cidr-v6": "2001:db8:1::/64"
}