'Invalid Trackable RESource (TRES) specification' error about slurm-drmaa

[kimchitsigai]

Hi,

I've added a logging instruction in slurm_drmaa/session.c:
fsd_log_debug(("job_desc.tres_per_node = %s", job_desc.tres_per_node));
and it produced this output:
job_desc.tres_per_node = gpu:1
And finally, I get the Invalid Trackable (TRES) specification RESource error.

By changing the input from --gres=gpu:1 to --gres=gres:gpu:1, the logging instruction outputs:
job_desc.tres_per_node = gres:gpu:1
as expected. The TRES error disappeared and the job was correctly submitted.

By looking at the code in https://github.com/SchedMD/slurm/blob/master/src/common/gres.c and https://github.com/SchedMD/slurm/blob/master/src/common/slurm_opt.c it seems that Slurm is expecting GPU resource requests formatted as gres:gpu:1 and not as gpu:1 as in the previous versions (20.x) of Slurm.

Shouldn't slurm-drmaa be updated to take into account this change in Slurm?

Best,
Kimchi

[scholtalbers]

I can confirm that changing --gres=gpu:1 to --gres=gres:gpu:1 solves the TRES error.

[natefoo]

Thanks, yes, it sounds like we should probably prepend gres: to the string in the job template. gres in the job template was generalized to tres_per_node in 18.08, I am not sure if this is when it broke.