%b may be wrong for pulling GRES

[johrstrom]

From discourse: https://discourse.openondemand.org/t/system-status-in-ood-4-0-not-showing-gpu-utilization/4024/3

Apparently %b may not be appropriate for GRES output. The discourse thread seems to imply %b is TresPerNode not AllocTres which is a more accurate accounting of the resources allocated to a job.

ood_core/lib/ood_core/job/adapters/slurm.rb

Line 322 in ec19744

gres: "%b", # must come at the end to fix a bug with Slurm 18

[treydock]

I see no mention of %b in the man page for squeue. If you want TRES allocated you have to use -O/--Format flag of tres-alloc.

$ squeue -u $USER -O tres-alloc
TRES_ALLOC
cpu=120,mem=515456M,

I also see no mention of %b in the Slurm source code related to squeue but maybe my grep was wrong.

Another example:

$ squeue -t R -O tres-alloc:100
TRES_ALLOC
<SNIP>
cpu=10,mem=100G,node=1,billing=10,gres/gpu=1,gres/gpu:a100=1

[johrstrom]

Yea it seems to be vestigial and hidden flag.

https://github.com/SchedMD/slurm/blob/2ddd8e28decbcf6cf36eb4bc50625e9c7860e050/src/squeue/opts.c#L783

What's worse is these two seem to be incompatible. For example we could use tres-per-node if we still wanted this erroneous thing, but the full names seem to be incompatible with the % type format flags. For example we're using %a for account, but we can't just swap that out with Account.

[johrstrom]

We're getting the GRES info out of %b which isn't reporting this GPU.

[johrstrom ~()] 🐷  squeue -j 2258809  -M ascend --format %b
CLUSTER: ascend
TRES_PER_NODE
N/A

When we likely should use whatever the flag for tres-alloc is.

[johrstrom ~()] 🐹  squeue -j 2258809  -M ascend -O 'tres-alloc:100'
CLUSTER: ascend
TRES_ALLOC                                                                                          
cpu=16,mem=171584M,node=1,billing=16,gres/gpu=1,gres/gpu:a100=1

But I think this points to a bigger issue in that we can't mix and match these % formats with word formats. So we'd likely need to refactor the entire thing to word formats to get this.