ERROR: Resolution loop on 'bids/freesurfer/latest' detected, RHEL 9

[LinuxPersonEC]

The modules do not appear with Lmod via module available. Testing freesurfer, using TCL the module appears as:
bids/freesurfer/latest/module.tcl

ml bids/freesurfer/latest/module.tcl results in:
ERROR: Resolution loop on 'bids/freesurfer/latest' detected

I've switched between TCL and Lmod. We use the Bright Computing provisioning management system but I don't think that is related.

To Reproduce
Steps to reproduce the behavior:

git clone https://github.com/singularityhub/singularity-hpc
cd singularity-hpc
pip install -e .[all]
 /usr/local/bin/shpc install bids/freesurfer:latest
INFO:    Using cached SIF image
Module bids/freesurfer:latest was created.
module use ./modules

This was installed within echo $MODULEPATH

Expected behavior
The module should appear.

apptainer version 1.3.0-1.el9
/usr/local/bin/shpc --version
0.1.28
rpm -qa|grep -i lmod
Lmod-8.7.32-1.el9.x86_64
rpm -qa|grep -i tcl
tcl-8.6.10-7.el9.x86_64
tcl-devel-8.6.10-7.el9.x86_64

Anything else?
If this is a question more than a bug, feel free to move it.

[vsoch]

I'm not great with developing environment modules, but this seems relevant:

@marcodelapierre have you seen this before?

[marcodelapierre]

I am testing with Lmod version 8.7.47, and latest Singularity-HPC, and cannot reproduce.

Would you be able to provide the output of ml av bids, as well as the contents of the directory modules/bids/freesurfer/latest/ ?

Thank you

[LinuxPersonEC]

ml av bids

It does not return any values.

ls modules/bids/freesurfer/latest
99-shpc.sh  bin  module.lua

Since we have both TCL and Lmod installed which option should be used? Is it expected to not load modules when we use Lmod?

[vsoch]

It should work for either or, but I've never tested shpc in an environment with both!

[LinuxPersonEC]

Is there any debugging or comments we can insert to see where the modules are failing to load?

[vsoch]

shpc is just using standard environment modules or lmod, so you'd want to look in documentation for those projects. @marcodelapierre is much better at these modules than me so he might have some advice! I do most in the cloud these days, or user space kubernetes.

[LinuxPersonEC]

Hello @marcodelapierre is there anything I can do to work around this?

[marcodelapierre]

I am surprised that ml av bids does not return anything.
Is the root directory of that module tree in the $MODULEPATH? You should always make sure you have it, module use </modules/root/path>.

I would like to see the outcome of that command in the same shell environment that also generates the Error.

When you say you have both TCL + Lmod installed, I am assuming you mean TCL as the language and then Lmod modules, and that you do NOT also have installed Environment Modules (the TCL variation of Lmod). Is this correct?
In other words, I would expect module --version to reveal that you have an Lmod version installed.

With the assumption above, Lmod can handle both modules written in lua and in tcl. Lua is the native one for Lmod, hence preferable because it has more functionalities; Lmod can handle tcl modules too though.

For what regards SHPC, I cannot test at present, but for what I remember from past exploration, Lmod is capable of correctly using both Lua and Tcl modules generated by SHPC. (as said above, lua preferred because it is the native language of Lmod modules).

[LinuxPersonEC]

I am surprised that ml av bids does not return anything. Is the root directory of that module tree in the $MODULEPATH? You should always make sure you have it, module use </modules/root/path>.

Yes here's what echo $MODULEPATH returns after I load an existing (non SHPC) module:

/cluster/shared/modulefiles/scotch/7.0.5:/cluster/shared/modulefiles/bids/freesurfer/V30-a43f1f/module.lua:/cluster/shared/modulefiles/bids/freesurfer/V30-a43f1f:/cluster/shared/modulefiles/bids/freesurfer:/etc/scl/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/cluster/shared/modulefiles
If I run ml on another non-SHPC module using the full path it shows up like this, with the full path:

ml  /cluster/shared/modulefiles/metis/5.1.0
 ml
Currently Loaded Modulefiles:
 1) anaconda/2023.09   2) scotch/7.0.5   3) /cluster/shared/modulefiles/metis/5.1.0  

These also show up as modules:

singularity-hpc/shpc/main/modules/templates/default_version  
singularity-hpc/shpc/main/modules/templates/docker.tcl       
singularity-hpc/shpc/main/modules/templates/singularity.tcl  
singularity-hpc/shpc/main/modules/templates/view_module.tcl  

I would like to see the outcome of that command in the same shell environment that also generates the Error.

Did the above help?

When you say you have both TCL + Lmod installed, I am assuming you mean TCL as the language and then Lmod modules, and that you do NOT also have installed Environment Modules (the TCL variation of Lmod). Is this correct?
Both are installed at the same time:

$ rpm -qa|grep -i envi
environment-modules-5.3.0-1.el9.x86_64
 rpm -qa|grep -i lmod
Lmod-8.7.32-1.el9.x86_64

In other words, I would expect module --version to reveal that you have an Lmod version installed.

 module --version
Modules Release 5.3.0 (2023-05-14)

With the assumption above, Lmod can handle both modules written in lua and in tcl. Lua is the native one for Lmod, hence preferable because it has more functionalities; Lmod can handle tcl modules too though.

For what regards SHPC, I cannot test at present, but for what I remember from past exploration, Lmod is capable of correctly using both Lua and Tcl modules generated by SHPC. (as said above, lua preferred because it is the native language of Lmod modules).

This must be a clue as to why the 4 singularity-hpc modules appear. The ones ending in .tcl appear:

ls -l /cluster/shared/modulefiles/singularity-hpc/shpc/main/modules/templates
total 83
-rw-r--r-- 1 root root  112 Dec 17 09:26 default_version
-rw-r--r-- 1 root root 7094 Dec 17 09:26 docker.lua
-rw-r--r-- 1 root root 8490 Dec 17 09:26 docker.tcl
-rw-r--r-- 1 root root 5200 Dec 17 09:26 docs.md
drwxr-xr-x 2 root root 4096 Dec 17 09:26 includes
-rw-r--r-- 1 root root    0 Dec 17 09:26 __init__.py
-rw-r--r-- 1 root root 8062 Dec 17 09:26 singularity.lua
-rw-r--r-- 1 root root 9799 Dec 17 09:26 singularity.tcl
-rwxr-xr-x 1 root root  405 Dec 17 09:26 test.sh
-rw-r--r-- 1 root root  256 Dec 17 09:26 view_module.lua
-rw-r--r-- 1 root root  307 Dec 17 09:26 view_module.tcl

Attempting to load one of those gets this error:

ml singularity-hpc/shpc/main/modules/templates/singularity.tcl
Loading singularity-hpc/shpc/main/modules/templates/singularity.tcl
  Module ERROR: invalid command name "% include "includes/load_view.tcl" %"
        while executing
    "{% include "includes/load_view.tcl" %}"
        (file "/cluster/shared/modulefiles/singularity-hpc/shpc/main/modules/templates/singularity.tcl" line 45)
    Please contact <root@localhost>

Using the full path to the module.lua file:

 ml /cluster/shared/modulefiles/bids/freesurfer/V30-a43f1f/module.lua 
Module ERROR: Magic cookie '#%Module' missing
  In '/cluster/shared/modulefiles/bids/freesurfer/V30-a43f1f/module.lua'
  Please contact <root@localhost>

[marcodelapierre]

Mmh OK. I would discourage having both Env Modules and Lmod active in the same shell environment, it adds unnecessary complexity and sources of issues.
Remember that ml is exclusive to Lmod, whereas module exists for both. So in your case you really have both active, as module --version mentions Env Modules.

You should not be seeing these:

singularity-hpc/shpc/main/modules/templates/default_version  
singularity-hpc/shpc/main/modules/templates/docker.tcl       
singularity-hpc/shpc/main/modules/templates/singularity.tcl  
singularity-hpc/shpc/main/modules/templates/view_module.tcl 

I think they come out of using module use on the wrong path.

From your first messages:

cd singularity-hpc
module use ./modules

To correctly use modules, you need to run module use with the modules subdirectory of the SHPC installation.

I would like to see the outcome of ml av bids after you run this.

[marcodelapierre]

What type of privileges do you have on that machine?

It would be good to be able to disable either Lmod or Env modules setup in the shell environment, to test in a more robust environment.

Sourcing of the corresponding scripts will typically be either in your .bashrc/.profile in your home, in which case you can comment out the corresponding line, or in the forms of scripts within /etc/profile.d/ directory. The latter case is trickier as it would require root privileges to move sourceable files out of it.

[LinuxPersonEC]

To correctly use modules, you need to run module use with the modules subdirectory of the SHPC installation.

I would like to see the outcome of ml av bids after you run this.

So within /cluster/shared/apps/singularity-hpc I ran

Tue Dec 17 root@node003 $ module use ./modules
Tue Dec 17 root@node003 $ ml av bids
Tue Dec 17 root@node003 $ ml
No Modulefiles Currently Loaded.
echo $MODULEPATH
/cluster/shared/apps/singularity-hpc/modules:/etc/scl/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/cluster/shared/modulefiles

What type of privileges do you have on that machine?

root

It would be good to be able to disable either Lmod or Env modules setup in the shell environment, to test in a more robust environment.

Sure what would you like me to try?

EDIT I went to another node. This time I made some progress:

[root@prov singularity-hpc]# module use ./modules
[root@prov singularity-hpc]# ml av bids

---------------------- /cluster/shared/apps/singularity-hpc/modules -----------------------
   bids/freesurfer/latest/module

------------------------------- /cluster/shared/modulefiles -------------------------------
   bids/freesurfer/V30-a43f1f/module (D)

So if I do

ml bids/freesurfer/latest/module
[root@2402-prov-002 singularity-hpc]# ml

Currently Loaded Modules:
  1) anaconda/2023.09   2) bids/freesurfer/latest/module



[root@prov singularity-hpc]# free
free                          freesurfer-inspect-deffile    freesurfer-shell
freesurfer-container          freesurfer-inspect-runscript  freetype-config
freesurfer-exec               freesurfer-run     

Yep this is a node that has only Lmod-8.7.32-1.el9.x86_64 so having both definitely confuses the modules themselves.

[marcodelapierre]

Oh great! Looks like the EnvModules vs Lmod clash was indeed causing your issues.

Let us know if you need anything else, or if we can close this one.

[vsoch]

@marcodelapierre 🙌 🙌 !

[LinuxPersonEC]

It would be good to be able to disable either Lmod or Env modules setup in the shell environment, to test in a more robust environment.

Can you suggest a way to do this in the nodes that have both?

Oh great! Looks like the EnvModules vs Lmod clash was indeed causing your issues.

Let us know if you need anything else, or if we can close this one.

Looks like both Lmod and environment-modules are in use to get specific modules to load only on compute nodes, so when trying to uninstall environment-modules it wants to remove mpich, openmpi, etc.

It'd be great to know how to work around this.