[BUG] SLURM autodetection causes OF3 to hang on compute cluster

[reedharrison]

Describe the bug
When an OpenFold3 job is run using SLURM, the SLURM job ends up spawning an excessive number of processes (likely due to the SLURM configuration). It would be preferable to disable the SLURMEnvironment and only use local resources. In my case, the job has been submitted to a VM with finite resources and the OpenFold3 job oversubscribes the CPU/GPU leading to the job appearing to hang on the cluster.

To Reproduce
Reproducing the issue may be difficult if you don't have a compute cluster configured in a similar way; however, the SLURM autodetection is a common problem with Pytorch Lightning. Note that my OpenFold3 job is not run in Docker. While that might be a solution for me in the long term, it would be nice if we could prevent SLURM autodetection as command-line argument.

Expected behavior
To avoid this issue, the run command could accept one or more arguments to disable SLURM autodetection or to manually specify the local resources to be used.

[jnwei]

Hi @reedharrison ,

If I understand correctly, the issue occurs when openfold is run interactively, rather than as a job submission? If instead, the workflow is being run as a slurm job, could you please share the slurm job submission script?

A few suggestions for potential workarounds:

• Pytorch lightning suggests using slurm to create an interactive session, and then running the command as usual guide from pl lightning
• You could modify the used strategy / cluster environment, similar to what this pytorch lightning user reports: Disable automatic SLURM Detection Lightning-AI/pytorch-lightning#6389 (comment)

When I looked at this issue, I wasn't able to identify set of pytorch lightning flags / arguments that can be passed to the trainer to resolve this issue. However, if you find a solution that requires passing arguments to the pytorch lightning Trainer class, you can do so in the pl_trainer_args section of the runner.yml.

[reedharrison]

Hi @jnwei ,

Unfortunately, I can't share the submission script - it isn't something that I can manage or change. The job is not run interactively, but it might be submitted in a slightly more complicated way than just being run as a slurm job.

I found similar PyTorch issues as you and tried the following:

1. Delete all environment variables containing "SLURM". This didn't solve the problem.
2. Reset all environment variables to the bare minimum needed to execute the Python script. This solved my problem - I'm not sure if some executable was still in PATH and PyTorch was finding it.

I did try to change the pl_trainer_args to control resources that would be used, and it did use a single GPU; however, it ended up spawning many more CPU child processes that caused the job to hang. I didn't see a way to control this. Changing data_module_args didn't seem to work for me, but perhaps I did it wrong.

It might be worth considering if you want users to be able to specify exact resources that the job should use. That could include the compute context - a local job, a slurm job, etc.

Otherwise, I think we can just close this for now.

[jnwei]

Thanks @reedharrison I'm glad you were able to resolve the issue for your use case!

We are currently relying heavily on pytorch lightning to manage the compute cluster environment, and the interaction with slurm based infrastructure can be tricky. To customize the cluster environment, we would need additional configuration settings to change the cluster / strategy used, possibly here

We could publish sample slurm scripts for running inference if that would be a helpful resource.

[reedharrison]

@jnwei , If I recall correctly, I noticed that "cluster_environment" was returning "None" here after I removed all "SLURM" environment variables. The job would still spin up too many CPU processes that caused the job to hang.

Slurm scripts might generally be helpful for people to know, but I was specifically trying to not use slurm.