Automatic slurm allocations

[MilesCranmer]

This adds the cluster_manager="slurm_native" option which uses https://github.com/kleinhenz/SlurmClusterManager.jl. The idea with this is that you could potentially just have

sbatch -N 2 --ntasks-per-node=64 search.py

and it will "just work" and set up the search over the cluster. This is the much more standard way to use slurm. This will get the right slurm configurations from the environment variables.

TODO:

• Do a multi-node test. Maybe @wkharold would be interested in testing too?
• Set up heap size hint We don't actually need to do this! Since slurm will partition the memory for us and each Julia process will be aware.
• Add a detailed example to docs for how to set this up.

[coveralls]

Pull Request Test Coverage Report for Build 13483409665

Details

• 11 of 26 (42.31%) changed or added relevant lines in 3 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.7%) to 92.664%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pysr/sr.py	0	1	0.0%
pysr/julia_extensions.py	9	14	64.29%
pysr/julia_helpers.py	2	11	18.18%

Totals
Change from base Build 13478272092:	-0.7%
Covered Lines:	1440
Relevant Lines:	1554

---

💛 - Coveralls

[MilesCranmer]

Ok it turns out we need the following PRs to be merged first before this will actually work:

• Add support for params[:env] (that is, the env kwarg to the addprocs() function) JuliaParallel/SlurmClusterManager.jl#20
• By default, propagate Base.active_project(), Base.LOAD_PATH, and Base.DEPOT_PATH to the workers (but allow the user to override that behavior by specifying JULIA_{PROJECT,LOAD_PATH,DEPOT_PATH} in params[:env]) JuliaParallel/SlurmClusterManager.jl#30

But looks like things are moving fast there which is good. Hopefully not too long.

[DilumAluthge]

I should hopefully be able to get the remaining upstream work done soon-ish.

[MilesCranmer]

Awesome. Thanks!

[codecov]

Codecov Report

Attention: Patch coverage is 42.30769% with 15 lines in your changes missing coverage. Please review.

Project coverage is 92.84%. Comparing base (a23b847) to head (88554ce).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
pysr/julia_helpers.py	18.18%	9 Missing ⚠️
pysr/julia_extensions.py	64.28%	5 Missing ⚠️
pysr/sr.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #794      +/-   ##
==========================================
- Coverage   93.54%   92.84%   -0.70%     
==========================================
  Files          21       21              
  Lines        1565     1580      +15     
==========================================
+ Hits         1464     1467       +3     
- Misses        101      113      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.