resolve templates in `CmdCp` easyblock by branfosj · Pull Request #3918 · easybuilders/easybuild-easyblocks

branfosj · 2025-09-10T11:28:48Z

(created using eb --new-pr)

Completes the fix for easybuilders/easybuild-framework#4990

The switch to self.cfg.get_ref('cmds_map') fixes

ERROR: Installation of STREAM-5.10-GCC-11.3.0.eb failed: 'Failed to resolve all templates in "$CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=10000000 -DNTIMES=1000 -o stream_1Kx10M; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=1000 -o stream_1Kx100M; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=1000000000 -DNTIMES=1000 -o stream_1Kx1B; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=2500000000 -DNTIMES=1000 -o stream_1Kx2.5B; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=5000000000 -DNTIMES=1000 -o stream_1Kx5B; " using template dictionary: {\'arch\': \'x86_64\', \'rpath_enabled\': \'true\', \'sysroot\': \'\', \'software_commit\': \'\', \'nameletter\': \'S\', \'toolchain_name\': \'GCC\', \'toolchain_version\': \'11.3.0\', \'version_major\': \'5\', \'version_minor\': \'10\', \'version_major_minor\': \'5.10\', \'bitbucket_account\': \'stream\', \'github_account\': \'stream\', \'name\': \'STREAM\', \'version\': \'5.10\', \'versionsuffix\': \'\', \'versionprefix\': \'\', \'namelower\': \'stream\', \'nameletterlower\': \'s\', \'cuda_compute_capabilities\': \'8.0\', \'cuda_cc_space_sep\': \'8.0\', \'cuda_cc_space_sep_no_period\': \'80\', \'cuda_cc_semicolon_sep\': \'8.0\', \'cuda_cc_cmake\': \'80\', \'cuda_int_comma_sep\': \'80\', \'cuda_int_space_sep\': \'80\', \'cuda_int_semicolon_sep\': \'80\', \'cuda_sm_comma_sep\': \'sm_80\', \'cuda_sm_space_sep\': \'sm_80\', \'module_name\': \'STREAM/5.10-GCC-11.3.0\', \'parallel\': \'8\', \'builddir\': \'/dev/shm/branfosj/build-up-EL8/STREAM/5.10/GCC-11.3.0\', \'installdir\': \'/rds/projects/2017/branfosj-rse/easybuild/EL8-ice/software/STREAM/5.10-GCC-11.3.0\', \'start_dir\': \'/dev/shm/branfosj/build-up-EL8/STREAM/5.10/GCC-11.3.0/\'}. This might cause failures or unexpected behavior, check for correct escaping if this is intended!'

The resolving of the pattern template variables fixes

ERROR: Installation of STREAM-5.10-GCC-11.3.0.eb failed: "No match for /rds/projects/b/branfosj-bear-apps/sources/s/STREAM/stream-5.10.c in [('stream-%(version)s.c', '$CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=10000000 -DNTIMES=1000 -o stream_1Kx10M; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=1000 -o stream_1Kx100M; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=1000000000 -DNTIMES=1000 -o stream_1Kx1B; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=2500000000 -DNTIMES=1000 -o stream_1Kx2.5B; $CC $CFLAGS %(source)s -mcmodel=large -DSTREAM_ARRAY_SIZE=5000000000 -DNTIMES=1000 -o stream_1Kx5B; ')], don't know which command to use."

branfosj · 2025-09-10T11:35:36Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS STREAM-5.10-GCC-11.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/490c0b794ff9eb0fff8e84c68db4b67f for a full test report.

branfosj · 2025-09-10T12:40:29Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS Cython-0.27.3-GCCcore-12.3.0-Python-2.7.18.eb
SUCCESS hypothesis-4.57.1-GCCcore-12.3.0-Python-2.7.18.eb
SUCCESS pytest-4.6.11-GCCcore-12.3.0-Python-2.7.18.eb
SUCCESS pybind11-2.9.2-GCCcore-12.3.0-Python-2.7.18.eb
SUCCESS numpy-1.16.6-foss-2023a-Python-2.7.18.eb
SUCCESS SciPy-bundle-2024.06-foss-2023a-Python-2.7.18.eb
SUCCESS matplotlib-2.2.5-foss-2023a-Python-2.7.18.eb
SUCCESS fastStructure-1.0-foss-2023a-Python-2.7.18.eb

Build succeeded for 8 out of 8 (1 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/0c2fd1fb0a4bdf8ae9ffc61a22b0fa57 for a full test report.

branfosj · 2025-09-10T13:10:46Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS CTPL-0.0.2-GCC-11.3.0.eb
SUCCESS DBG2OLC-20200724-GCC-11.3.0.eb
SUCCESS FastTree-2.1.11-GCCcore-10.3.0.eb
SUCCESS FastTree-2.1.11-GCCcore-11.2.0.eb
SUCCESS FastTree-2.1.11-GCCcore-11.3.0.eb
SUCCESS FastTree-2.1.11-GCCcore-12.2.0.eb
SUCCESS FastTree-2.1.11-GCCcore-12.3.0.eb
SUCCESS FastTree-2.1.11-GCCcore-13.2.0.eb
SUCCESS sbt-1.3.13-Java-8.eb
FAIL (build issue) fgbio-2.2.1-Java-8.eb (partial log available at https://gist.github.com/branfosj/c79df1b470e954453a83592fd3692668)
SUCCESS LDC-1.24.0-x86_64.eb
SUCCESS LDC-1.39.0-GCCcore-13.2.0.eb
SUCCESS dub-1.38.1-GCCcore-13.2.0.eb

Build succeeded for 12 out of 13 (10 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/1e817213c657cf52a0c0e870899135e6 for a full test report.

branfosj · 2025-09-10T13:28:53Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS GLM-AED-3.3.0a5-gompi-2021b.eb
FAIL (build issue) Lep-MAP3-20250312-Java-21.eb (partial log available at https://gist.github.com/branfosj/e9361cc52adc625cabbb75adea4af326)
SUCCESS lpsolve-5.5.2.11-GCC-10.3.0.eb
SUCCESS lpsolve-5.5.2.11-GCC-11.2.0.eb
SUCCESS lpsolve-5.5.2.11-GCC-11.3.0.eb
SUCCESS lpsolve-5.5.2.11-GCC-12.2.0.eb
SUCCESS lpsolve-5.5.2.11-GCC-12.3.0.eb
SUCCESS lpsolve-5.5.2.11-GCC-13.2.0.eb
SUCCESS lpsolve-5.5.2.11-GCC-13.3.0.eb
SUCCESS minimap2-2.22-GCCcore-11.2.0.eb
SUCCESS HTSlib-1.14-GCC-11.2.0.eb
SUCCESS SAMtools-1.14-GCC-11.2.0.eb
SUCCESS Racon-1.5.0-GCCcore-11.2.0.eb
SUCCESS Iris-1.0.5-GCC-11.2.0-Java-15.eb
SUCCESS Jasmine-1.1.4-GCC-11.2.0-Java-15.eb

Build succeeded for 14 out of 15 (11 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/23cd6a3e2a276950508a6372f0cb3a68 for a full test report.

branfosj · 2025-09-10T13:39:14Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS APOST3D-20240527-intel-compilers-2023.1.0.eb
SUCCESS ARAGORN-1.2.41-foss-2021b.eb
SUCCESS Austin-3.2.0.eb
SUCCESS btllib-1.7.0-GCC-12.3.0.eb
SUCCESS Boost-1.55.0-GCC-11.2.0.eb
SUCCESS Boost-1.55.0-GCC-12.3.0.eb
SUCCESS BGEN-enkre-1.1.7-GCC-11.2.0.eb
SUCCESS BGEN-enkre-1.1.7-GCC-12.3.0.eb

Build succeeded for 8 out of 8 (7 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/bba1a7082b6d898e4c603e7986d37aeb for a full test report.

branfosj · 2025-09-10T13:41:28Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS M1QN3-3.3-GCC-10.3.0.eb
SUCCESS mcqd-1.0.0-GCC-13.2.0.eb
SUCCESS Nextflow-20.10.0.eb
SUCCESS Nextflow-21.03.0.eb
SUCCESS Nextflow-22.04.0.eb
SUCCESS Nextflow-22.10.0.eb
SUCCESS Ninja-1.10.2-GCCcore-10.3.0.eb
SUCCESS Ninja-1.10.2-GCCcore-11.2.0.eb
SUCCESS Ninja-1.10.2-GCCcore-11.3.0.eb
SUCCESS Ninja-1.11.1-GCCcore-12.2.0.eb
SUCCESS Ninja-1.11.1-GCCcore-12.3.0.eb
SUCCESS Ninja-1.11.1-GCCcore-13.2.0.eb
SUCCESS Ninja-1.12.1-GCCcore-13.3.0.eb
SUCCESS Ninja-1.12.1-GCCcore-14.2.0.eb
SUCCESS Ninja-1.13.0-GCCcore-14.3.0.eb

Build succeeded for 15 out of 15 (15 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/517998a4056a26e75a57d93420a3422d for a full test report.

branfosj · 2025-09-10T13:54:43Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS PGPLOT-5.2.2-GCCcore-11.2.0.eb
SUCCESS PGPLOT-5.2.2-GCCcore-11.3.0.eb
SUCCESS PGPLOT-5.2.2-GCCcore-13.2.0.eb
SUCCESS PGPLOT-5.2.2-GCCcore-13.3.0.eb
SUCCESS Pindel-0.2.5b9-20170508-GCC-11.2.0.eb
SUCCESS QuPath-0.5.0-GCCcore-12.3.0-Java-17.eb
SUCCESS HTSlib-1.15.1-GCC-11.3.0.eb
SUCCESS Pindel-0.2.5b9-20170508-GCC-11.3.0.eb

Build succeeded for 8 out of 8 (7 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/de8543f6de02781f7f6c584b06f7e212 for a full test report.

boegel

lgtm

branfosj · 2025-09-10T14:16:57Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS AMICA-2024.1.19-intel-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/0eb366ed1f2478102a40ac5702ec299e for a full test report.

branfosj · 2025-09-10T14:39:04Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS TALYS-1.95-GCCcore-10.3.0.eb
SUCCESS tbl2asn-20220427-linux64.eb
SUCCESS tbl2asn-20230713-GCCcore-12.3.0.eb
SUCCESS tbl2asn-20230713-GCCcore-13.2.0.eb
SUCCESS tbl2asn-20230713-GCCcore-13.3.0.eb
SUCCESS tbl2asn-20230713-linux64.eb
SUCCESS TM-align-20190822-GCC-11.3.0.eb
SUCCESS wgsim-20111017-GCC-11.2.0.eb
SUCCESS ISL-0.26-GCCcore-13.3.0.eb
SUCCESS GDB-14.2-GCCcore-13.3.0.eb
SUCCESS tmpi-1.0.0-gompi-2024a.eb

Build succeeded for 11 out of 11 (9 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/42bfe37f70ac31cd5f777373c2beb90a for a full test report.

branfosj · 2025-09-10T14:41:16Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

**FAIL (unhandled exception: 'source')Traceback (most recent call last):
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/main.py", line 161, in build_and_install_software
(ec_res['success'], app_log, err_msg, err_code) = build_and_install_one(ec, init_env)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 5074, in build_and_install_one
result = app.run_all_steps(run_test_cases=run_test_cases)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 4887, in run_all_steps
self.run_step(step_name, step_methods)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 4728, in run_step
current_method()
File "/dev/shm/branfosj/tmp-up-EL8/included-easyblocks-dwgr0ajl/easybuild/easyblocks/generic/cmdcp.py", line 72, in build_step
regex_cmd = regex_cmd % self.cfg.template_values
KeyError: 'source'
** STREAM-5.10-GCC-11.3.0.eb
**FAIL (unhandled exception: 'source')Traceback (most recent call last):
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/main.py", line 161, in build_and_install_software
(ec_res['success'], app_log, err_msg, err_code) = build_and_install_one(ec, init_env)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 5074, in build_and_install_one
result = app.run_all_steps(run_test_cases=run_test_cases)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 4887, in run_all_steps
self.run_step(step_name, step_methods)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 4728, in run_step
current_method()
File "/dev/shm/branfosj/tmp-up-EL8/included-easyblocks-dwgr0ajl/easybuild/easyblocks/generic/cmdcp.py", line 72, in build_step
regex_cmd = regex_cmd % self.cfg.template_values
KeyError: 'source'
** STREAM-5.10-intel-compilers-2022.2.1.eb
**FAIL (unhandled exception: 'source')Traceback (most recent call last):
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/main.py", line 161, in build_and_install_software
(ec_res['success'], app_log, err_msg, err_code) = build_and_install_one(ec, init_env)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 5074, in build_and_install_one
result = app.run_all_steps(run_test_cases=run_test_cases)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 4887, in run_all_steps
self.run_step(step_name, step_methods)
File "/rds/projects/2017/branfosj-rse/easybuild/src/easybuild-framework/easybuild/framework/easyblock.py", line 4728, in run_step
current_method()
File "/dev/shm/branfosj/tmp-up-EL8/included-easyblocks-dwgr0ajl/easybuild/easyblocks/generic/cmdcp.py", line 72, in build_step
regex_cmd = regex_cmd % self.cfg.template_values
KeyError: 'source'
** STREAM-5.10-intel-compilers-2024.2.0.eb

Build succeeded for 0 out of 3 (3 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/406c8fd3cd05f959df7f8af8fe005af7 for a full test report.

branfosj · 2025-09-10T15:03:53Z

Test report by @branfosj

Overview of tested easyconfigs (in order)

SUCCESS RevBayes-1.1.1-GCC-11.2.0.eb
SUCCESS RevBayes-1.2.1-gompi-2022a.eb
SUCCESS RTG-Tools-3.12.1-Java-11.eb
SUCCESS Short-Pair-20170125-foss-2021b.eb
FAIL (build issue) SISSO-3.0.2-iimpi-2021b.eb (partial log available at https://gist.github.com/branfosj/b73a9398bd462c56636566a9f42f1d0d)
SUCCESS SISSO-3.1-20220324-iimpi-2021b.eb
SUCCESS SNAPE-pooled-20150707-GCC-11.3.0.eb

Build succeeded for 6 out of 7 (7 easyconfigs in total)
bear-pg0105u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8
See https://gist.github.com/branfosj/4f396c593a357753678c409f5f571a69 for a full test report.

branfosj · 2025-09-10T15:04:00Z

Failures from this PR:

STREAM - fixed in use %%(source)s in STREAM easyconfigs easybuild-easyconfigs#23841

Failures not related to this PR:

fgbio-2.2.1-Java-8.eb - unresolved dependency: com.github.samtools#htsjdk;2.24.1-26-ga38c78d-SNAPSHOT: not found / Unable to find credentials for [Artifactory Realm @ broadinstitute.jfrog.io].
Lep-MAP3-20250312-Java-21.eb - Couldn't find file lep-map3-code-e5dd07c7a6236f303ca6482bb9afbce158536f43.zip anywhere
SISSO-3.0.2-iimpi-2021b.eb - Couldn't find file SISSO.3.0.2.tar anywhere

Flamefire · 2025-09-10T15:06:35Z

easybuild/easyblocks/generic/cmdcp.py

+            for pattern, regex_cmd in self.cfg.get_ref('cmds_map'):
+                pattern = pattern % self.cfg.template_values
+                regex_cmd = regex_cmd % self.cfg.template_values


Isn't that equivalent? The getter recursively resolves the templates so how is this different than before?

Quick print shows that those are equivalent, so reverting this in #3921

Can you check that PR?

…2843_new_pr_cmdcp" This reverts commit 7429645, reversing changes made to 59fa0db.

resolve templates in CmdCp

5fbb0e3

branfosj added the bug fix label Sep 10, 2025

branfosj added this to the next release (5.1.2) milestone Sep 10, 2025

also resolve regex_cmd

0f8cb4a

boegel approved these changes Sep 10, 2025

View reviewed changes

boegel merged commit 7429645 into easybuilders:develop Sep 10, 2025
17 checks passed

boegel mentioned this pull request Sep 10, 2025

use of %(installdir)s template results in "Failed to resolve all templates" easybuilders/easybuild-framework#4990

Closed

branfosj deleted the 20250910122843_new_pr_cmdcp branch September 10, 2025 14:17

branfosj mentioned this pull request Sep 10, 2025

use %%(source)s in STREAM easyconfigs easybuilders/easybuild-easyconfigs#23841

Merged

Flamefire reviewed Sep 10, 2025

View reviewed changes

Flamefire added a commit to Flamefire/easybuild-easyblocks that referenced this pull request Sep 11, 2025

Revert "Merge pull request easybuilders#3918 from branfosj/2025091012…

c74fd77

…2843_new_pr_cmdcp" This reverts commit 7429645, reversing changes made to 59fa0db.

Flamefire mentioned this pull request Sep 11, 2025

Revert resolve templates in CmdCp #3918 #3921

Open

boegel changed the title ~~resolve templates in CmdCp~~ resolve templates in CmdCp easyblock Sep 24, 2025

Conversation

branfosj commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Overview of tested easyconfigs (in order)

Uh oh!

branfosj commented Sep 10, 2025

Uh oh!

Flamefire Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Flamefire Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

branfosj commented Sep 10, 2025 •

edited

Loading