Enable use of custom Slurm builds (#163)

* remove drain and resume functionality

* allow install and runtime taskbooks to be used directly

* fix linter complaints

* fix slurmctld state

* move common tasks to pre.yml

* remove unused openhpc_slurm_service

* fix ini_file use for some community.general versions

* fix var precedence in molecule test13

* fix var precedence in all molecule tests

* fix slurmd always starting on control node

* move install to install-ohpc.yml

* remove unused ohpc_slurm_services var

* add install-generic for binary-only install

* distinguish between system and user slurm binaries for generic install

* remove support for CentOS7 / OpenHPC

* remove post-configure, not needed as of slurm v20.02

* add openmpi/IMB-MPI1 by default for generic install

* allow removal of slurm.conf options

* update README

* enable openhpc_extra_repos for both generic and ohpc installs

* README tweak

* add openhpc_config_files parameter

* change library_dir to lib_dir

* fix perms

* fix/silence linter warnings

* remove packages only required for hpctests

* document openhpc_config_files restart behaviour

* bugfix missing newline in slurm.conf

* make path for slurm.conf configurable

* make slurm.conf template src configurable

* symlink slurm user tools so monitoring works

* fix slurm directories

* fix slurmdbd path for non-default slurm.conf paths

* default gres.conf to correct directory

* document <absent> for openhpc_config

* minor merge diff fixes

* Fix EPEL not getting installed

* build RL9.3 container images with systemd

* allow use on image containing slurm binaries

* prepend slurm binaries to PATH instead of symlinking

* ensure cgroup.conf is always next to slurm.conf and allow overriding template

* Add group.node_params to partitions/groups. (#182) (#185)

* Add group.node_params to partitions/groups. (#182)

Allows Features, etc., to be added to partitions.

* update SelectType from legacy to current default (#167)

---------

Co-authored-by: Kurt Bendl <[email protected]>

* update readme

* fixup mode parameters

* tidy slurmd restart line

---------

Co-authored-by: Kurt Bendl <[email protected]>

Author: sjpb

README.md

@@ -2,35 +2,35 @@
 
 # stackhpc.openhpc
 
-This Ansible role installs packages and performs configuration to provide an OpenHPC v2.x Slurm cluster.
+This Ansible role installs packages and performs configuration to provide a Slurm cluster. By default this uses packages from [OpenHPC](https://openhpc.community/) but it can also use user-provided Slurm binaries.
 
 As a role it must be used from a playbook, for which a simple example is given below. This approach means it is totally modular with no assumptions about available networks or any cluster features except for some hostname conventions. Any desired cluster fileystem or other required functionality may be freely integrated using additional Ansible roles or other approaches.
 
 The minimal image for nodes is a RockyLinux 8 GenericCloud image.
 
+## Task files
+This role provides four task files which can be selected by using the `tasks_from` parameter of Ansible's `import_role` or `include_role` modules:
+- `main.yml`: Runs `install-ohpc.yml` and `runtime.yml`. Default if no `tasks_from` parameter is used.
+- `install-ohpc.yml`: Installs repos and packages for OpenHPC.
+- `install-generic.yml`: Installs systemd units etc. for user-provided binaries.
+- `runtime.yml`: Slurm/service configuration.
+
 ## Role Variables
 
+Variables only relevant for `install-ohpc.yml` or `install-generic.yml` task files are marked as such below.
+
 `openhpc_extra_repos`: Optional list. Extra Yum repository definitions to configure, following the format of the Ansible
-[yum_repository](https://docs.ansible.com/ansible/2.9/modules/yum_repository_module.html) module. Respected keys for
-each list element:
-* `name`: Required
-* `description`: Optional
-* `file`: Required
-* `baseurl`: Optional
-* `metalink`: Optional
-* `mirrorlist`: Optional
-* `gpgcheck`: Optional
-* `gpgkey`: Optional
-
-`openhpc_slurm_service_enabled`: boolean, whether to enable the appropriate slurm service (slurmd/slurmctld).
+[yum_repository](https://docs.ansible.com/ansible/2.9/modules/yum_repository_module.html) module.
+
+`openhpc_slurm_service_enabled`: Optional boolean, whether to enable the appropriate slurm service (slurmd/slurmctld). Default `true`.
 
 `openhpc_slurm_service_started`: Optional boolean. Whether to start slurm services. If set to false, all services will be stopped. Defaults to `openhpc_slurm_service_enabled`.
 
 `openhpc_slurm_control_host`: Required string. Ansible inventory hostname (and short hostname) of the controller e.g. `"{{ groups['cluster_control'] | first }}"`.
 
 `openhpc_slurm_control_host_address`: Optional string. IP address or name to use for the `openhpc_slurm_control_host`, e.g. to use a different interface than is resolved from `openhpc_slurm_control_host`.
 
-`openhpc_packages`: additional OpenHPC packages to install.
+`openhpc_packages`: Optional list. Additional OpenHPC packages to install (`install-ohpc.yml` only).
 
 `openhpc_enable`:
 * `control`: whether to enable control host
@@ -44,7 +44,15 @@ each list element:
 
 `openhpc_login_only_nodes`: Optional. If using "configless" mode specify the name of an ansible group containing nodes which are login-only nodes (i.e. not also control nodes), if required. These nodes will run `slurmd` to contact the control node for config.
 
-`openhpc_module_system_install`: Optional, default true. Whether or not to install an environment module system. If true, lmod will be installed. If false, You can either supply your own module system or go without one.
+`openhpc_module_system_install`: Optional, default true. Whether or not to install an environment module system. If true, lmod will be installed. If false, You can either supply your own module system or go without one (`install-ohpc.yml` only).
+
+`openhpc_generic_packages`: Optional. List of system packages to install, see `defaults/main.yml` for details (`install-generic.yml` only).
+
+`openhpc_sbin_dir`: Optional. Path to slurm daemon binaries such as `slurmctld`, default `/usr/sbin` (`install-generic.yml` only).
+
+`openhpc_bin_dir`: Optional. Path to Slurm user binaries such as `sinfo`, default `/usr/bin` (`install-generic.yml` only).
+
+`openhpc_lib_dir`: Optional. Path to Slurm libraries, default `/usr/lib64/slurm` (`install-generic.yml` only).
 
 ### slurm.conf
 
@@ -122,6 +130,16 @@ that this is *not the same* as the Ansible `omit` [special variable](https://doc
 
 `openhpc_state_save_location`: Optional. Absolute path for Slurm controller state (`slurm.conf` parameter [StateSaveLocation](https://slurm.schedmd.com/slurm.conf.html#OPT_StateSaveLocation))
 
+`openhpc_slurmd_spool_dir`: Optional. Absolute path for slurmd state (`slurm.conf` parameter [SlurmdSpoolDir](https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdSpoolDir))
+
+`openhpc_slurm_conf_template`: Optional. Path of Jinja template for `slurm.conf` configuration file. Default is `slurm.conf.j2` template in role. **NB:** The required templating is complex, if just setting specific parameters use `openhpc_config` intead.
+
+`openhpc_slurm_conf_path`: Optional. Path to template `slurm.conf` configuration file to. Default `/etc/slurm/slurm.conf`
+
+`openhpc_gres_template`: Optional. Path of Jinja template for `gres.conf` configuration file. Default is `gres.conf.j2` template in role.
+
+`openhpc_cgroup_template`: Optional. Path of Jinja template for `cgroup.conf` configuration file. Default is `cgroup.conf.j2` template in role.
+
 #### Accounting
 
 By default, no accounting storage is configured. OpenHPC v1.x and un-updated OpenHPC v2.0 clusters support file-based accounting storage which can be selected by setting the role variable `openhpc_slurm_accounting_storage_type` to `accounting_storage/filetxt`<sup id="accounting_storage">[1](#slurm_ver_footnote)</sup>. Accounting for OpenHPC v2.1 and updated OpenHPC v2.0 clusters requires the Slurm database daemon, `slurmdbd` (although job completion may be a limited alternative, see [below](#Job-accounting). To enable accounting:

defaults/main.yml

@@ -49,8 +49,12 @@ openhpc_cgroup_default_config:
 openhpc_config: {}
 openhpc_cgroup_config: {}
 openhpc_gres_template: gres.conf.j2
+openhpc_cgroup_template: cgroup.conf.j2
 
 openhpc_state_save_location: /var/spool/slurm
+openhpc_slurmd_spool_dir: /var/spool/slurm
+openhpc_slurm_conf_path: /etc/slurm/slurm.conf
+openhpc_slurm_conf_template: slurm.conf.j2
 
 # Accounting
 openhpc_slurm_accounting_storage_host: "{{ openhpc_slurmdbd_host }}"
@@ -80,6 +84,15 @@ openhpc_enable:
   database: false
   runtime: false
 
+# Only used for install-generic.yml:
+openhpc_generic_packages:
+  - munge
+  - mariadb-connector-c # only required on slurmdbd
+  - hwloc-libs # only required on slurmd
+openhpc_sbin_dir: /usr/sbin # path to slurm daemon binaries (e.g. slurmctld)
+openhpc_bin_dir: /usr/bin  # path to slurm user binaries (e.g sinfo)
+openhpc_lib_dir: /usr/lib64/slurm # path to slurm libraries
+
 # Repository configuration
 openhpc_extra_repos: []
 
@@ -127,12 +140,9 @@ ohpc_default_extra_repos:
       gpgcheck: true
       gpgkey: "https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8"
 
-# Concatenate all repo definitions here
-ohpc_repos: "{{ ohpc_openhpc_repos[ansible_distribution_major_version] + ohpc_default_extra_repos[ansible_distribution_major_version] + openhpc_extra_repos }}"
-
 openhpc_munge_key_b64:
 openhpc_login_only_nodes: ''
-openhpc_module_system_install: true
+openhpc_module_system_install: true # only works for install-ohpc.yml/main.yml
 
 # Auto detection
 openhpc_ram_multiplier: 0.95

tasks/install-generic.yml

@@ -0,0 +1,72 @@
+- include_tasks: pre.yml
+
+- name: Create a list of slurm daemons
+  set_fact:
+    _ohpc_daemons: "{{ _ohpc_daemon_map | dict2items | selectattr('value') | items2dict | list }}"
+  vars:
+    _ohpc_daemon_map:
+      slurmctld: "{{ openhpc_enable.control }}"
+      slurmd: "{{ openhpc_enable.batch }}"
+      slurmdbd: "{{ openhpc_enable.database }}"
+
+- name: Ensure extra repos
+  ansible.builtin.yum_repository: "{{ item }}" # noqa: args[module]
+  loop: "{{ openhpc_extra_repos }}"
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Install system packages
+  dnf:
+    name: "{{ openhpc_generic_packages }}"
+
+- name: Create Slurm user
+  user:
+    name: slurm
+    comment: SLURM resource manager
+    home: /etc/slurm
+    shell: /sbin/nologin
+
+- name: Create Slurm unit files
+  template:
+    src: "{{ item }}.service.j2"
+    dest: /lib/systemd/system/{{ item }}.service
+    owner: root
+    group: root
+    mode: ug=rw,o=r
+  loop: "{{ _ohpc_daemons }}"
+  register: _slurm_systemd_units
+
+- name: Get current library locations
+  shell:
+    cmd: "ldconfig -v | grep -v ^$'\t'" # noqa: no-tabs risky-shell-pipe
+  register: _slurm_ldconfig
+  changed_when: false
+
+- name: Add library locations to ldd search path
+  copy:
+    dest: /etc/ld.so.conf.d/slurm.conf
+    content: "{{ openhpc_lib_dir }}"
+    owner: root
+    group: root
+    mode: ug=rw,o=r
+  when: openhpc_lib_dir not in _ldd_paths
+  vars:
+    _ldd_paths: "{{ _slurm_ldconfig.stdout_lines | map('split', ':') | map('first') }}"
+
+- name: Reload Slurm unit files
+  # Can't do just this from systemd module
+  command: systemctl daemon-reload # noqa: command-instead-of-module no-changed-when no-handler
+  when: _slurm_systemd_units.changed
+
+- name: Prepend $PATH with slurm user binary location
+  lineinfile:
+    path: /etc/environment
+    line: "{{ new_path }}"
+    regexp: "^{{ new_path | regex_escape }}"
+    owner: root
+    group: root
+    mode: u=gw,go=r
+  vars:
+    new_path: PATH="{{ openhpc_bin_dir }}:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
+
+- meta: reset_connection # to get new environment

tasks/install.yml renamed to tasks/install-ohpc.yml

@@ -3,16 +3,14 @@
 - include_tasks: pre.yml
 
 - name: Ensure OpenHPC repos
-  ansible.builtin.yum_repository:
-    name: "{{ item.name }}"
-    description: "{{ item.description | default(omit) }}"
-    file: "{{ item.file }}"
-    baseurl: "{{ item.baseurl | default(omit) }}"
-    metalink: "{{ item.metalink | default(omit) }}"
-    mirrorlist: "{{ item.mirrorlist | default(omit) }}"
-    gpgcheck: "{{ item.gpgcheck | default(omit) }}"
-    gpgkey: "{{ item.gpgkey | default(omit) }}"
-  loop: "{{ ohpc_repos }}"
+  ansible.builtin.yum_repository: "{{ item }}" # noqa: args[module]
+  loop: "{{ ohpc_openhpc_repos[ansible_distribution_major_version] }}"
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Ensure extra repos
+  ansible.builtin.yum_repository: "{{ item }}" # noqa: args[module]
+  loop: "{{ ohpc_default_extra_repos[ansible_distribution_major_version] + openhpc_extra_repos }}"
   loop_control:
     label: "{{ item.name }}"
 

tasks/main.yml

@@ -8,7 +8,7 @@
 
 - name: Install packages
   block:
-    - include_tasks: install.yml
+    - include_tasks: install-ohpc.yml
   when: openhpc_enable.runtime | default(false) | bool
   tags: install
 

tasks/runtime.yml

@@ -11,12 +11,19 @@
 
 - name: Ensure Slurm directories exists
   file:
-    path: "{{ openhpc_state_save_location }}"
+    path: "{{ item.path }}"
     owner: slurm
     group: slurm
-    mode: 0755
+    mode: '0755'
     state: directory
-  when: inventory_hostname == openhpc_slurm_control_host
+  loop:
+    - path: "{{ openhpc_state_save_location }}" # StateSaveLocation
+      enable: control
+    - path: "{{ openhpc_slurm_conf_path | dirname }}"
+      enable: control
+    - path: "{{ openhpc_slurmd_spool_dir }}" # SlurmdSpoolDir
+      enable: batch
+  when: "openhpc_enable[item.enable] | default(false) | bool"
 
 - name: Retrieve Munge key from control host
   # package install generates a node-unique one
@@ -32,7 +39,7 @@
     dest: "/etc/munge/munge.key"
     owner: munge
     group: munge
-    mode: 0400
+    mode: '0400'
   register: _openhpc_munge_key_copy
 
 - name: Ensure JobComp logfile exists
@@ -41,15 +48,15 @@
     state: touch
     owner: slurm
     group: slurm
-    mode: 0644
+    mode: '0644'
     access_time: preserve
     modification_time: preserve
   when: openhpc_slurm_job_comp_type == 'jobcomp/filetxt'
 
 - name: Template slurmdbd.conf
   template:
     src: slurmdbd.conf.j2
-    dest: /etc/slurm/slurmdbd.conf
+    dest: "{{ openhpc_slurm_conf_path | dirname }}/slurmdbd.conf"
     mode: "0600"
     owner: slurm
     group: slurm
@@ -58,11 +65,11 @@
 
 - name: Template slurm.conf
   template:
-    src: slurm.conf.j2
-    dest: /etc/slurm/slurm.conf
+    src: "{{ openhpc_slurm_conf_template }}"
+    dest: "{{ openhpc_slurm_conf_path }}"
     owner: root
     group: root
-    mode: 0644
+    mode: '0644'
   when: openhpc_enable.control | default(false)
   notify:
     - Restart slurmctld service
@@ -72,7 +79,7 @@
 - name: Create gres.conf
   template:
     src: "{{ openhpc_gres_template }}"
-    dest: /etc/slurm/gres.conf
+    dest: "{{ openhpc_slurm_conf_path | dirname }}/gres.conf"
     mode: "0600"
     owner: slurm
     group: slurm
@@ -85,8 +92,8 @@
 - name: Template cgroup.conf
   # appears to be required even with NO cgroup plugins: https://slurm.schedmd.com/cgroups.html#cgroup_design
   template:
-    src: cgroup.conf.j2
-    dest: /etc/slurm/cgroup.conf
+    src: "{{ openhpc_cgroup_template }}"
+    dest: "{{ openhpc_slurm_conf_path | dirname }}/cgroup.conf"
     mode: "0644" # perms/ownership based off src from ohpc package
     owner: root
     group: root
@@ -96,15 +103,6 @@
   register: ohpc_cgroup_conf
   # NB uses restart rather than reload as this is needed in some cases
 
-- name: Remove local tempfile for slurm.conf templating
-  ansible.builtin.file:
-    path: "{{ _slurm_conf_tmpfile.path }}"
-    state: absent
-  when: _slurm_conf_tmpfile.path is defined
-  delegate_to: localhost
-  changed_when: false # so molecule doesn't fail
-  become: no
-
 - name: Ensure Munge service is running
   service:
     name: munge
@@ -129,7 +127,9 @@
   changed_when: true
   when:
     - openhpc_slurm_control_host in ansible_play_hosts
-    - hostvars[openhpc_slurm_control_host].ohpc_slurm_conf.changed or hostvars[openhpc_slurm_control_host].ohpc_cgroup_conf.changed or hostvars[openhpc_slurm_control_host].ohpc_gres_conf.changed # noqa no-handler
+    - hostvars[openhpc_slurm_control_host].ohpc_slurm_conf.changed or
+      hostvars[openhpc_slurm_control_host].ohpc_cgroup_conf.changed or
+      hostvars[openhpc_slurm_control_host].ohpc_gres_conf.changed # noqa no-handler
   notify:
     - Restart slurmd service
 
@@ -143,7 +143,7 @@
     create: yes
     owner: root
     group: root
-    mode: 0644
+    mode: '0644'
   when:
     - openhpc_enable.batch | default(false)
   notify:

templates/slurm.conf.j2

@@ -2,7 +2,7 @@ ClusterName={{ openhpc_cluster_name }}
 
 # PARAMETERS
 {% for k, v in openhpc_default_config | combine(openhpc_config) | items %}
-{%  if v != "omit" %}{# allow removing items using setting key: null #}
+{%  if v != "omit" %}{# allow removing items using setting key: omit #}
 {%    if k != 'SlurmctldParameters' %}{# handled separately due to configless mode #}
 {{ k }}={{ v | join(',') if (v is sequence and v is not string) else v }}
 {%    endif %}

templates/slurmctld.service.j2

@@ -0,0 +1,22 @@
+[Unit]
+Description=Slurm controller daemon
+After=network-online.target munge.service
+Wants=network-online.target
+ConditionPathExists={{ openhpc_slurm_conf_path }}
+
+[Service]
+Type=simple
+EnvironmentFile=-/etc/sysconfig/slurmctld
+EnvironmentFile=-/etc/default/slurmctld
+ExecStart={{ openhpc_sbin_dir }}/slurmctld -D -s -f {{ openhpc_slurm_conf_path }} $SLURMCTLD_OPTIONS
+ExecReload=/bin/kill -HUP $MAINPID
+LimitNOFILE=65536
+TasksMax=infinity
+
+# Uncomment the following lines to disable logging through journald.
+# NOTE: It may be preferable to set these through an override file instead.
+#StandardOutput=null
+#StandardError=null
+
+[Install]
+WantedBy=multi-user.target