Skip to content

Commit 1ad0099

Browse files
richelbilderbeekrichelbilderbeek
committed
Add results
2 parents c35d95e + 5dd73fa commit 1ad0099

3 files changed

Lines changed: 61 additions & 47 deletions

File tree

‎docs/day3/big_data.rst‎

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,6 @@ High-Performance Data Analytics (HPDA)
5151
Why we need to take special actions
5252
...................................
5353

54-
Remember this one?
5554

5655
.. image:: ../img/when-to-use-pandas.png
5756
:width: 600 px
@@ -105,7 +104,7 @@ Allocating RAM
105104

106105
.. important::
107106

108-
- You do not have to explicitly run threads or other parallelism.
107+
- You do not have to explicitly run threads or other parallelism (see
109108
- Allocating several nodes for one one big memory problem is not useful. (Unless you are "chunking")
110109

111110
- Note that shared memory among the cores works within node only.
@@ -130,6 +129,19 @@ Use the Slurm options for either "BATCH", "INTERACTIVE" from command line or fro
130129
- You ask here for a non-default partition.
131130
- How to do this, search your cluster documentation, see exercise below.
132131

132+
Example for Tetralith with one core but 50 GB RAM for one hour
133+
134+
.. code-block:: console
135+
136+
interactive -A naiss2026-4-66 --mem 50G -t 1:0:0
137+
138+
.. code-block: bash
139+
140+
#!/bin/bash
141+
#SBATCH -A naiss2026-4-66
142+
#SBATCH --mem 50G
143+
#SBATCH -t 1:0:0
144+
133145
.. note::
134146

135147
- "core-hours" drawn from your project may be set to the maximum of "number of cores" and "memory part of node" requested.
@@ -138,17 +150,19 @@ Use the Slurm options for either "BATCH", "INTERACTIVE" from command line or fro
138150
Exercise: Memory allocation (10 min)
139151
------------------------------------
140152

141-
.. admonition:: Break-out rooms per Cluster or Cluster Type (OnDemand vs terminal)
153+
1a. Log in to a Desktop (ThinLinc or OnDemand) (see :ref:`common-login`)
154+
155+
.. admonition:: Links and addresses
142156

143-
1. Log in to a Desktop (ThinLinc or OnDemand) (see :ref:`common-login`)
157+
- Tetralith (ThinLinc client: ``tetralith.nsc.liu.se``)
158+
- Dardel (ThinLinc client: ``dardel-vnc.pdc.kth.se``)
159+
- Alvis (https://alvis.c3se.chalmers.se/)
160+
- Bianca (https://bianca.uppmax.uu.se/)
161+
- Pelle (https://pelle-gui.uppmax.uu.se/)
162+
- Cosmos (ThinLinc client: ``cosmos-dt.lunarc.lu.se``)
163+
- Kebnekaise (https://portal.hpc2n.umu.se/public/landing_page.html)
144164

145-
- Tetralith (ThinLinc client: ``tetralith.nsc.liu.se``)
146-
- Dardel (ThinLinc client: ``dardel-vnc.pdc.kth.se``)
147-
- Alvis (<https://alvis.c3se.chalmers.se/>)
148-
- Bianca (<https://bianca.uppmax.uu.se/>)
149-
- Pelle (<https://pelle-gui.uppmax.uu.se/>)
150-
- Cosmos (ThinLinc client: ``cosmos-dt.lunarc.lu.se``)
151-
- Kebnekaise (<https://portal.hpc2n.umu.se/public/landing_page.html>)
165+
1b. If You cannot use a desktop it is all fine with a command line: ``ssh ...``
152166

153167
.. discussion::
154168

@@ -212,25 +226,25 @@ Exercise: Memory allocation (10 min)
212226

213227
- Multiply the RAM per core with number of allocated cores..
214228

215-
- Example: 8 GB per core
229+
- Example: ~8 GB per core
216230
- ~40 GB
217231

218232
.. challenge:: Do you remember how to allocate several cores?
219233
:class: dropdown
220234

221-
- Slurm flag ``-n <number of cores>``
235+
- Slurm option ``-n <number of cores>``
222236

223237
.. challenge:: Actually start an interactive session with 4 cores for 3 hours.
224238

225239
- We will use it for the exercises later.
226240
- Since it may take some time to get the allocation we do it now already!
227241
- Follow the best procedure for your cluster, e.g. from **command-line** or **OnDemand**.
242+
- **Continue with the other exercises below, while session is waiting to be started.**
228243

229244
.. admonition:: How?
230245
:class: dropdown
231246

232-
233-
The following Slurm options needs to be set
247+
- Start with the command ``salloc`` or ``interactive`` (depending on system) followed by the Slurm options:
234248

235249
- ``-t 3:0:0``
236250
- ``-n 4``

‎docs/day4/ml.rst‎

Lines changed: 32 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -67,20 +67,20 @@ In this course we will look at examples for these, and show how you run them at
6767
The loading are slightly different at the clusters
6868
- C3SE:
6969
- For TensorFlow: ``module load TensorFlow/2.15.1-foss-2023a-CUDA-12.1.1``
70-
- For Pytorch: ``PyTorch-bundle/2.1.2-foss-2023a-CUDA-12.1.1``
70+
- For Pytorch: ``PyTorch-bundle/2.1.2-foss-2023a-CUDA-12.1.1`` and ``PyTorch/2.7.1-foss-2024a-CUDA-12.6.0``
7171
- Datasets and models: `Many <https://www.c3se.chalmers.se/documentation/software/machine_learning/datasets/>`_
7272
- UPPMAX:
73-
- For TensorFlow: install yourself
73+
- For TensorFlow: ``TensorFlow/2.18.1-foss-2024a```
7474
- For Pytorch: ``PyTorch/2.6.0-foss-2024a``
7575
- Bianca has all tools in : ``python_ML_packages/3.9.5``
7676
- HPC2N:
77-
- For TensorFlow ``ml GCC/12.3.0 OpenMPI/4.1.5 TensorFlow/2.15.1-CUDA-12.1.1 scikit-learn/1.4.2 Tkinter/3.11.3 matplotlib/3.7.2``
78-
- For the Pytorch: ``ml GCC/12.3.0 OpenMPI/4.1.5 PyTorch/2.1.2-CUDA-12.1.1 scikit-learn/1.4.2 Tkinter/3.11.3 matplotlib/3.7.2``
77+
- For TensorFlow ``module load GCC/13.3.0 OpenMPI/5.0.3 TensorFlow/2.18.1-CUDA-12.6.0``
78+
- For the Pytorch: ``module load GCC/13.3.0 OpenMPI/5.0.3 PyTorch/2.7.1-CUDA-12.6.0``
7979
- LUNARC:
8080
- For TensorFlow ``module load GCC/11.3.0 Python/3.10.4 SciPy-bundle/2022.05 TensorFlow/2.11.0-CUDA-11.7.0 scikit-learn/1.1.2``
8181
- For Pytorch ``module load GCC/11.3.0 Python/3.10.4 SciPy-bundle/2022.05 PyTorch/1.12.1-CUDA-11.7.0 scikit-learn/1.1.2``
8282
- NSC:
83-
- For Tetralith, use virtual environment. Pytorch and TensorFlow might coming soon to the cluster!
83+
- For Tetralith, use virtual environment.
8484
- PDC:
8585
- For both TensorFlow and Pytorch : ``module load PDC singularity/4.2.0-cpeGNU-24.11``
8686

@@ -115,7 +115,7 @@ The list is not exhaustive, but lists the more popular ML/DL libraries. I encour
115115

116116
* - Tool
117117
- Pelle (Python 3.12.3)
118-
- Kebnekaise (Python 3.11.3/3.11.5)
118+
- Kebnekaise (Python 3.12.3)
119119
- Cosmos (Python 3.11.3/3.11.5)
120120
- Tetralith (Python 3.11.3/3.11.5)
121121
- Dardel (Python 3.11.7)
@@ -143,7 +143,7 @@ The list is not exhaustive, but lists the more popular ML/DL libraries. I encour
143143
- scikit-learn
144144
* - TensorFlow
145145
- N.A.
146-
- TensorFlow
146+
- GCC OpenMPI TensorFlow
147147
- TensorFlow
148148
- N.A.
149149
- PDC singularity/4.1.1-cpeGNU-23.12
@@ -153,34 +153,34 @@ The list is not exhaustive, but lists the more popular ML/DL libraries. I encour
153153
- Keras (up to Python 3.8.6), TensorFlow (Python 3.11.3)
154154
- TensorFlow
155155
- N.A.
156-
- PDC singularity/4.1.1-cpeGNU-23.12
156+
- PDC singularity
157157
- TensorFlow
158158
* - PyTorch (torch)
159159
- PyTorch
160-
- PyTorch
160+
- GCC OpenMPI PyTorch
161161
- PyTorch
162162
- N.A.
163-
- PDC singularity/4.1.1-cpeGNU-23.12
163+
- PDC singularity
164164
- PyTorch
165165
* - Pandas
166166
- SciPy-bundle
167167
- SciPy-bundle
168168
- SciPy-bundle
169-
- SciPy-bundle
169+
- N.A.
170170
- cray-python
171171
- SciPy-bundle
172172
* - Matplotlib
173173
- matplotlib
174174
- matplotlib
175175
- matplotlib
176176
- buildtool-easybuild GCC matplotlib
177-
- PDC/23.12 matplotlib/3.8.2-cpeGNU-23.12
177+
- PDC matplotlib
178178
- matplotlib
179179
* - Beautiful Soup (beautifulsoup4)
180180
- BeautifulSoup
181181
- BeautifulSoup
182182
- BeautifulSoup
183-
- BeautifulSoup
183+
- buildtool-easybuild GCC BeautifulSoup
184184
- N.A.
185185
- BeautifulSoup
186186
* - Seaborn
@@ -766,10 +766,10 @@ We now learn by submitting a batch job which consists of loading python module,
766766
# Load any modules you need, here Python 3.13.5.
767767
load Python/3.13.5-GCCcore-14.3.0
768768
769-
source ../my_env/bin/activate
769+
source my_env/bin/activate
770770
771771
# Run your Python script
772-
python fashion_mnist.py
772+
python [pytorch/tf]_mnist.py
773773
774774
.. tab:: HPC2N
775775

@@ -785,10 +785,10 @@ We now learn by submitting a batch job which consists of loading python module,
785785
# Load any modules you need, here for Python/3.11.3
786786
module load GCC/12.3.0 Python/3.11.3
787787
788-
source ../my_env/bin/activate
788+
source my_env/bin/activate
789789
790790
# Run your Python script
791-
python fashion_mnist.py
791+
python [pytorch/tf]_mnist.py
792792
793793
794794
.. tab:: LUNARC
@@ -807,10 +807,10 @@ We now learn by submitting a batch job which consists of loading python module,
807807
# Load any modules you need, here for Python/3.11.5 and compatible SciPy-bundle
808808
module load GCC/13.2.0 Python/3.11.5
809809
810-
source ../my_env/bin/activate
810+
source my_env/bin/activate
811811
812812
# Run your Python script
813-
python fashion_mnist.py
813+
python [pytorch/tf]_mnist.py
814814
815815
816816
.. tab:: NSC
@@ -827,9 +827,9 @@ We now learn by submitting a batch job which consists of loading python module,
827827
ml load buildtool-easybuild/4.8.0-hpce082752a2 GCCcore/13.2.0
828828
ml load Python/3.11.5
829829
830-
source ../my_env/bin/activate
830+
source my_env/bin/activate
831831
832-
python fashion_mnist.py
832+
python [pytorch/tf]_mnist.py
833833
834834
.. tab:: PDC
835835

@@ -847,9 +847,9 @@ We now learn by submitting a batch job which consists of loading python module,
847847
module load cray-python/3.11.5
848848
module load craype-accel-amd-gfx90a
849849
850-
source ../my_env/bin/activate
850+
source my_env/bin/activate
851851
852-
python fashion_mnist.py
852+
python [pytorch/tf]_mnist.py
853853
854854
.. tab:: C3SE
855855

@@ -863,9 +863,9 @@ We now learn by submitting a batch job which consists of loading python module,
863863
864864
module load Python/3.11.3-GCCcore-12.3.0
865865
866-
source ../my_env/bin/activate
866+
source my_env/bin/activate
867867
868-
python fashion_mnist.py
868+
python [pytorch/tf]_mnist.py
869869
870870
871871
Tips and Tricks (Lessons Learned):
@@ -991,7 +991,7 @@ Miscellaneous examples
991991
module purge > /dev/null 2>&1
992992
module load Python/3.13.5-GCCcore-14.3.0
993993
994-
source ../my_env/bin/activate
994+
source my_env/bin/activate
995995
# Output to file - not needed if your job creates output in a file directly
996996
# In this example I also copy the output somewhere else and then run another executable (or you could just run the same executable for different parameters).
997997
python tf_program.py 1 2 > myoutput1 2>&1
@@ -1018,7 +1018,7 @@ Miscellaneous examples
10181018
ml load buildtool-easybuild/4.8.0-hpce082752a2 GCCcore/13.2.0
10191019
ml load Python/3.11.5
10201020
1021-
source ../my_env/bin/activate
1021+
source my_env/bin/activate
10221022
# Output to file - not needed if your job creates output in a file directly
10231023
# In this example I also copy the output somewhere else and then run another executable (or you could just run the same executable for different parameters).
10241024
python tf_program.py 1 2 > myoutput1 2>&1
@@ -1046,7 +1046,7 @@ Miscellaneous examples
10461046
# Load any modules you need, here for Python/3.11.5 and compatible SciPy-bundle
10471047
module load GCC/13.2.0 Python/3.11.5
10481048
1049-
source ../my_env/bin/activate
1049+
source my_env/bin/activate
10501050
10511051
# Output to file - not needed if your job creates output in a file directly
10521052
# In this example I also copy the output somewhere else and then run another executable (or you could just run the same executable for different parameters).
@@ -1075,7 +1075,7 @@ Miscellaneous examples
10751075
# Load any modules you need, here for Python/3.11.5 and compatible SciPy-bundle
10761076
module load cray-python/3.11.7
10771077
1078-
source ../my_env/bin/activate
1078+
source my_env/bin/activate
10791079
10801080
# Output to file - not needed if your job creates output in a file directly
10811081
# In this example I also copy the output somewhere else and then run another executable (or you could just run the same executable for different parameters).
@@ -1106,7 +1106,7 @@ Miscellaneous examples
11061106
module load Python/3.13.5-GCCcore-14.3.0
11071107
module load TensorFlow/2.18.0-CUDA-11.8-1
11081108
1109-
#optionally: source ../my_env/bin/activate
1109+
#optionally: source my_env/bin/activate
11101110
# Output to file - not needed if your job creates output in a file directly
11111111
# In this example I also copy the output somewhere else and then run another executable (or you could just run the same executable for different parameters).
11121112
python tf_program.py 1 2 > myoutput1 2>&1
@@ -1202,7 +1202,7 @@ Miscellaneous examples
12021202
module purge > /dev/null 2>&1
12031203
module load Python/3.13.5-GCCcore-14.3.0
12041204
1205-
source ../my_env/bin/activate
1205+
source my_env/bin/activate
12061206
12071207
# Run your Python script
12081208
python example-tf.py
@@ -1222,7 +1222,7 @@ Miscellaneous examples
12221222
ml load buildtool-easybuild/4.8.0-hpce082752a2 GCCcore/13.2.0
12231223
ml load Python/3.11.5
12241224
1225-
source ../my_env/bin/activate
1225+
source my_env/bin/activate
12261226
12271227
# Run your Python script
12281228
python example-tf.py

‎evaluations/20260423_day_3/results.ods‎

-47.4 KB
Binary file not shown.

0 commit comments

Comments
 (0)