use_isolated_environments.md rearrangement

bclaremar · web-flow · commit f9ed222b1b6e · 2026-04-21T23:31:49.000+02:00
diff --git a/docs/day2/use_isolated_environments.md b/docs/day2/use_isolated_environments.md
@@ -112,7 +112,7 @@ HPC cluster| Conda vs venv
 Alvis      | venv, conda in container        
 Bianca     | conda/latest, venv via wharf    
 COSMOS     | Anaconda3/2024.02-1             
-Dardel     | miniconda3/24.7.1-0-cpeGNU-23.12
+Dardel     | miniconda3/25.3.1-1-cpeGNU-24.11
 Kebnekaise | venv **only**                   
 LUMI       | venv,  conda-containerize       
 Pelle      | venv, Miniforge3/24.11.3-0      
@@ -136,7 +136,6 @@ LUMI       | conda-containerize
 - [documentation](https://pip.pypa.io/en/stable/cli/pip_list)
 :::
 
-
 ### Virtual environment - venv & virtualenv
 
 With this tool you can download and install with ``pip`` from the [PyPI repository](https://pypi.org/)
@@ -193,12 +192,9 @@ With this tool you can download and install with ``pip`` from the [PyPI reposito
 
    - You can use "pip list" on the command line (after loading the python module) to see which packages are available and which versions.
    - Some packages may be inherited from the modules you have loaded
-   - You can do ``pip list --local`` to see what is installed by you in the environment.
-   - Some IDE:s like Spyder may only find those "local" packages
    - To save space, you should load any other Python modules you will need that are system installed before installing your own packages! Remember to choose ones that are compatible with the Python version you picked!
        -   ``--system-site-packages`` includes the packages already installed in the loaded python module.
    - The ``--no-cache-dir"`` option is required to **avoid it from reusing earlier installations from the same user in a different environment**.
-   - The ``--no-build-isolation`` is to make sure that it uses the loaded modules from the module system when **building any Cython libraries**.
 :::
 
 ### Conda
@@ -225,81 +221,62 @@ With this tool you can download and install with ``pip`` from the [PyPI reposito
    - scilifelab-lts
 
     You reach them all by loading the conda module. You don't have to state the specific channel when using UPPMAX. Otherwise you do with ``conda -c <channel> ...``
-
+:::
 
 :::{warning}
 
-   Drawbacks
+Drawbacks
 
-   - Conda cannot use already install packages from the Python modules and libraries already installed, and hence installs them anyway
-   - Conda is therefore known for creating **many** *small* files. Your disk space is not only limited in GB, but also in number of files (typically ``300000`` in $HOME).
-   - Check your disk usage and quota limit
-       - Do a ``conda clean -a`` once in a while to remove unused and unnecessary files
+- Conda cannot use already install packages from the Python modules and libraries already installed, and hence installs them anyway
+- Conda is therefore known for creating **many** *small* files. Your disk space is not only limited in GB, but also in number of files (typically ``300000`` in $HOME).
+- Check your disk usage and quota limit
+    - Do a ``conda clean -a`` once in a while to remove unused and unnecessary files
 :::
 
 :::{tip}
 
    - The conda environments including many small files are by default stored in ``~/.conda`` folder that is in your $HOME directory with limited storage.
-   - Move your ``.conda`` directory to your project folder and make a soft link to it from ``$HOME``
-   - Do the following (``mkdir -p`` ignores error output and will not recreate another folder if it already exists):
-        - (replace what is inside ``<>`` with relevant path)
-
-   - Solution 1
-
-      This works nicely if you have several projects. Then you can change these variables according to what you are currently working with.
-
-      ```bash
+   - Use the commands
 
-         export CONDA_ENVS_PATH="path/to/your/project/(subdir)"
-         export CONDA_PKG_DIRS="path/to/your/project/(subdir)"
-         mamba create --prefix=$CONDA_ENVS_PATH/<conda env name>
-      ```
-
-   - Solution 2
-
-      - This may not be a good idea if you have several projects.
-
-      ```bash
+   ```bash
+   export CONDA_ENVS_PATH="path/to/your/project/(subdir)"
+   export CONDA_PKG_DIRS="path/to/your/project/(subdir)"
+   mamba create --prefix=$CONDA_ENVS_PATH/<conda env name>
+   ```
 
-         $ mkdir -p ~/.conda
-         $ mv ~/.conda /<path-to-project-folder>/<username>/
-         $ ln -s /<path-to-project-folder>/<username>/.conda ~/.conda
-      ```
 :::
 
 :::{admonition} Example NSC
 
-   ```console
+```console
 
-      module load Miniforge/24.7.1-2-hpc1
-      export CONDA_PKG_DIRS=/proj/spring-courses-naiss/users/$USER
-      export CONDA_ENVS_PATH=/proj/spring-courses-naiss/users/$USER
-      mamba create --prefix=$CONDA_ENVS_PATH/numpy-proj-39 python=3.9.5 -c conda-forge
-      mamba activate nsc-example
-      # A prompt "(/path-to/nsc-example/)" should show up
-      # double-check we are using python from the Conda environment!
-      which python  # should point to the conda environment!
-      python -V     # should give python version 3.9.5
-      mamba install numpy
-   ```
+module load Miniforge/24.7.1-2-hpc1
+export CONDA_PKG_DIRS=/proj/spring-courses-naiss/users/$USER
+export CONDA_ENVS_PATH=/proj/spring-courses-naiss/users/$USER
+mamba create --prefix=$CONDA_ENVS_PATH/numpy-proj-39 python=3.9.5 -c conda-forge
+mamba activate nsc-example
+# A prompt "(/path-to/nsc-example/)" should show up
+# double-check we are using python from the Conda environment!
+which python  # should point to the conda environment!
+python -V     # should give python version 3.9.5
+conda install numpy
+```
 
-   ```python
+```python
 
-      >>> import numpy
-   ```
+>>> import numpy
+```
 :::
 
-:::{admonition} Comments
-   :class: dropdown
-
-   - When pinning with Conda, use single ``=`` instead of double (as used by pip)
+- Note, hen pinning with Conda, use single ``=`` instead of double (as used by pip)
 :::
 
 :::{admonition} Conda base env
    :class: dropdown
 
-   - When conda is loaded you will by default be in the base environment, which works in the same way as other conda environments.
-   - It includes a Python installation and some core system libraries and dependencies of Conda. It is a “best practice” to avoid installing additional packages into your base software environment.
+   - When conda is loaded you will by default be in the *base environment*, which works in the same way as other conda environments.
+   - It includes a Python installation and some core system libraries and dependencies of Conda.
+   - It is a “best practice” to avoid installing additional packages into your base software environment.
 
 :::{admonition} Conda cheat sheet
 
@@ -321,29 +298,28 @@ With this tool you can download and install with ``pip`` from the [PyPI reposito
    - [what-is-the-difference-with-conda-mamba-poetry-pip](https://pixi.sh/latest/misc/FAQ/#what-is-the-difference-with-conda-mamba-poetry-pip)
 :::
 
-:::{admonition} What to do when a problem arises?
-   :class: dropdown
-
-   - If you experience unexpected problems with the conda provided by the module system on Rackham or anaconda3 on Dardel, you can easily install your own and maintain it yourself.
-   - Read more at [Pavlin Mitev's page about conda on Rackham/Dardel](https://hackmd.io/@pmitev/conda_on_Rackham) and change paths to relevant one for your system.
-   - Or [Conda - "best practices" - UPPMAX](https://hackmd.io/@pmitev/module_conda_Rackham)
-:::
-
 ## Install from file
 
-- All centers has had different approaches in what is included in the module system and not.
-- Therefore the solution to complete the necessary packages needed for the course lessons, different approaches has to be made.
-- This is left as exercise for you, see Exercise 4 and 5.
+- This is handy when you want to move your Python environment somewhere else. 
+- Also, when giving your code to someone else (in research group or to a community).
+- Good for reproducibility
 
-### venv
+:::{admonition} Principle for both ``venv`` & ``conda``
+1. Activate the environment and make sure it works
+2. Save a environment file
+3. Distribute the file
+4. Receiver installs the environment from the file
+:::
 
+### venv
 
 Make a requirements file:
 
 ```console
 
-   pip freeze --local > requirements.txt
+pip freeze --local > requirements.txt
 ```
+
 :::{admonition} How does it look like?
    :class: dropdown
 
@@ -360,6 +336,7 @@ Make a requirements file:
       python-dateutil==2.9.0.post0
       six==1.17.0
    ```
+:::
 
 Install packages from a file
 
@@ -440,6 +417,10 @@ Create an environment from a file. Do this on another computer or rename.
 
 ## Exercises
 
+- All centers has had different approaches in what is included in the module system and not.
+- Therefore the solution to complete the necessary packages needed for the course lessons, different approaches has to be made.
+- This is left as exercise for you, see Exercise 4 and 5.
+
 :::{challenge} Exercise 0: Make a decision between ``venv`` or ``conda``.
 
    - We recommend `conda` for LUNARC.
@@ -1028,9 +1009,7 @@ Breakout room according to grouping
    ```
 :::
 
-Summary
--------
-
+## Summary
 
 :::{keypoints}
 
@@ -1045,6 +1024,11 @@ Summary
             - install in project folder due to many files.
 :::
 
+Summary of the workflows:
+
+- [Venv](../extra/extra_isolated.md#)
+- [Conda](../extra/extra_isolated.md#
+
 :::{admonition} Documentation at the centres
    :class: dropdown
 
@@ -1076,105 +1060,7 @@ Summary
    - https://docs.lumi-supercomputer.eu/software/installing/container-wrapper/#examples-of-using-the-lumi-container-wrapper
 :::
 
-## Summary
-
-### Workflow ``venv``
-
-1. Start from a Python version you would like to use (load the module):
-    - This step are different at different clusters since the naming is different
-
-2. Load the Python module you will be using, as well as any site-installed package modules (requires the ``--system-site-packages`` option later)
-    - ``module load <python module>``
-
-The next points will be the same for all clusters
-
-3. Create the isolated environment with something like ``python -m venv <name-of-environment>``
-    - use the ``--system-site-packages`` to include all "non-base" packages
-    - include the full path in the name if you want the environment to be stored other than in the "present working directory".
-
-4. Activate the environment with ``source <path to virtual environment>/bin activate``
-
-:::{note}
-
-   - ``source`` can most often be replaced by ``.``, like in ``. Example/bin/activate``. Note the important <space> after ``.``
-   - For clarity we use the ``source`` style here.
-:::
-
-5. Install (or update) the environment with the packages you need with the ``pip install`` command
-
-    - Note that ``--user`` must be omitted: else the package will be installed in the global user folder.
-    - The ``--no-cache-dir"`` option is required to avoid it from reusing earlier installations from the same user in a different environment. The ``--no-build-isolation`` is to make sure that it uses the loaded modules from the module system when building any Cython libraries.
 
-6. Work in the isolated environment
-   - When activated you can always continue to add packages!
-7. Deactivate the environment after use with ``deactivate``
-
-:::{note}
-
-   To save space, you should load any other Python modules you will need that are system installed before installing your own packages! Remember to choose ones that are compatible with the Python version you picked!
-         ``--system-site-packages`` includes the packages already installed in the loaded python module.
-
-   At HPC2N, NSC and LUNARC, you often have to load SciPy-bundle. This is how you on Tetralith (NSC) could create a venv (Example) with a SciPy-bundle included which is compatible with Python/3.11.5:
-
-   ```console
-
-       $ module load buildtool-easybuild/4.8.0-hpce082752a2 GCC/13.2.0 Python/3.11.5 SciPy-bundle/2023.11 # for NSC
-       $ python -m venv --system-site-packages Example
-   ```    
-:::
-
-:::{warning}
-
-   Draw-backs
-
-   - Only works for Python environments
-   - Only works with Python versions already installed
-:::
-
-Typical workflow Conda
-......................
-
-The first 2 steps are cluster dependent and will therefore be slightly different.
-
-1. Make conda available from a software module, like ``ml load conda`` or similar, or use own installation of miniconda or miniforge.
-2. First time
-
-   :::{admonition} First time
-      :class: dropdown
-
-      - The variables CONDA_ENVS_PATH and CONDA_PKG_DIRS contains the location of your environments. Set it to your project's environments folder, if you have one, instead of the $HOME folder.
-      - Otherwise, the default is ``~/.conda/envs``.
-      - Example:
-
-      ```console
-
-         $ export CONDA_ENVS_PATH="path/to/your/project/(subdir)"
-         $ export CONDA_PKG_DIRS="path/to/your/project/(subdir)"
-      ```
-      
-   :::
-
-Next steps are the same for all clusters
-
-3. Create the conda environment ``conda create -n <name-of-env>``
-4. Activate the conda environment by: ``source activate <conda-env-name>``
-
-    - You can define the packages to be installed here already.
-    - If you want another Python version, you have to define it here, like: ``conda ... python=3.6.8``
-
-5. Install the packages with ``conda install ...`` or ``pip install ...``
-6. Now do your work!
-
-    - When activated you can always continue to add packages!
-
-7. Deactivate
-
-:::{prompt}
-    :language: bash
-    :prompts: (python-36-env) $
-
-    conda deactivate
-:::
 
 :::{seealso}
 
