Skip to content

Commit 392530e

Browse files
authored
updating tar and conda pack examples to use parallelism (#981)
1 parent 48c82a1 commit 392530e

File tree

1 file changed

+14
-11
lines changed

1 file changed

+14
-11
lines changed

software/python/sbcast_conda.rst

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Installing ``conda-pack`` will let you use the ``conda pack`` command which can
3131
.. code-block:: bash
3232
3333
# Pack environment located at an explicit path into my_env.tar.gz
34-
conda pack -p /explicit/path/to/my_env
34+
conda pack --format tar.gz --n-threads -1 --prefix /explicit/path/to/my_env --output ./my_env.tar.gz
3535
3636
After packing your environment, it can then be moved to the NVMe using ``sbcast`` when in a compute job.
3737
Packing your environment will also put a ``conda-unpack`` script into the same ``.tar.gz`` archive.
@@ -70,7 +70,8 @@ Next, let's pack our new conda environment:
7070
.. code-block:: bash
7171
7272
cd $MEMBERWORK/<PROJECT_ID>
73-
conda pack -p $MEMBERWORK/<PROJECT_ID>/torch_env
73+
# use all cores available for packing
74+
conda pack --format tar.gz --n-threads -1 --prefix $MEMBERWORK/<PROJECT_ID>/torch_env --output ./torch_env.tar.gz
7475
7576
Finally, let's run a compute job:
7677

@@ -118,7 +119,7 @@ Below is an example batch script that uses ``sbcast``, unpacks our environment,
118119
# Untar the environment file (only need 1 task per node to do this)
119120
srun -N8 --ntasks-per-node 1 mkdir /mnt/bb/${USER}/torch_env
120121
echo "untaring torchenv"
121-
srun -N8 --ntasks-per-node 1 tar -xzf /mnt/bb/${USER}/torch_env.tar.gz -C /mnt/bb/${USER}/torch_env
122+
srun -N8 --ntasks-per-node 1 -c56 tar --use-compress-program=pigz -xf /mnt/bb/${USER}/torch_env.tar.gz -C /mnt/bb/${USER}/torch_env
122123
123124
# Unpack the env
124125
source activate /mnt/bb/${USER}/torch_env
@@ -183,16 +184,18 @@ Here are the timings from the ``sbcast`` **NVMe** run:
183184

184185
.. code-block::
185186
186-
JobID Start Elapsed
187-
--------------- ---------------- --------------------
188-
jobid . 00:01:13
189-
jobid.batch . 00:01:13
190-
jobid.extern . 00:01:13
191-
jobid.0 . 00:00:01 mkdir
192-
jobid.1 . 00:00:49 untar
193-
jobid.2 . 00:00:00 unpack
187+
JobID Start Elapsed
188+
--------------- ---------------- ---------------------
189+
jobid . 00:01:17
190+
jobid.batch . 00:01:17
191+
jobid.extern . 00:01:17
192+
jobid.0 . 00:00:00 mkdir
193+
jobid.1 . 00:00:39 untar
194+
jobid.2 . 00:00:03 conda unpack
194195
jobid.3 . 00:00:02 example.py
195196
197+
198+
196199
Here are the timings if the environment was never broadcast from **Orion**:
197200

198201
.. code-block::

0 commit comments

Comments
 (0)