@@ -31,7 +31,7 @@ Installing ``conda-pack`` will let you use the ``conda pack`` command which can
31
31
.. code-block :: bash
32
32
33
33
# Pack environment located at an explicit path into my_env.tar.gz
34
- conda pack -p /explicit/path/to/my_env
34
+ conda pack --format tar.gz --n-threads -1 --prefix /explicit/path/to/my_env --output ./my_env.tar.gz
35
35
36
36
After packing your environment, it can then be moved to the NVMe using ``sbcast `` when in a compute job.
37
37
Packing your environment will also put a ``conda-unpack `` script into the same ``.tar.gz `` archive.
@@ -70,7 +70,8 @@ Next, let's pack our new conda environment:
70
70
.. code-block :: bash
71
71
72
72
cd $MEMBERWORK /< PROJECT_ID>
73
- conda pack -p $MEMBERWORK /< PROJECT_ID> /torch_env
73
+ # use all cores available for packing
74
+ conda pack --format tar.gz --n-threads -1 --prefix $MEMBERWORK /< PROJECT_ID> /torch_env --output ./torch_env.tar.gz
74
75
75
76
Finally, let's run a compute job:
76
77
@@ -118,7 +119,7 @@ Below is an example batch script that uses ``sbcast``, unpacks our environment,
118
119
# Untar the environment file (only need 1 task per node to do this)
119
120
srun -N8 --ntasks-per-node 1 mkdir /mnt/bb/${USER} /torch_env
120
121
echo " untaring torchenv"
121
- srun -N8 --ntasks-per-node 1 tar -xzf /mnt/bb/${USER} /torch_env.tar.gz -C /mnt/bb/${USER} /torch_env
122
+ srun -N8 --ntasks-per-node 1 -c56 tar --use-compress-program=pigz -xf /mnt/bb/${USER} /torch_env.tar.gz -C /mnt/bb/${USER} /torch_env
122
123
123
124
# Unpack the env
124
125
source activate /mnt/bb/${USER} /torch_env
@@ -183,16 +184,18 @@ Here are the timings from the ``sbcast`` **NVMe** run:
183
184
184
185
.. code-block ::
185
186
186
- JobID Start Elapsed
187
- --------------- ---------------- --------------------
188
- jobid . 00:01:13
189
- jobid.batch . 00:01:13
190
- jobid.extern . 00:01:13
191
- jobid.0 . 00:00:01 mkdir
192
- jobid.1 . 00:00:49 untar
193
- jobid.2 . 00:00:00 unpack
187
+ JobID Start Elapsed
188
+ --------------- ---------------- ---------------------
189
+ jobid . 00:01:17
190
+ jobid.batch . 00:01:17
191
+ jobid.extern . 00:01:17
192
+ jobid.0 . 00:00:00 mkdir
193
+ jobid.1 . 00:00:39 untar
194
+ jobid.2 . 00:00:03 conda unpack
194
195
jobid.3 . 00:00:02 example.py
195
196
197
+
198
+
196
199
Here are the timings if the environment was never broadcast from **Orion **:
197
200
198
201
.. code-block ::
0 commit comments