Running an existing model on HPC (e.g. Slurm) - EDS-NLP

[giscus[bot]]

Running an existing model on HPC (e.g. Slurm) - EDS-NLP

https://edsnlp-slurm-integration.vercel.app/slurm-integration/tutorials/hpc/

[lucas-sedran]

[File of time-stamped logs]

If you want to put your logs in time-stamped files with the id of the SLURM job, you can change these lines:

#SBATCH --output=slurm-%j-stdout.log
#SBATCH --error=slurm-%j-stderr.log

set -euo pipefail

To :

#SBATCH --output=/dev/null # Direct the logs into a "black hole" file
#SBATCH --error=/dev/null

log_time=$(date '+%Y-%m-%d_%H-%M-%S')
base_log_dir=/<my_path>/logs
log_dir="${base_log_dir}/${log_time}-Job_${SLURM_JOB_ID}"
mkdir -p "$log_dir"

# Redirect stdout & stderr to this folder
exec > "$log_dir/slurm-${SLURM_JOB_ID}-stdout.log" 2> "$log_dir/slurm-${SLURM_JOB_ID}-stderr.log"

set -euo pipefail

[lucas-sedran]

[GPU performance monitoring file]

If you want to monitor the performance of your GPU and save it in a .csv file, you can write these lines just before running the python code:

nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,memory.total \
           --format=csv -l 5 > "gpu_usage_${SLURM_JOB_ID}.csv" &
MONITOR_PID=$!
printf "\n\nGPU monitoring launched (PID=$MONITOR_PID)\n\n"

# And then :
cd path/to/your/project
python inference.py

Here we retrieve timestamp, utilization.gpu [%], memory.used and memory.total every 5s.
If you have a log folder, you can put it in : "${log_dir}/gpu_usage_${SLURM_JOB_ID}.csv"

[lucas-sedran]

And don't forget to kill GPU monitoring once you've finished:

python inference.py

kill $MONITOR_PID
printf "\nGPU monitoring stopped.\n"

[0dey]

[data/scratch/]

If you have not requested access to your data/scratch/ folder to AP-HP research support team, you should remove "/data/scratch/$USER:/data/scratch/$USER" from the line --container-mounts