Memory usage scaling

[Mikolaj-A-Kowalski]

We need to collect data on how the memory usage scales with:

• grid size
• number of particles (?)

The priority is to focus on the memory usage on the GPU, since the CPU is usually compute bound (or so I have been told ;-))

Tasks

• Figure out a way to track resident memory of the OceanBioME calculation on the GPU
  • Using Nsight
  • Using CUDA.jl memory pool. We need to check if it is feasible
• Construct memory usage scaling plots
  • Against the grid size
  • Against the number of particles

[Mikolaj-A-Kowalski]

It turned out that I was again a bit blind and missed some features in the CUDA.jl API.

To enquire about the resident memory we can use:

• used_memory() function to get the number of bytes currently used from the memory pool
• cached_memory() function to get the current size of the memory pool.

In addition I have missed that there is a CUDA.@timed macro which extends the Base.@timed macro with extra information about GPU allocations!

Hence what we need to do is to extend the run_scaling script to use CUDA.@timed to include extra information about the GPU memory allocations in the output.

EDIT: Caveat to the above is that CUDA.@timed requires CUDA to be initialised which requires a GPU to be present. Hence we need to switch between macros for GPU and CPU runs.

[Mikolaj-A-Kowalski]

Here is a plot of the GPU memory usage versus the grid size. It shows:

• range between minimum and maximum memory usage recorded (blue region - is thin hence looks like a line)
• Total number of bytes allocated on GPU throughout the calculation (red line - traces the top of the blue region)
• Two linear least-squares fit to estimate GPU memory usage as a function of grid cells number (with and without intercept)

The memory usage was recorded at each 'progress update' (i.e. every 20 iterations). However fact that the total allocations match resident memory shows that there are no allocations of any temporary variables taking place on the GPU during the run.

@jagoosw Should we try to compare these numbers versus 'pen and paper' estimate of the memory usage (akin to CliMA/Oceananigans.jl#3557)? Does the ~600 bytes per grid cell look sensible?

Raw numbers for reference: scaling_run_gpu_mem.csv

EDIT: It was not noted before sadly :-( The above results are for runs with IO disabled.

[jagoosw]

600 bytes per cell sounds reasonable since thats ~37.5 fields and we have 3 fields (state, tendency $G^n$, and previous tendency $G^-$), so thats ~12 fields which adds up since we have 7 BGC + 3 velocity + 2 buoyancy (T and S). There is also at least 1 (probably 2) pressure fields so we actually end up at (7+3+2)*3 + 2 = 38 fields ~600Bytes

[Mikolaj-A-Kowalski]

Just to note. I run some cases with IO today and it seems that the memory usage pattern may be different in that case!
The simulations were eating much more memory and it seems to grow a bit as the calculation progresses.

I had some issues with the calculations dying mid run with mysterious 'exception' (without the details as it was thrown while on a GPU). I suspect it may be running out of memory. I have been trying to snoop a bit running in an interactive job and peeking at nvidia-smi, but this only allows me to see how big memory pool julia has (i.e. its memory usage as reported), but it does not tell me how full the pool is.

Basically I will have to re-run the memory scaling study with IO and see if any differences emerge.

[johnryantaylor]

Interesting, thanks! I look forward to seeing the results of the scaling test with IO

[Mikolaj-A-Kowalski]

After checking it seems that I have hallucinated the memory usage problems. The scaling run with IO yielded exactly the same results. As shown in the plot below, the values effectively overlap

Raw values: scaling_run_gpu_withio.csv; scaling_run_gpu_withoutio.csv

I still see the problem that the 512x512x256 LOBSTER script crashes mid calculation, but if i make it report memory usage in the 'progress update' callback there is nothing unusual (uses ~ 30 GiB as expected).

Last week the run that made me suspicious I made with -i flag and checked the pool status after the crash, which gave me a value just below 80 GIB, but I was not able to reproduce that so I am not sure what went on there :-/

[MarionBWeinzierl]

@AdelekeBankole Can you add the same for PISCES?

[AdelekeBankole]

Memory scaling for PISCES, we have similar memory usage results to LOBSTER.