Parallel processing slowing down with time on a large dataset

[Karltlbm]

Hello lidR,

Not sure if it is a lidR or future topic, or even something else because I am not an expert in parallel computing.

I am using the catalog_apply function along with the future library for parallel processing and my data originates from drone acquisitions which makes them very heavy in terms of size, which is why I have set a relatively small tile size.

The main catalog engine parameters are:
resolution = 1.0
n_cores = 20
size_tile = 100
size_buffer = 25

At the beginning of the process, I can clearly see around 20 tiles and cores active, which is perfect.

Then, it seems that the processing speed slows down as the processing advances.

For example, almost at the end of the tile processing:

We can see that only 7 tiles are being processed.

Also, my cores do not seem to work at their full potential, looking at the htop screenshot, right ?

Have you already experienced this behavior ?

Thanks in advance :)

[Jean-Romain]

Show me your code

[Karltlbm]

This code aims to obtain a raster of the main classification labels.
I use rm() and gc() to optimize ram usage as the dataset is very heavy.

resolution = 1.0
n_cores = 20
size_tile = 100
size_buffer = 25

ctg <- readLAScatalog(path_data)
crs_scan <- crs(ctg)

opt_chunk_size(ctg)   <- size_tile
opt_chunk_buffer(ctg) <- size_buffer
opt_output_files(ctg) <- file.path(path_output_local, "chunk_{ID}_initial_classif")
set_lidr_threads(1)
opt_progress(ctg) <- T

future::plan(future::multisession, workers = n_cores)

majority_class <- function(z) {
      ux <- unique(z)
      ux[which.max(tabulate(match(z, ux)))]
}
            
catalog_apply(ctg, function(las_chunk) {
      r <- grid_metrics(las_chunk, ~majority_class(Classification), res = resolution)
      rm(las_chunk)
      gc()
      return(r)
})

future::plan(future::sequential)

[Jean-Romain]

It looks good. Honestly I don't know.

[Karltlbm]

Do you think that changing the set_lidr_threads() could have an impact ?
I will try with different sets of n_cores and set_lidr_threads() values and report if it solves the problem.

[Jean-Romain]

No. Yhis I what I wanted to verify in your code. You have no control on set_lidr_threads(). To avoided nested parallelism it is set to 1 anyway. Or at least it is supposed to be.

[danielrode]

I am experiencing this issue as well. I wonder if there is a problem with newer future releases.

[bengwill]

I was having this same issue and I reverted back to a previous version of R, lidR, and future with all their contemporary dependencies (from Dec 2022 as that was last when I knew the code worked) and my issue seemed resolved.

I cant confirm which package had the issue specifically but it seems an update somewhere has broken compatibility with the future package.

My problem was I would use multisession for 10 workers, and I would get 10 processes spin up in task manager, but only two of them would get any load and the rest would sit at 0%.

Try using an older package set and see if that helps. Maybe a more recent version would work as I was more conservative in my approach.

[Jean-Romain]

@bengwill thank you for your feed back. The code in lidR did not change for years. So I'm inclined to think there is an issue with future. Could you try with current lidR + old future?