Parallelisation for speedier CPU inference?

[jpcompartir]

Should we add a dependency to furrr/future and provide functions for cutting datasets + processing in parallel, or should this be left to the user?

[samterfa]

I would love to figure out paralleliz(s)ation. We could have a use_parallel parameter, and if it's true, check that furrr or future is installed. We wouldn't need a dependency that way. I really like the furrr package; it's super simple to use.

[jpcompartir]

Nice, not sure if this is an anti-pattern if using reticulate, but it could be as simple as adding a cuts variable to data where cuts = availableCores() then group_split(cuts) then future_map_dfr()? calling the pipeline inside the future_map_dfr(). This would be taking care of the parallelisation totally in R - which may not be the best choice. I'll get a bit more acquainted with the functions so far in the package and test it out in the next couple of days.

[samterfa]

Actually R parallelism doesn't work with Reticulate objects so it's not a slick as it usually is in R. I've tried this for speed ups and you have to load the model in each process. https://cran.r-project.org/web/packages/future/vignettes/future-4-non-exportable-objects.html

[jpcompartir]

I see, rather than creating multiple sessions with multisession, is it possible with multicore? Will dip into the Natural Language Processing with Transformers book, for the section on accelerated inference with a CPU- see if it's better to do it directly in the model/pipe instantiation somehow...

This wasn't figured out in spacyr either: quanteda/spacyr#93

[jpcompartir]

Latest update on this is that we’re using torch to send models and pipelines to the GPU. Currently testing/benchmarking this to get a better gauge of speed up. Currently not quite as fast as expected - but thinking that when it comes to training (future versions of this package will look to utilise the trainer API and the datasets library) the speed up would be higher - calculating gradients etc.

Do also need to test with different batch sizes, pushing the GPU to its limit etc.