Parallel chains - worker nodes can't find tmbstan library when non-standard .libPaths are in use #30
Description
Hi kaskr,
Thanks for the great tmbstan package!
I've run into an issue running chains on multiple cores whereby the worker nodes can't find the tmbstan library (or any of it's dependencies).
This happens when a user doesn't keep their R libraries in the standard location (e.g., .../R/R-4.4.2/library), but instead uses a custom library folder setup via .libPaths().
In this case, when each worker node Rscript.exe tries to execute the library(tmbstan) command from the tmpfile that you create at line 75 in R/tmbstan.R, it will fail to find the library and the worker Rscript.exe will exit. This will cause parallel::makeCluster() to hang when it tries to create the cluster.
In order to fix this, the code written to tmpfile needs to take the current .libPaths() into account.
Suggest changing R/tmbstan.R, line 76 from:
cat("library(tmbstan)\n", file=tmpfile)
to:
cat(".libPaths(c(", toString(dQuote(.libPaths(), q = FALSE)), "))\n", file = tmpfile)
cat("library(tmbstan)\n", file = tmpfile, append = TRUE)
With this change in place, when I run the following test:
library(TMB)
library(tmbstan)
runExample("simple")
cores <- 4
options(mc.cores = cores)
init.fn <- function()
list(u=rnorm(114), beta=rnorm(2), logsdu=runif(1,0,10), logsd0=runif(1,0,1))
fit <- tmbstan(obj, chains=cores, open_progress=FALSE, init=init.fn)
The tmpfile will contain:
.libPaths(c( "C:/Users/bakerk/Documents/R/Rlibs", "C:/Users/bakerk/AppData/Local/Programs/R/R-4.4.2/library" ))
library(tmbstan)
dyn.load('C:/Users/bakerk/Documents/R/Rlibs/TMB/examples/x64/simple.dll')
and the worker Rscripts can now find the tmbstan library (and all its dependencies).
Thanks!
Dave