Can I use mirai on a slurm cluster with foreach

[statquant]

Hello,
I was wondering if mirai would integrate with foreach while working on a remote slurm cluster.
While the documentation gives an example for a local cluster I see nothing equivalent for a remote slurm cluster.
My goal is to move completely away from clustermq while still retaining the foreach api

[shikokuchuo]

Hi @statquant yes, Slurm is definitely supported, even using foreach.

If you were looking at the parallel vignette then it is very succinct, and you might have missed it:

A ‘miraiCluster’ is recognised as one of the official cluster types in R 4.5, and may be created by parallel::makeCluster(type = "MIRAI"). This function calls make_cluster(), which may also be used to create a ‘miraiCluster’ directly.

Specify ‘n’ to launch nodes on the local machine.
Specify ‘url’ for receiving connections from remote nodes.
Optionally, specify ‘remote’ to launch remote daemons using a remote configuration generated by remote_config() or ssh_config().

I guess I need to update this to mention cluster_config() explicitly as well.

See this section of the reference vignette on how you would use mirai's cluster launcher:
https://mirai.r-lib.org/articles/mirai.html#iii--hpc-cluster-resource-managers

Basically to create a cluster instead of the usual daemons, swap the daemons() call for a make_cluster() call with the same arguments.

[statquant]

Thanks you @shikokuchuo, I am not on R 4.5 yet but with your help I could make the following work

library(future.mirai)                                                       
library(doFuture)                                                           
library(mirai)                                                              
                                                                            
# config the cluster                                                        
slurm_config <- cluster_config(                                             
                               command = "sbatch",                          
                               options = "#SBATCH --job-name=mirai          
                               #SBATCH --output=job.out",                   
                               rscript = file.path(R.home("bin"), "Rscript")
)                                                                           
# start the cluster                                                         
daemons(n = 4,                                                              
        url = host_url(),                                                   
        remote = slurm_config                                               
        )                                                                   
#                                                                           
registerDoFuture()                                                          
# plan mirai_cluster                                                        
plan(mirai_cluster)                                                         
                                                                            
# NOW ITS WORKING                                                           
foreach(i = 1:4) %dopar% {                                                  
    Sys.info()                                                              
} -> res                                                                    
data.table::as.data.table(res)  

So with 4.5 I can hide the call to deamons which is much better, and I should be able to totally remove the need for future correct?

Something else: starting workers as the above has a drawback in my company where we manage slurm allocation by checking the compute time. It would be much better in this context to start the workers when the tasks are to be sent to the cluster and automatically close the cluster on completion. Do you see a good way of doing it in the context of using foreach ?

[statquant]

Note that the following code is failing for me

library(mirai)                                                              
                                                                            
# config the cluster                                                        
slurm_config <- cluster_config(                                             
                               command = "sbatch",                          
                               options = "#SBATCH --job-name=mirai          
                               #SBATCH --output=job.out",                   
                               rscript = file.path(R.home("bin"), "Rscript")
)                                                                           
# start the cluster                                                         
mirai::make_cluster(n = 4,                                                  
        url = host_url(),                                                   
        remote = slurm_config                                               
        ) -> cl                                                             
#                                                                           
library(doParallel)                                                         
registerDoParallel(cl)                                                      
                                                                            
# NOW ITS WORKING                                                           
foreach(i = 1:4) %dopar% {                                                  
    Sys.info()                                                              
} -> res                                                                    
                                                                            
mirai::stop_cluster(cl)     

Error in checkForRemoteErrors(val) :                                                                         
  2 nodes produced errors; first error: Error in as.environment(pos): using 'as.environment(NULL)' is defunct

[shikokuchuo]

The code looks right. Perhaps you can check that it works locally - it does for me.

It might be because you're trying to use it whilst not all daemons have connected yet. You can check the number of connections using status(cl).