Nested Mirai Maps

[DavZim]

I have a package A that internally uses mirai_map() in function inner_loop.

The problem is that this function might or might not be wrapped in another mirai_map() call by the user. That is, I need to structure my code that it works in three cases: 1) the user did not specify daemons, 2) the user called daemons, 3) the user used mirai_map() to call the function in parallel.

1 and 2 are working, but the nested mirai_maps calls I cannot get to work.
https://mirai.r-lib.org/articles/v01-map.html#nested-maps mentions that including a call to daemons() in the function provided to the outer map should work, but it does not work for me.

So besides a solution here, maybe it would be great to have a working sample in the vignettes (happy to write something up if I have a solution).

A simple MWE would be the following.

# a function inside my package
outer_loop <- function(x) {
  # setup the mirai cluster
  if (mirai::on_daemon()) {
    # call daemons to set deamons used by the inner loop
    # https://mirai.r-lib.org/articles/v01-map.html#nested-maps
    print("Connecting to existing Daemons!")
    # this is where the error lies
    mirai::daemons()
  } else if (mirai::status()$connections == 0) {
    # start daemons if not already running
    print("Starting new Daemons!")
    mirai::daemons(2)
    on.exit({
      print("Stopping Daemons!")
      mirai::daemons(0)
    }, add = TRUE)
  } else {
    print("Using Existing Daemons")
  }
  
  Sys.sleep(1)
  
  # inner loop: loop over the sequence of x
  r <- mirai::mirai_map(
    seq(x),
    function(xx) {
      Sys.sleep(0.1)
      data.frame(val = xx ^ 2, pid = Sys.getpid())
    }
  )
  vals <- r[mirai::.stop]
  
  list(vals = do.call(rbind, vals), pid = Sys.getpid())
}


##############################################################
# 1: Run the outer_loop function without mirai

# reset daemons as if the user did not do anything
mirai::daemons(0)
#> [1] 0

# should work as no daemons are running:
# works!
outer_loop(3)
#> [1] "Starting new Daemons!"
#> [1] "Stopping Daemons!"
#> $vals
#>   val     pid
#> 1   1 2402149
#> 2   4 2402153
#> 3   9 2402153
#> 
#> $pid
#> [1] 2387474


##############################################################
# 2: Run the outer_loop function **without** mirai::mirai_map but on existing daemons

# start daemons
mirai::daemons(5, output = TRUE)
#> [1] 5

# check the status of the daemons
mirai::status()
#> $connections
#> [1] 5
#> 
#> $daemons
#> [1] "abstract://fdbb82cad357e9c57c9434aa"
#> 
#> $mirai
#> awaiting executing completed 
#>        0         0         0 

# this should still work and should not mean that the daemons are used
# works!
outer_loop(3)
#> [1] "Using Existing Daemons"
#> $vals
#> val     pid
#>   1   1 2402409
#>   2   4 2402426
#>   3   9 2402413
#> 
#> $pid
#> [1] 2387474

mirai::daemons(0)
#> [1] 0


##############################################################
# 3: Run the outer_loop function with mirai::mirai_map
mirai::daemons(5)
#> [1] 5

# launch the outer_loop using mirai_map
rr <- mirai::mirai_map(
  1:3,
  outer_loop
)
#> [1][1] "Connecting to existing Daemons!"
#> "Connecting to existing Daemons!"
#> [1] "Connecting to existing Daemons!"
# Note I'll spare the joke about race conditions here :D

rr[mirai::.stop]
#> Error: In index 1:
#> No daemons set - use e.g. `mirai::daemons(6)` to set 6 local daemons.

# clean up
mirai::daemons(0)

Ideally, I would like to use the existing daemons, eg. I set up 5 daemons, but the outer loop only utilizes 3 so the other daemons + the idle cores of the 3 could be used by the inner loop.

[shikokuchuo]

For use on a single machine, you want to avoid these nested hierarchies.

A single set of 5 daemons would be optimal in your example, rather than splitting them into 2 levels. Either run the inner loop sequentially, or the outer loop. You don't gain anything from doing a nested map, you only add the coordination overhead of running more things than necessary in parallel.

[DavZim]

Makes sense.

Is there a way to still have the nested mirai_maps but reuse the same daemons or is this addressed by #268 or not possible?
That would be ideal for my use-case. But I have to admit I am still developing the mental model for mirai.

[shikokuchuo]

It's not possible to have everything use the same daemons, as this can easily lead to deadlock. E.g. 4 daemons launch 4 mirai tasks. Each of those then tries to launch 4 mirai tasks. All daemons are already busy at this point, so none of the tasks from the daemons get done, and those daemons keep on waiting.

The best case would be for #268, but even there it's more for multiple applications on the computer sharing the same resources, e.g. if you have multiple Shiny or Plumber apps on the same server.