Some very basic questions

[olivier7121]

First, thanks a lot for this package. I didn't use it yet because I still have some unanswered basic questions about its functioning but it looks very promising.

1. Which approach of parallelisation (fork or socket) is chosen and is it possible for the developper to choose/change it? The very existence of the function everywhere suggests the socket approach was chosen but I'm not sure (I'm familiar with local synchronous parallelisation but not yet with local asynchronous parallelisation).
2. Closely related question: How to use everywhere to simply pass variables, functions etc. already defined within the host environment? Is there a possibility similar to parallel::clusterExport(cluster, vector_of_variables_names_to_be_shared_with_daemons)? Would something like everywhere({list(variable_1, variable_2, function_1, function_2)}) be sufficient?
3. Is it possible to limit the number of cores available (I guess through options(mc.cores = xx)?) to the daemons?

[shikokuchuo]

1. Which approach of parallelisation (fork or socket) is chosen and is it possible for the developper to choose/change it?

mirai takes a socket (message-passing) approach. It does not fork as this is unsafe in many cases.

2. Closely related question: How to use everywhere to simply pass variables, functions etc. already defined within the host environment? Is there a possibility similar to parallel::clusterExport(cluster, vector_of_variables_names_to_be_shared_with_daemons)? Would something like everywhere({list(variable_1, variable_2, function_1, function_2)}) be sufficient?

You could do everywhere({}, variable1, variable2) i.e. not execute anything but export those global variables, which will remain there as you are doing it in an everywhere() call.

3. Is it possible to limit the number of cores available (I guess through options(mc.cores = xx)?) to the daemons?

I'm not sure what those options refer to, but mirai does not use any options or environment variables for explicitness and ease of debugging apart from anything else.

[olivier7121]

Thank you very much for your answers.

Regarding 2., can I also export functions without having to define them within everywhere, exactly as global variables, just by referencing (but not defining) them within everywhere?

Regarding 3., there is then no way to control the maximum number of cores used by daemons?

[shikokuchuo]

Thank you very much for your answers.

Regarding 2., can I also export functions without having to define them within everywhere, exactly as global variables, just by referencing (but not defining) them within everywhere?

To explain, as mirai evaluation takes place within a temporary environment on the daemon, rather than the global environment, any variables you assign will not persist when it returns, even in an everywhere() call.
If you want to pass in a list of objects named_list, then a convenient way of doing that is:
everywhere({}, as.environment(named_list))

Regarding 3., there is then no way to control the maximum number of cores used by daemons?

If you're using local daemons then each process will use a single core, unless it's running multi-threaded code. There's no additional control that mirai imposes on code so that the results are fully transparent (won't be different just because mirai is running it). If you want to have things run differently if on a daemon, there's a new function in development mirai on_daemon(), so you can condition on that: if (on_daemon()) { } else { }.

[olivier7121]

Thank you again for your answers.

To explain, as mirai evaluation takes place within a temporary environment on the daemon, rather than the global environment, any variables you assign will not persist when it returns, even in an everywhere() call.

OK, so that's why everywhere({}, .args = named_list) doesn't work but everywhere({}, as.environment(named_list)) does, right?

A few new basic questions:

1. Is there any limitations (beyond obvious physical memory limitations) on the total number or size of variables and functions that can be shared/exported/transported to all the active daemons through everywhere? I am asking because in one case I'm transporting 29 variables and functions that are relatively small in size and I don't experience any issue accessing them in my subsequent ExtentedTask call, whatever the number of daemons I initially set; on the other hand, in another case, I'm transporting 45 variables and functions that are much larger, and while I don't experience any issue accessing them in my subsequent ExtentedTask call (the structure is identical to the first case) when I use one single daemon, I don't have access to any of these shared variables and functions when I initially set 2 or more daemons. Apparently the transport didn't succeed but I don't get any error message and I have no idea why this happens.
2. Do you confirm that everywhere is a blocking operation (this can indeed take time when large variables and functions have to be shared)?
3. Is mirai_map equivalent to using lapply within a mirai()? In the later case, is the parallelisation implicitly recognised and activated?

[shikokuchuo]

1. An everywhere() call returns a list of mirai (each is run on a separate daemon). So you can try everywhere()[] to see if any of the mirai returned an error.
2. everywhere() is not blocking in the main session, but it will take place before anything else happens on the daemons. If you want to also wait in the main process then use everywhere()[].
3. No. mirai doesn't shim lapply or do anything like that. mirai simply evaluates the code in a separate process, so using lapply() within a mirai() just means you're doing that sequentially in another process.