how do I control daemon output?

[ElianHugh]

Is it possible to provide more granular control over the output of a daemon? I am interested in formatting the stdout/stderr of the output to ensure the terminal isn't garbled.

I've been using output = TRUE, which works but often the output is hard to read. I'm not sure if I'm missing a way of controlling the output, or maybe accessing the connections directly?

Thanks!

[shikokuchuo]

You can set output=TRUE and sink stdout/stderr as you normally would. If you want this to apply to all daemons you probably want to use an everywhere() call.

Or is it that you want everything to still output to the terminal but just ensuring they are nicely interleaved?

[ElianHugh]

Yes, mainly I'm interested in having things interleaved nicely in the terminal. I find that I often would like to add a line break to the mirai output as it breaks things like the {cli} package's progress bars.

Is the implication re: sinking that separate connections exist for a mirai's stdout and stderr? I tried looking through the daemon.R file but admittedly couldn't find much outside of how a nullfile is used when the output isn't sinked.

[shikokuchuo]

Just arrived at useR, so will have to take a look at this afterwards.

However I'm finding things behaving on my side e.g. the below has stdout and stderr nicely interleaved.

library(mirai)
daemons(4, output = TRUE)
#> [1] 4
m <- mirai_map(1:10, function(x) {cat(sprintf("%s new event\n", format(Sys.time())), file = stdout()); cat(sprintf("%s error event\n", format(Sys.time())), file = stderr())})
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event
2024-07-08 19:00:23 new event
2024-07-08 19:00:23 error event

It would help if you give me a reprex of what you're encountering and I'll see what I can do.

As for you question, a mirai is just an Rscript process so there is nothing special going on. The default is to have the daemon process silent, hence all output is sent to the nul device. I'm aware that R itself may not guarantee that nice things happen if multiple processes try to write at the same time but I don't remember if this is the case or not either way.

[ElianHugh]

Thank you, I think it's more precise for me to say that the mirai output is not interleaved with terminal outputs. i.e. when printing to the terminal, mirai output can overwrite text when they happen to coincide. It might have something to do with cli's progress bars messing things up... Some ANSI magic perhaps?

I will provide a reprex soon!

[shikokuchuo]

Did you manage to solve this in the end? From your description it sounds like it's because you're using something like cat() which uses Rprintf() under the hood which streams the output, and different processes can be trying to do this at the same time. You'd need a different function that writes the entire output in one go - I think data.table::fwrite() would work here.

[shikokuchuo]

I believe using a non-buffered write function would work as I suggested. If you still have an issue, please feel free to re-open.

[mikkmart]

Thanks! Unfortunately that didn't seem to help either. (At this point it's probably relevant that I'm trying this on Windows.)

// R CMD SHLIB mirai_output.c
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <Rinternals.h>

SEXP rwrite(SEXP x, SEXP con) {
    if (!Rf_isString(x) || Rf_xlength(x) != 1) {
        Rf_error("`x` must be a character vector with length 1."); 
    }
    
    int fd = Rf_asInteger(con);
    const char *message = CHAR(STRING_ELT(x, 0));
    size_t message_len = strlen(message);
    
    if (write(fd, message, message_len) == -1) {
        Rf_error("%s", strerror(errno));
    };
    
    return R_NilValue;
}

# mirai_output.R
library(mirai)

cwrite <- function(x, con = stdout()) {
  dynlib_path <- paste0("mirai_output", .Platform$dynlib.ext)
  dyn.load(dynlib_path)
  on.exit(dyn.unload(dynlib_path))
  invisible(.Call("rwrite", as.character(x), con))
}

args <- commandArgs(TRUE)
func <- if (length(args)) args[[1]] else "cat"

invisible(daemons(4, output = TRUE))

hello_world <- func |> switch(
  cat        = \(i) cat("Hello, World!", "\n", sep = ""),
  writeLines = \(i) writeLines("Hello, World!"),
  fwrite     = \(i) data.table::fwrite(list("Hello, World!"), quote = FALSE),
  cwrite     = \(i) cwrite("Hello, World!\n"),
  stop("unknown output function: ", func)
)

invisible(mirai_map(1:5, hello_world, cwrite = cwrite)[])

$ Rscript mirai_output.R cwrite
HelHHlello, World!o, Woello, rld!
HW
o, rWorlldd!!

Hello, World!

[shikokuchuo]

Nice, I believe this would work on Linux/Mac. For Windows, it seems you'd need to use the Win32 native APIs i.e. WriteFile() and get stdout as a HANDLE.

[mikkmart]

Thanks again. With some LLM assistance I managed to implement a basicWriteFile approach and can confirm I'm consistently getting the messages on separate lines on Windows with the following:

// R CMD SHLIB mirai_output.c
#include <string.h>
#include <errno.h>
#include <Rinternals.h>

#ifdef _WIN32
    #include <windows.h>
    #include <io.h>
#else
    #include <unistd.h>
#endif

SEXP rwrite(SEXP x, SEXP con) {
    if (!Rf_isString(x) || Rf_xlength(x) != 1) {
        Rf_error("`x` must be a character vector with length 1."); 
    }

    const char *message = CHAR(STRING_ELT(x, 0));
    size_t message_len = strlen(message);

#ifdef _WIN32
    HANDLE handle = (HANDLE)_get_osfhandle(Rf_asInteger(con));
    if (!WriteFile(handle, message, message_len, NULL, NULL)) {
        Rf_error("%s", strerror(errno));
    }
#else
    int fd = Rf_asInteger(con);
    if (write(fd, message, message_len) == -1) {
        Rf_error("%s", strerror(errno));
    }
#endif

    return R_NilValue;
}

[shikokuchuo]

Thanks for confirming. It's a bit crazy that there isn't a function that wraps this in R, so I've implemented writec() in nanonext in this PR r-lib/nanonext#122. It's fairly basic, but I've deliberately kept it as such for now. The idea is not for nanonext to collect these general utility functions, but it is meant to be a 'concurrency toolbox', so I thought this was just about in scope.

[shikokuchuo]

Apologies if you've already started using it, but this is now re-named nanonext::write_stdout() which is semantically much more obvious.