Constant-time lookup for a resolved task

[wlandau]

crew implements a special queue to efficiently manage large collections of tasks. Even then, I still need to frequently loop over all the pending mirai objects in the compute profile and call .unresolved() on each one. (Solving wlandau/crew#221 should help, but overhead will still be noticeable at around 100k-1M tasks.) Is there a constant-time way to look up a resolved task, knowing just the compute profile?

[wlandau]

The condition variable from nextget("cv") is fantastic because it's O(1) and tells me that there exists a resolved task. But then digging for such a task is O(n).

[shikokuchuo]

Following our discussion last week, this is perhaps further evidence that a condition variable may be unsuited to the task in hand here. Condition variables are designed for control flow, rather than to provide state neither in terms of a cumulative number of events, nor identify which in your case above.

Instead it's perhaps time for you to fully embrace 'later'! The good things is it's already a dependency of crew. I think things would work out nicely if you make each mirai a promise and run your logic from the promise action. As these are tied to the mirai, the above problem no longer exists.

You'd simply replace until(cv, n) with later::run_now(n/1000).

[wlandau]

{crew} only tracks the name of each task and how many times it crashed so far. Then {targets} is responsible for reloading the data from storage and resupplying the data and the command when it retries the task.

[shikokuchuo]

So do I have this right that crew has some kind of account-keeping that tracks the name of each task, and number of crashes. Targets reads this and decides whether to re-launch a task. At this point a new crew/mirai task gets launched that re-uses the same name but with a updated no. of times run. Or is there a more accurate way to think about things?

[wlandau]

That’s pretty much it. One minor point though is that targets always blindly resubmits the task and then crew errors out if the retry quota for that task is exceeded. But otherwise, that’s exactly how crew and targets work.

[shikokuchuo]

Thanks. I need to think if it makes sense for mirai() to have an ID. Btw. are you planning to release the next crew version after mirai or are there simplifications you can get in before the next mirai release? Either's fine, but just so I have an idea.

[wlandau]

I think it may be a while before I am ready for another crew release. In addition to replacing condition variables with callbacks (if there is a way to make the latter fast enough), I want to fundamentally simplify the auto-scaling model (to drop reliance on worker IDs and network events reported by mirai::status()). Unless I missed anything, these deep changes should be enough to let other mirai-based workflows use an existing pool of crew daemons, so e.g. everyone using purrr or tidymodels should be able to use crew’s full set of launchers, and even create crew launchers of their own (see the crew plugins vignette). These changes should also provide a good foundation to support job arrays.

So that’s where I am trying to go with this. I am eager, but it will take time, and so far I’m having trouble making a million callbacks light enough to avoid inundating the R event loop.