ENH for `AzureVMCluster`: timeout?

[nickeubank]

One of the really nice AzureMLCluster features was a timeout option (scheduler_idle_timeout). Is that something it might be possible to port to AzureVMCluster? I work a lot with students, and it's a guardrail I find really useful for them to have. There's currently a "Shutdown the VM if the Dask process exits." option, but no "kill an idle cluster" option that I see.

[jacobtomlinson]

Is that something it might be possible to port to AzureVMCluster?

Sadly this is non-trivial on Azure.

The Shutdown the VM if the Dask process exits was intended to do this. On AWS you can set a VM to automatically destroy itself if it shuts down, so things behave as you would expect there.

I naively assumed other cloud providers would also have this option, but it seems to be an AWS specific thing. I'd love to be wrong about this if anyone can point me to docs on doing this.

I think when Azure VMs shut down you are no longer billed for them, but you will still need to destroy them yourself.

[jacobtomlinson]

Discussions like this are the main reason I'm exploring improving the Dask cluster lifecycle management tooling. Today cluster management is very ad-hoc and differs between sub-projects.

See dask/distributed#4263 for more info.