[Transform] Stuck in Stopping

[prwhelan]

There is a chance that a Transform can get stuck in Stopping if it fails to update its status and register the indexer thread.

• TransformTask updates the cluster state with the Transform's persistent Task. If that update fails, the persistent task will remain alive but the Transform indexer thread gets marked as Stopped.
• TransformStats displays a Stopped indexer as Stopping, because the persistent task exists but the indexer thread is not running.

Workaround:

• Stop the Transform with ?force=true, which will remove the persistent task
• Start the Transform

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)