[CI] CancellableTasksIT testRemoveBanParentsOnDisconnect failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #9447 / openjdk21_checkpart1_java-matrix
• elasticsearch-pull-request #81235 / part-1
• elasticsearch-intake #25601 / part1

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.action.admin.cluster.node.tasks.CancellableTasksIT.testRemoveBanParentsOnDisconnect" -Dtests.seed=D7A101AA69F3B6D6 -Dtests.locale=wo-SN -Dtests.timezone=America/Grand_Turk -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: safeGet: listener was completed exceptionally

Issue Reasons:

• [main] 3 failures in test testRemoveBanParentsOnDisconnect (0.5% fail rate in 640 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testRemoveBanParentsOnDisconnect (0.5% fail rate in 640 executions)

Build Scans:

• elasticsearch-periodic #9447 / openjdk21_checkpart1_java-matrix
• elasticsearch-pull-request #81235 / part-1
• elasticsearch-intake #25601 / part1

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DiannaHohensee]

There's another open test failure on this suite, #123568. May or may not be related.

[DiannaHohensee]

java.lang.AssertionError: safeGet: listener was completed exceptionally
at __randomizedtesting.SeedInfo.seed([D7A101AA69F3B6D6:5B7D96AF32635360]:0) |  
-- | --
  | at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2571) |  
  | at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2554) |  
  | at org.elasticsearch.test.ESIntegTestCase.doEnsureClusterStateConsistency(ESIntegTestCase.java:1450) |  
  | at org.elasticsearch.test.ESIntegTestCase.ensureClusterStateConsistency(ESIntegTestCase.java:1363) |  
  | at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:601) |  
  | at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2599)

Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "masterId" is null
at org.elasticsearch.action.support.PlainActionFuture$Sync.getValue(PlainActionFuture.java:279) |  
-- | --
  | at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:253) |  
  | at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:74) |  
  | at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2566) |  
  | at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2554) |  
  | at org.elasticsearch.test.ESIntegTestCase.doEnsureClusterStateConsistency(ESIntegTestCase.java:1450) |  
  | at org.elasticsearch.test.ESIntegTestCase.ensureClusterStateConsistency(ESIntegTestCase.java:1363) |  
  | at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:601) |  
  | at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2599)

I think the NPE is coming from this getMasterNodeId() call. DiscoveryNodes#getMasterNodeId() callers in DiscoveryNodes typically check for null before use. I expect there are times when there is no elected master. The following logs seem to suggest that could be the case here:

[2025-07-18T17:31:34,723][INFO ][o.e.c.s.MasterService    ][node_t2][masterService#updateTask][T#1] elected-as-master ([3] nodes joined in term 1)[_FINISH_ELECTION_, {node_t1}{_-O4SDnTT164zaklt4mDIA}{DJGvES_hQ9CluBZQQOg9Yw}{node_t1}{127.0.0.1}{127.0.0.1:26831}{cdfhilmrstw}{9.2.0}{8000099-9033000} completing election, {node_t2}{-HQ1dQMFRLSaWA1LL4JdXA}{jbPxmgZzQP-qVH7c380bLQ}{node_t2}{127.0.0.1}{127.0.0.1:26833}{cdfhilmrstw}{9.2.0}{8000099-9033000} completing election, {node_t0}{_DRNW649Ra-1mtJWPUwbAg}{bTK432ZWR5SaXYrzJaQNHQ}{node_t0}{127.0.0.1}{127.0.0.1:26832}{cdfhilmrstw}{9.2.0}{8000099-9033000} completing election], term: 1, version: 1, delta: master node changed {previous [], current [{node_t2}{-HQ1dQMFRLSaWA1LL4JdXA}{jbPxmgZzQP-qVH7c380bLQ}{node_t2}{127.0.0.1}{127.0.0.1:26833}{cdfhilmrstw}{9.2.0}{8000099-9033000}]}, added {{node_t1}{_-O4SDnTT164zaklt4mDIA}{DJGvES_hQ9CluBZQQOg9Yw}{node_t1}{127.0.0.1}{127.0.0.1:26831}{cdfhilmrstw}{9.2.0}{8000099-9033000}, {node_t0}{_DRNW649Ra-1mtJWPUwbAg}{bTK432ZWR5SaXYrzJaQNHQ}{node_t0}{127.0.0.1}{127.0.0.1:26832}{cdfhilmrstw}{9.2.0}{8000099-9033000}}	
...
...
[2025-07-18T17:31:35,086][INFO ][o.e.a.a.c.n.t.CancellableTasksIT][testRemoveBanParentsOnDisconnect] [CancellableTasksIT#testRemoveBanParentsOnDisconnect]: all set up test	
[2025-07-18T17:31:35,138][INFO ][o.e.t.ClusterConnectionManager][testRemoveBanParentsOnDisconnect] transport connection to [{node_t1}{_-O4SDnTT164zaklt4mDIA}{DJGvES_hQ9CluBZQQOg9Yw}{node_t1}{127.0.0.1}{127.0.0.1:26831}{cdfhilmrstw}{9.2.0}{8000099-9033000}] closed by remote; if unexpected, see [https://www.elastic.co/docs/troubleshoot/elasticsearch/troubleshooting-unstable-cluster?version=master#troubleshooting-unstable-cluster-network] for troubleshooting guidance	
[2025-07-18T17:31:35,139][INFO ][o.e.t.ClusterConnectionManager][testRemoveBanParentsOnDisconnect] transport connection to [{node_t0}{_DRNW649Ra-1mtJWPUwbAg}{bTK432ZWR5SaXYrzJaQNHQ}{node_t0}{127.0.0.1}{127.0.0.1:26832}{cdfhilmrstw}{9.2.0}{8000099-9033000}] closed by remote; if unexpected, see [https://www.elastic.co/docs/troubleshoot/elasticsearch/troubleshooting-unstable-cluster?version=master#troubleshooting-unstable-cluster-network] for troubleshooting guidance	
[2025-07-18T17:31:35,148][INFO ][o.e.t.ClusterConnectionManager][testRemoveBanParentsOnDisconnect] transport connection to [{node_t2}{-HQ1dQMFRLSaWA1LL4JdXA}{jbPxmgZzQP-qVH7c380bLQ}{node_t2}{127.0.0.1}{127.0.0.1:26833}{cdfhilmrstw}{9.2.0}{8000099-9033000}] closed by remote; if unexpected, see [https://www.elastic.co/docs/troubleshoot/elasticsearch/troubleshooting-unstable-cluster?version=master#troubleshooting-unstable-cluster-network] for troubleshooting guidance	
[2025-07-18T17:31:35,148][INFO ][o.e.c.s.MasterService    ][node_t2][masterService#updateTask][T#1] node-left[{node_t1}{_-O4SDnTT164zaklt4mDIA}{DJGvES_hQ9CluBZQQOg9Yw}{node_t1}{127.0.0.1}{127.0.0.1:26831}{cdfhilmrstw}{9.2.0}{8000099-9033000} reason: disconnected, {node_t0}{_DRNW649Ra-1mtJWPUwbAg}{bTK432ZWR5SaXYrzJaQNHQ}{node_t0}{127.0.0.1}{127.0.0.1:26832}{cdfhilmrstw}{9.2.0}{8000099-9033000} reason: disconnected], term: 1, version: 8, delta: removed {{node_t1}{_-O4SDnTT164zaklt4mDIA}{DJGvES_hQ9CluBZQQOg9Yw}{node_t1}{127.0.0.1}{127.0.0.1:26831}{cdfhilmrstw}{9.2.0}{8000099-9033000}, {node_t0}{_DRNW649Ra-1mtJWPUwbAg}{bTK432ZWR5SaXYrzJaQNHQ}{node_t0}{127.0.0.1}{127.0.0.1:26832}{cdfhilmrstw}{9.2.0}{8000099-9033000}}	
[2025-07-18T17:31:35,149][INFO ][o.e.c.c.Coordinator      ][node_t0][cluster_coordination][T#1] master node [{node_t2}{-HQ1dQMFRLSaWA1LL4JdXA}{jbPxmgZzQP-qVH7c380bLQ}{node_t2}{127.0.0.1}{127.0.0.1:26833}{cdfhilmrstw}{9.2.0}{8000099-9033000}] disconnected, restarting discovery	
[2025-07-18T17:31:35,157][INFO ][o.e.a.a.c.n.t.CancellableTasksIT][testRemoveBanParentsOnDisconnect] [CancellableTasksIT#testRemoveBanParentsOnDisconnect]: cleaning up after test	
[2025-07-18T17:31:35,167][INFO ][o.e.c.s.ClusterApplierService][node_t0][clusterApplierService#updateTask][T#1] master node changed {previous [{node_t2}{-HQ1dQMFRLSaWA1LL4JdXA}{jbPxmgZzQP-qVH7c380bLQ}{node_t2}{127.0.0.1}{127.0.0.1:26833}{cdfhilmrstw}{9.2.0}{8000099-9033000}], current []}, term: 1, version: 7, reason: becoming candidate: onLeaderFailure	
[2025-07-18T17:31:35,169][WARN ][o.e.c.s.MasterService    ][node_t2][masterService#updateTask][T#1] failing [node-left[{node_t1}{_-O4SDnTT164zaklt4mDIA}{DJGvES_hQ9CluBZQQOg9Yw}{node_t1}{127.0.0.1}{127.0.0.1:26831}{cdfhilmrstw}{9.2.0}{8000099-9033000} reason: disconnected, {node_t0}{_DRNW649Ra-1mtJWPUwbAg}{bTK432ZWR5SaXYrzJaQNHQ}{node_t0}{127.0.0.1}{127.0.0.1:26832}{cdfhilmrstw}{9.2.0}{8000099-9033000} reason: disconnected]]: failed to commit cluster state version [8]	
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication of cluster state version [8] in term [1] failed [committed=false]	
...
...
[2025-07-18T17:31:35,171][INFO ][o.e.c.s.ClusterApplierService][node_t2][clusterApplierService#updateTask][T#1] master node changed {previous [{node_t2}{-HQ1dQMFRLSaWA1LL4JdXA}{jbPxmgZzQP-qVH7c380bLQ}{node_t2}{127.0.0.1}{127.0.0.1:26833}{cdfhilmrstw}{9.2.0}{8000099-9033000}], current []}, term: 1, version: 7, reason: becoming candidate: Publication.onCompletion(false)	

[DiannaHohensee]

IIUC, this is failing during tearDown, so this failure has nothing to do with the test on which it occurred. Unless the test left some state behind... I wonder if this code

elasticsearch/server/src/internalClusterTest/java/org/elasticsearch/action/admin/cluster/node/tasks/CancellableTasksIT.java

Lines 348 to 361 in b941518

	// Disconnect some outstanding child connections
	for (DiscoveryNode node : nodes) {
	TaskManager taskManager = internalCluster().getInstance(TransportService.class, node.getName()).getTaskManager();
	for (TaskId bannedParent : bannedParents) {
	if (bannedParent.getNodeId().equals(node.getId()) && randomBoolean()) {
	Collection<Transport.Connection> childConns = taskManager.startBanOnChildTasks(bannedParent.getId(), "", () -> {});
	for (Transport.Connection connection : randomSubsetOf(childConns)) {
	if (connection.getNode().equals(node) == false) {
	connection.close();
	}
	}
	}
	}
	}

could close a connection important in maintaining master election some how.