[CI] SecurityRolesMultiProjectIT testUpdatingFileBasedRoleAffectsAllProjects failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #8874 / rhel-8_platform-support-unix
• elasticsearch-pull-request #76242 / part-1-fips

Reproduction Line:

./gradlew ":test:external-modules:test-multi-project:javaRestTest" --tests "org.elasticsearch.xpack.security.SecurityRolesMultiProjectIT.testUpdatingFileBasedRoleAffectsAllProjects" -Dtests.seed=8294A2C903659D52 -Dtests.locale=kde-TZ -Dtests.timezone=Asia/Tehran -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: iterable containing ["monitor"]
     but: no item was "monitor"

Issue Reasons:

• [main] 2 failures in test testUpdatingFileBasedRoleAffectsAllProjects (0.6% fail rate in 345 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testUpdatingFileBasedRoleAffectsAllProjects (0.6% fail rate in 345 executions)

Build Scans:

• elasticsearch-periodic-platform-support #8874 / rhel-8_platform-support-unix
• elasticsearch-pull-request #76242 / part-1-fips

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[tvernum]

Setting to low-risk because this relates to functionality that is not currently usable in production deployments.

[tvernum]

I'm pretty confident this is a real bug.

PR #129295 tried to fix the same issue (#126223) by increasing the timeout, but that's not the problem.

We can see in the test output that the roles.yml is loaded, and 2 roles are found, but yet /_security/user/_privileges seems to return the wrong values

The test starts:

[2025-06-20T17:09:59,502][INFO ][o.e.x.s.SecurityRolesMultiProjectIT][testUpdatingFileBasedRoleAffectsAllProjects] before test

We write to roles.yml

elasticsearch/test/external-modules/multi-project/src/javaRestTest/java/org/elasticsearch/xpack/security/SecurityRolesMultiProjectIT.java

Lines 115 to 119 in e437163

	rolesFile.update(Resource.fromString(Strings.format("""
	%s:
	cluster:
	- monitor
	""", roleName)));

[2025-06-20T17:09:59,504][INFO ][o.e.t.c.l.AbstractLocalClusterFactory][testUpdatingFileBasedRoleAffectsAllProjects] Updating roles.yml for node 'test-cluster-0'
[2025-06-20T17:09:59,508][INFO ][o.e.t.c.l.AbstractLocalClusterFactory][testUpdatingFileBasedRoleAffectsAllProjects] Successfully updated roles.yml for node 'test-cluster-0'

Then we create two projects per

elasticsearch/test/external-modules/multi-project/src/javaRestTest/java/org/elasticsearch/xpack/security/SecurityRolesMultiProjectIT.java

Lines 124 to 125 in e437163

	createProject(project1.id());
	createProject(project2.id());

[2025-06-20T17:09:59,536][INFO ][o.e.x.s.SecurityRolesMultiProjectIT][testUpdatingFileBasedRoleAffectsAllProjects] Created project Ip5hFQAAQAC_14Xu_____w : HTTP/1.1 200 OK
[2025-06-20T17:09:59,568][INFO ][o.e.x.s.SecurityRolesMultiProjectIT][testUpdatingFileBasedRoleAffectsAllProjects] Created project s-HexP__T_-PqVIiAAAAAA : HTTP/1.1 200 OK

We add 2 users

elasticsearch/test/external-modules/multi-project/src/javaRestTest/java/org/elasticsearch/xpack/security/SecurityRolesMultiProjectIT.java

Lines 127 to 131 in e437163

	String username1 = randomAlphaOfLength(5);
	createUser(project1, username1, roleName);
	String username2 = randomAlphaOfLength(7);
	createUser(project2, username2, roleName);

[2025-06-20T17:09:59,871][INFO ][o.e.x.s.a.u.TransportPutUserAction] [test-cluster-0] added user [EfBZW]
[2025-06-20T17:10:00,168][INFO ][o.e.x.s.a.u.TransportPutUserAction] [test-cluster-0] added user [HarYtZL]

Now the server sees that the roles file has been updated, and there are 2 roles in it (the normal _es_test_root role and the new test_role)

[2025-06-20T17:10:00,453][INFO ][o.e.x.s.a.s.FileRolesStore] [test-cluster-0] parsed [2] roles from file [/opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1750424511885841510/elastic/elasticsearch-periodic-platform-support/test/external-modules/multi-project/build/testrun/javaRestTest/temp/test-cluster1544977419028689753/test-cluster-0/config/roles.yml]
[2025-06-20T17:10:00,454][INFO ][o.e.x.s.a.s.FileRolesStore] [test-cluster-0] updated roles (roles file [/opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1750424511885841510/elastic/elasticsearch-periodic-platform-support/test/external-modules/multi-project/build/testrun/javaRestTest/temp/test-cluster1544977419028689753/test-cluster-0/config/roles.yml] changed)

Then we wait 20 seconds for the assertion to keep tripping, and shut down the test cluster

[2025-06-20T17:10:20,444][INFO ][o.e.x.s.SecurityRolesMultiProjectIT][testUpdatingFileBasedRoleAffectsAllProjects] Waiting for all cluster updates up to this moment to be processed
[2025-06-20T17:10:20,539][INFO ][o.e.x.s.SecurityRolesMultiProjectIT][testUpdatingFileBasedRoleAffectsAllProjects] There are still tasks running after this test that might break subsequent tests [health-node[c]].
[2025-06-20T17:10:20,540][INFO ][o.e.x.s.SecurityRolesMultiProjectIT][testUpdatingFileBasedRoleAffectsAllProjects] after test

Which implies that the server loaded a modifed roles.yml but /_security/user/privileges didn't change its output for the next 20 seconds (which is sufficient time to realise it ain't happening).