roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum failed with artifacts on release-25.2 @ a7ec8f927bf1bb46443080b7d3655869e7507416:

(test_runner.go:2323).func1: task ``task-139`` returned error: error reading data: pq: internal error while retrieving user account memberships: operation "get-user-session" timed out after 10s (given timeout 10s): internal error while retrieving user account: get auth info error: interrupted during singleflight load-value:authinfo-roachprod-2-2: context deadline exceeded
(follower_reads.go:823).verifyHighFollowerReadRatios: roachtestutil.addCookies: unable to extract sessionID: roachtestutil.GetSessionID: failed to authenticate: context canceled
test artifacts and logs in: /artifacts/follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicBufferedSender=false
• metamorphicLeases=epoch
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-50998

[stevendanna]

From what I can see, node 6 was never that happy in this test. It couldn't update its liveness record from nearly the beginning of its logs forward.

Looking at the implementation of the test, one thing I noted is that while we wait for the main database to be fully replicated, we don't really wait for any of the system tables to be fully replicated.