Stream cannot be replicated on non-leader nodes after partition #13768
-
Community Support Policy
RabbitMQ version used4.0.8 Erlang version used27.3.x Operating system (distribution) usedUbuntu 22.04 How is RabbitMQ deployed?Community Docker image rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics Logs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs rabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location Steps to deploy RabbitMQ clusterThree node cluster Steps to reproduce the behavior in question
advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application code# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKSWhat problem are you trying to solve?I run a 3-node setup with a stream created with node 2 as leader. After network partition and restoring to 3 nodes again, the leader cannot create replica on the non-leader nodes. Is this a known issue, what's might be missing in the setup to handle a situation like this? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
@malmetom according to the logs, all replicas report a missing local file. This is not a known problem. Perhaps the stream could never finish replicating even the initial "chunk". Can you share data directories of these nodes? That's the only way to suggest anything specific. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin thanks for reaching out I got the issue again, see below for directories before(everything looks fine) and after one node has been isolated Before partition: Node 1: Node 2: Node 3: Leader was on node 58 58 was isolated and leader moved to node 57 After partition: log getting repeated errors: Node 2: user@57: Log getting repeated warnings: Sorry for the strikethroughs in the attachd text above, shouldn't be there Also those are the feature flags in use: Flag: classic_mirrored_queue_version, state: enabled Seems like directory exists, but files inside are different and even increased on the new leader. The stream uses x-max-length-bytes: 3072 and x-stream-max-segment-size-bytes: 256 |
Beta Was this translation helpful? Give feedback.
-
Yes most definitely. It is on oversight on our behalf to allow such small I would strongly recommend re-declaring the stream with at least 16MB Related: rabbitmq/osiris#170 |
Beta Was this translation helpful? Give feedback.
-
|
Ok, thanks, will change. Got same issue with 4.0.9. I can also see that there is a period of only one node up I had a case with node 58 being the leader for the stream and in the log I see 2025-04-17 10:09:54.025656+02:00 [info] <0.419.0> rabbit on node '[email protected]' down then starts to get a few warnings about replication 2025-04-17 10:11:54.561324+02:00 [warning] <0.23956.0> rabbit_stream_coordinator: Error while starting replica for __rssi_ble_stream_1744874840828243762 on node [email protected] in 2 : {{bad_return_value,missing_file},{gen_server,call,[<14766.452.0>,await,infinity]}} Before 57 comes up again 2025-04-17 10:11:54.646499+02:00 [info] <0.419.0> rabbit on node '[email protected]' up So seems like there is a period of ~24s with only leader up and running, should it be possible for the stream to survive that as long as the max-length-bytes and max-segment-size-bytes have reasonable values? |
Beta Was this translation helpful? Give feedback.
-
|
I haven't spent a lot of time investigating but try a larger segment size and see if it works better. I'm not sure why you would have such a tiny segment size anyway. |
Beta Was this translation helpful? Give feedback.
-
|
we're considering shipping a change to claim segment sizing to a reasonable size if configuration requested is too small. |
Beta Was this translation helpful? Give feedback.
-
|
@malmetom also, if you can reproduce this behavior reliably, please use |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin Will give it a try with a newer rabbitmq, and come back with result |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin @kjnilsson Haven't seen the issue since removing x-stream-max-segment-size-bytes: 256, so guess it's working when using the default value |
Beta Was this translation helpful? Give feedback.
-
|
@malmetom max segment of 256 bytes is way too low to be practically useful. That setting supports values up to 3 GiB. |
Beta Was this translation helpful? Give feedback.
@malmetom max segment of 256 bytes is way too low to be practically useful. That setting supports values up to 3 GiB.