Quorum Queue synchronisation

[luos]

Hi,

I was testing quorum queues and I had this setup:

3 node, single queue with a million 2kb messages.
I was doing a rolling restart, and I was also dropping the storage from the restarted node, which meant that when the follower came up it was empty.

Seemingly everything went fine, if i restart a single node, nothing really happens, but when I publish the first message the memory usage jumps very high - i guess this triggers a resync.

If I just restart all three nodes in sequence without publishing a message then the queue "crashes" and displays:
2021-03-25 08:39:50.953 [info] <0.583.0> queue 'test' in vhost '/': leader call - leader not known. Command will be forwarded once leader is known.

Probably there are some issues catching up existing followers when they lost state? Tested with 3.8.9, with policy of 10MB max message contents in memory.

[johanrhodin]

Hi,

I saw this issue (I think) today and was able to fix it by running:
rabbitmqctl eval "ra:overview()." to get the "Ra-name" of all the queues and then trigger a re-election for them:
rabbitmqctl eval "ra:trigger_election('something_something.queue.name1')."
rabbitmqctl eval "ra:trigger_election('something_something.queue.name2').", etc.

Note: It started on RabbitMQ 3.8.5 and I upgraded it to 3.8.23 before running the commands for re-elections.

[johanrhodin]

@kjnilsson is there a ra-command for triggering election for all QQs?

[kjnilsson]

No. It should never be necessary to do so.

[johanrhodin]

Understood. In this case it was a case of "pest eller kolera" (lesser of two evils), as we had to remove /quorum to not have the {timeout,{gen_statem,call,[{mqtt_node,'rabbit@server-name'},trigger_election,5000]}-issue on 3.8.5. See issue ref.

[kjnilsson]

Please retest with 3.9.9.

[kjnilsson]

Also you can’t just drop the storage. The guarantees only apply with persistent storage in place.

[luos]

It can happen that the storage goes away. :)
In any case... this was a configuration of some kubernetes cluster, because it was allocating non persistent volumes and they got deleted on restarts, which was fixed after it was discovered but still, it was somewhat unexpected.