Cluster becomes unstable after NRE in "DotNetty.Common.ThreadLocalPool.WeakOrderQueue.Transfer"

[boris-rg]

Version Information
1.5.40

Describe the bug
After the following error repeats over and over, the cluster becomes unstable, and the SBR decides to take down the cluster.

 System.NullReferenceException: Object reference not set to an instance of an object.
   at DotNetty.Common.ThreadLocalPool.WeakOrderQueue.Transfer(Stack dst)
   at DotNetty.Common.ThreadLocalPool.Stack.ScavengeSome()
   at DotNetty.Common.ThreadLocalPool`1.Take()
   at Akka.Remote.EndpointWriter.WriteSend(Send send)
   --- End of inner exception stack trace ---
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level, Boolean needToThrow)
   at Akka.Remote.EndpointWriter.WriteSend(Send send)
   at Akka.Remote.EndpointWriter.<Writing>b__28_0(Send s)
   at lambda_method315(Closure, Object, Action`1, Action`1, Action`1)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)"]

To Reproduce
Occurs randomly.

Environment
.NET 9, AWS EC2, Linux

[Aaronontheweb]

@boris-rg I'm not even sure we can fix that - looks like a DotNetty bug.

[boris-rg]

@Aaronontheweb I see there are plans to replace dotnetty? This issue takes down our cluster every week.

for instance,

[Akka.Remote.ReliableDeliverySupervisor] Association with remote system "akka.tcp://...@..." has failed; address is now gated for 5000 ms. Reason is: ["Akka.Remote.EndpointException: Failed to write message [ActorSelectionMessage - Message: Akka.Cluster.ClusterHeartbeatSender+Heartbeat - WildCartFanOut: False - Elements: system/cluster/heartbeatReceiver] to the transport
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at DotNetty.Common.ThreadLocalPool.WeakOrderQueue.Transfer(Stack dst)
   at DotNetty.Common.ThreadLocalPool.Stack.ScavengeSome()
   at DotNetty.Common.ThreadLocalPool`1.Take()
   at Akka.Remote.EndpointWriter.WriteSend(Send send)

[Aaronontheweb]

We do for Akka.NET v1.6 but it's a ways off still - we could look into replacing DotNetty with Akka.IO.Tcp now that we've significantly improved the performance of it, but we'd still have to address issues like TLS support first before we could fully replace it.

[Aaronontheweb]

Looks like this is an ARM64-specific issue, due to the way DotNetty's work-stealing lock-free queue is implemented - I'll do a fuller write up later but for the time being I'm going to see if we can repro the issue in our ARM test lab environment.

[Aaronontheweb]

dev

Remote throughput over a duplex connection:

OSVersion:                         Unix 6.9.3.76060903                                                              
ProcessorCount:                    16                                                                               
ClockSpeed:                        0 MHZ                                                                            
Actor Count:                       32                                                                               
Messages sent/received per client: 200000  (2e5)                                                                    
Is Server GC:                      True                                                                             
Thread count:                      43                                                                               
                                                                                                                    
Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads                                          
         1,  200000,    107354,    1863.05,            43,              69                                          
         5, 1000000,    311333,    3212.11,            77,              84                                          
        10, 2000000,    689418,    2901.23,            92,              92                                          
        15, 3000000,    735655,    4078.55,           100,             100                                          
        20, 4000000,    750892,    5327.42,           108,             105                                          
        25, 5000000,    756659,    6608.57,           113,              97                                          
        30, 6000000,    657751,    9122.20,           105,              88             

dev with io.netty.recycler.maxCapacityPerThread=0

sudo io.netty.recycler.maxCapacityPerThread=0 dotnet run -c Release --framework net8.0

OSVersion:                         Unix 6.9.3.76060903
ProcessorCount:                    16
ClockSpeed:                        0 MHZ
Actor Count:                       32
Messages sent/received per client: 200000  (2e5)
Is Server GC:                      True
Thread count:                      43

Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads
         1,  200000,    118484,    1688.39,            43,              69
         5, 1000000,    231001,    4329.39,            77,              85
        10, 2000000,    801604,    2495.18,            93,              93
        15, 3000000,    681664,    4401.60,           101,             107
        20, 4000000,    755573,    5294.18,           115,             107
        25, 5000000,    731315,    6837.63,           115,              99
        30, 6000000,    736649,    8145.23,           107,              95

No real perf impact from turning this off - and #7677 validated that the io.netty.recycler.maxCapacityPerThread environment variable definitely reached DotNetty's active configuration.

[Aaronontheweb]

Setting the environment variable io.netty.recycler.maxCapacityPerThread to 0 should alleviate this issue - that disables the functionality that causes this problem in DotNetty. In essence, this should disable any sort of local pooling of tasks / resources and lets the garbage collector handle all of it instead.

[Aaronontheweb]

ARM64 Benchmark Data

dev

OSVersion:                         Unix 6.6.47.2712                                                                 
ProcessorCount:                    4                                                                                
ClockSpeed:                        0 MHZ                                                                            
Actor Count:                       8                                                                                
Messages sent/received per client: 200000  (2e5)                                                                    
Is Server GC:                      True                                                                             
Thread count:                      29                                                                               
                                                                                                                    
Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads                                          
         1,  200000,     40593,    4927.28,            29,              44                                          
         5, 1000000,    124425,    8037.13,            52,              54                                          
        10, 2000000,    135640,   14745.89,            62,              53                                          
        15, 3000000,    125167,   23968.23,            61,              40                                          
        20, 4000000,    132957,   30085.08,            48,              39                                          
        25, 5000000,    135748,   36833.71,            47,              39                                          
        30, 6000000,    127810,   46945.43,            47,              40                                          

dev with io.netty.recycler.maxCapacityPerThread=0

Attempted to elevate process priority, but failed due to Permission denied - carrying on at normal process priority.OSVersion:                         Unix 6.6.47.2712
ProcessorCount:                    4
ClockSpeed:                        0 MHZ
Actor Count:                       8
Messages sent/received per client: 200000  (2e5)
Is Server GC:                      True
Thread count:                      29

Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads
         1,  200000,     40462,    4943.67,            29,              42
         5, 1000000,    117661,    8499.81,            50,              52
        10, 2000000,    131840,   15170.72,            60,              44
        15, 3000000,    126626,   23692.41,            52,              39
        20, 4000000,    120399,   33223.90,            47,              38
        25, 5000000,    122444,   40835.48,            47,              42
        30, 6000000,    121793,   49264.40,            50,              41
Done..