The host system hang when sending bursty packets

[vinawara]

Hello team,

I'm currently testing OpenNIC shell and it's device driver with Alveo U55N/C. As mentioned in topic, the system hang after sending a kind of bursty packets. Here's the details of the issue.

Test configuration
[HOST A [U55N/C]]<----[100G Optic]---->[[U55N/C] HOST B]
both hosts have almost identical hardware and software configurations.

• Host OS : UBT 20.04
• OpenNIC shell releae : ce85c8d
• OpenNCI Driver release : 12ff0cd

Sending a somewhat small-sized at a low rate works without any issues. However, when attempting to send a large-sized packet(e.g. 6000 bytes, which is fragmented into 1500-byte packets), the sender hangs. Desptie pressing 'Magic SysRq key', there is no oops or panic message on the console and the system appears to be frozen. Where should I begin troubleshooting this case?

Regards,
Jeff

[cneely-amd]

Hi @vinawara ,

Can you say more about the version of Vivado that was used to build the open-nic-shell? I've been seeing an issue recently that Vivado 2022.1 seems to work, but I've been observing issues with Vivado 2022.2 and potentially 2023.1.

Best regards,
--Chris

[vinawara]

Hi @cneely-amd,

Thank you for the comment.
I forgot to write the version information of Vivado that I used to build. The version was 2022.2 here's the information from 'vivado -version' command:

ubuntu@jeff-dev:~$ vivado -version
Vivado v2022.2 (64-bit)
SW Build 3671981 on Fri Oct 14 04:59:54 MDT 2022
IP Build 3669848 on Fri Oct 14 08:30:02 MDT 2022
Tool Version Limit: 2022.10
Copyright 1986-2022 Xilinx, Inc. All Rights Reserved.

Ii'll try to build with Vivado 2022.1 and get back to you.

BR,
Jeff

[vinawara]

Hi @cneely-amd,

Thanks to your comment. With the image built by Vivado 2022.1, I was able to observe there is no issue when sending bursty packets. But, the maximum throughput was about 7.8Gbps, it was measured with iperf3. Which performance is lower than I expected.
Can you suggest any possible reasons for this performance?

Thanks,
Jeff

[cneely-amd]

It depends on the capabilities of your machines, and iperf3 is single threaded. Typically for me, a single iperf3 will get ~14 Gbps or so, and multiple cores need to be used for sending and receiving at higher throughput. For example, I have two 16 core machines that I use for testing, and I run iperf3 like the following (running N=15 instances, binding each iperf3 instance to a processor core):

iperf Client 1
                client sport = 35980, server dport = 36690
taskset -c 1 iperf3 -c 192.168.20.2 -p 36690 --bind 192.168.20.4 --cport 35980 -t 40 -b 5G > /home/cneely/results/iperf_client_1.log &