After compiling in CentOS8, a malloc error occurred during runtime

[baiyesong]

Sub

I compiled the msquic-2.4.7 source code on the CentOS8 system. After configuring the environment, the compilation was successful (mabye?). Subsequently, an error occurred during local runtime. When I copied the same code to the Ubuntu system, it ran successfully without errors.

Linux system: Centos8
Kernel version: 4.18
cmake version: 3.20
msquic version: 2.4.7

Question

This error did not occur in the Ubuntu system. Could it be caused by the absence of UDP_SEGMENT on my end? I personally suspect that it might be due to the kernel version issue, but it's also possible that the versions of some dependencies are too low. Are there any relevant clues?

---

The following is the background information I provided.

Compilation stage

The only difference between the compilation in CentOS and that in Ubuntu is

-- Looking for UDP_SEGMENT
-- Looking for UDP_SEGMENT - not found

Its location in the CMakeList
check_symbol_exists(UDP_SEGMENT netinet/udp.h HAS_UDP_SEGMENT)
In fact, in the kernel version 4.18 of Centos8, I checked

/usr/include/linux/udp.h
/usr/src/kernels/4.18.0-348.7.1.el8_5.x86_64/include/uapi/linux/udp.h

It contains UDP_SEGMENT (but I haven't had time to change the path in the CMakeList)

The detailed compilation output is as follows:
compile.txt

Runtime stage

The commands I used during the runtime stage are

./secnetper -port:8500
./secnetper -target:localhost -port:8500 -cc:bbr -exec:maxtput -up:134217728 -conns:1 -streams:1 -ptput:1

The GDB information after the failure of the run is as follows:

Thread 6 "secnetperf" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff489f700 (LWP 26295)]
0x00007ffff68db37f in raise () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install libatomic-8.5.0-4.el8_5.x86_64 libgcc-8.5.0-4.el8_5.x86_64 libstdc++-8.5.0-4.el8_5.x86_64 numactl-libs-2.0.12-13.el8.x86_64
(gdb) thread 6
[Switching to thread 6 (Thread 0x7ffff489f700 (LWP 26295))]
#0  0x00007ffff68db37f in raise () from /usr/lib64/libc.so.6
(gdb) bt full
#0  0x00007ffff68db37f in raise () from /usr/lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff68c5db5 in abort () from /usr/lib64/libc.so.6
No symbol table info available.
#2  0x000000000041f3bc in quic_bugcheck (File=File@entry=0x6145d8 "/root/msquic_origin/src/platform/datapath_epoll.c", Line=Line@entry=2183, 
    Expr=Expr@entry=0x614b30 "SendData->TotalSize + MaxBufferLength <= sizeof(SendData->Buffer)") at /root/msquic_origin/src/platform/platform_posix.c:93
No locals.
#3  0x000000000041e641 in SendDataAllocBuffer (SendData=0x7fffe4017900, MaxBufferLength=MaxBufferLength@entry=1452) at /root/msquic_origin/src/platform/datapath_epoll.c:2192
No locals.
#4  0x000000000041a43d in CxPlatSendDataAllocBuffer (SendData=<optimized out>, MaxBufferLength=MaxBufferLength@entry=1452) at /root/msquic_origin/src/platform/datapath_xplat.c:310
No locals.

more gdb info:
gdb.txt

Subsequently, I conducted analysis using Valgrind.:

==113521==  Uninitialised value was created by a heap allocation
==113521==    at 0x4C360A5: malloc (vg_replace_malloc.c:380)
==113521==    by 0x41F709: CxPlatAlloc (platform_posix.c:288)
==113521==    by 0x41E3B6: CxPlatPoolAlloc (quic_platform_posix.h:553)
==113521==    by 0x41E3B6: SendDataAlloc (datapath_epoll.c:2097)
==113521==    by 0x41A371: CxPlatSendDataAlloc (datapath_xplat.c:281)
==113521==    by 0x50383D: QuicPacketBuilderPrepare (packet_builder.c:271)
==113521==    by 0x503FB6: QuicPacketBuilderPrepareForControlFrames (packet_builder.c:608)
==113521==    by 0x4FCC18: QuicSendFlush (send.c:1349)
==113521==    by 0x435E9C: QuicConnDrainOperations (connection.c:7654)
==113521==    by 0x4EC11A: QuicWorkerProcessConnection (worker.c:559)
==113521==    by 0x4ECAA6: QuicWorkerLoop (worker.c:720)
==113521==    by 0x4ECEA1: QuicWorkerThread (worker.c:788)
==113521==    by 0x5D95179: start_thread (in /usr/lib64/libpthread-2.28.so)
==113521== 

more Valgrind info:
Valgrind.txt

[baiyesong]

ldd info:

[root@localhost Debug]# ldd ./secnetperf 
        linux-vdso.so.1 (0x00007fff28919000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa6f157a000)
        libatomic.so.1 => /lib64/libatomic.so.1 (0x00007fa6f1372000)
        libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fa6f1166000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa6f0dd1000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fa6f0a4f000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa6f0837000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa6f0617000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa6f0252000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa6f177e000)

[baiyesong]

Thanks to @nibanks for the help and reminder.

When I disabled the UDP_SEGMENT (with #undef UDP_SEGMENT in the epoll file) under Ubuntu, the exact same error that occurred in CentOS also showed up in Ubuntu.
Now I have temporarily resolved the issue that occurred in CentOS. I have temporarily written in src/platform/datapath_epoll.c:
#define UDP_SEGMENT 103
I need to conduct further experiments on the correct path of UDP_SEGMENT in the CMakeList file.

[baiyesong]

Through experiments, msquic will by default get the UDP_SEGMENT macro from the /usr/include/netinet/udp.h file (this is the path in both Ubuntu and CentOS8), which is defined as

#define UDP_SEGMENT 103

As shown in the CMakeList filecheck_symbol_exists(UDP_SEGMENT netinet/udp.h HAS_UDP_SEGMENT)
msquic uses the UDP_SEGMENT in the./src/platform/datapath_epoll.c file.

---

In CentOS 8, the /usr/include/netinet/udp.hfile does not contain UDP_SEGMENT.
the UDP_SEGMENT macro is defined in

/usr/include/linux/udp.h
/usr/src/kernels/4.18.0-348.7.1.el8_5.x86_64/include/uapi/linux/udp.h

This results in the UDP_SEGMENT not being recognizable during the compilation and runtime phases in CentOS8.
Moreover, the code of the msquic-2.4.7 version has a bug in the logic for handling the absence of UDP_SEGMENT, which leads to a memory out-of-bounds error during the runtime phase.

---

Before the problem is fixed, mabye there are two temporary solutions:

1. Manually introduce #define UDP_SEGMENT 103 in./src/platform/datapath_epoll.c.
2. Explicitly introduce the /usr/include/linux/udp.h header file in./src/platform/datapath_epoll.c.