Max number of TCP based lookups per second peaked at about ~300 QPS

[yg8106]

We're building a DNS solution with PowerDNS as frontend and an API based backend.
In an environment of 3 PowerDNS Pods and 3 API backend Pods, we're able to reach 50K QPS when lookups are UDP based, but can only reach 900 QPS when lookups are TCP based. Scaling up PowerDNS Pods to 6 gets us 1800 QPS for TCP, so about 300 QPS per PowerDNS Pod and this seems to be where the bottleneck is for us to get to higher QPS for TCP based lookups.

Performance testing tool being used is: https://github.com/DNS-OARC/dnsperf, and we ran tests with,

dnsperf -T 4 -m tcp -c 250 -s <IP> -d tcp.txt -l 6000 -Q 60000 -S 10
More clients doesn't give us more total QPS.

PowerDNS configuration is as below,

root@powerdns-6dbfc49ff7-bd24d:/etc/powerdns# cat pdns.conf 
# basic configuration
launch=remote
consistent-backends=no
zone-cache-refresh-interval=0
# remote backend configuration
remote-connection-string=http:url=http://backend-api-svc:8080/v1/dns,post,post_json,url-suffix=,timeout=2000
loglevel=4
# no caching
cache-ttl=0
query-cache-ttl=0
negquery-cache-ttl=0
distributor-threads=32
receiver-threads=32
max-tcp-connections=1000
webserver=yes
webserver-address=0.0.0.0
webserver-allow-from=0.0.0.0/0
webserver-port=9090
api=yes

PowerDNS version,

root@powerdns-6b946cbc5d-f768q:/# pdns_server --version
May 21 05:07:56 PowerDNS Authoritative Server 4.9.5 (C) PowerDNS.COM BV
May 21 05:07:56 Using 64-bits mode. Built using gcc 11.4.0 on May  7 2025 09:30:51 by root@localhost.
May 21 05:07:56 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
May 21 05:07:56 Features: libcrypto-ecdsa libcrypto-ed25519 libcrypto-ed448 libcrypto-eddsa libgeoip libmaxminddb lua lua-records PKCS#11 protobuf sodium curl DoT scrypt 
May 21 05:07:56 Built-in modules: 
May 21 05:07:56 Loading '/usr/lib/x86_64-linux-gnu/pdns/libremotebackend.so'
May 21 05:07:56 Loaded modules: remote
May 21 05:07:56 Configured with: " '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--sysconfdir=/etc/powerdns' '--enable-systemd' '--with-systemd=/lib/systemd/system' '--with-dynmodules=bind ldap lmdb lua2 pipe gmysql godbc gpgsql gsqlite3 geoip remote tinydns' '--with-modules=' '--enable-ixfrdist' '--enable-tools' '--with-protobuf' '--enable-unit-tests' '--enable-lua-records' '--enable-experimental-pkcs11' '--enable-dns-over-tls' '--disable-silent-rules' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/pdns/pdns-4.9.5=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/pdns/pdns-4.9.5=. -fstack-protector-strong -Wformat -Werror=format-security'"
root@powerdns-6b946cbc5d-f768q:/#

Any tuning suggestions or troubleshooting ideas are welcome, thank you!