c-kzg verifyCellKzgProofBatch high process_heap_bytes

[twoeths]

Describe the bug

Got different behavior of nodes regarding process_heap_bytes

• node 0 - MIN_DATA_COLUMN_SIDECAR_BATCH_SIZE = 4: it's >= 45GB and keeps growing

here we batch ~4 messages per ckzg.verifyCellKzgProofBatch() call

• node 1 - MIN_DATA_COLUMN_SIDECAR_BATCH_SIZE = 16: it started with 3.3GB then grew to ~16GB then ~47GB in the end

here we batch ~10 messages per ckzg.verifyCellKzgProofBatch() call

• node 2 - MIN_DATA_COLUMN_SIDECAR_BATCH_SIZE = 32: it stayed at < 3GB then jumped to ~35GB in the end

here we batch ~13.5 messages per ckzg.verifyCellKzgProofBatch() call

• node 3 - MIN_DATA_COLUMN_SIDECAR_BATCH_SIZE = 64, it's ~3GB for ~6h then started to jumped to 4.7GB

here we batch ~15 messages per ckzg.verifyCellKzgProofBatch() call

Expected behavior

it should not grow over time, ideally should be ~<= 13GB, similar to a mainnet node subscribing to all subnets

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

te/validate_data_column_sidecar_in_batch

[twoeths]

• node 0 rss / process_heap_bytes increased to 72.5GB

• same to node 1 and node 2

• node 3 spiked to ~52.5GB then dropped back down to ~17GB

[twoeths]

it also happens on peerDAS so it's an issue of c-kgz verifyCellKzgProofBatch itself