High memory usage on ClusterResourceSet controllers

[rikatz]

What steps did you take and what happened?

On a scale test we've been running, we have realized that ClusterResourceSet is consuming a large amount of memory from the environment.

The environment:

• 200 clusters created
• Every cluster contains 3 ClusterResourceSet (CNI = Antrea, CSI = vsphere-csi and CPI=vsphere-cloud-provider), with spec.strategy=Reconcile

The growth of memory is exponential with the cluster creation. Following is attached an extraction of the pprof flamegraph from the time the problem happened

What did you expect to happen?

• First of all, to check if this is the expected behavior
• If not, that the memory growth is not exponential with the ClusterResourceSet usage

Cluster API version

CAPI 1.5.4, CAPV 1.8.4

Kubernetes version

No response

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[sbueringer]

/triage accepted

[fabriziopandini]

/priority important-longterm

[vincepri]

Has this been investigated further? Wondering if we're talking about a memory leak, or memory consumption due to the nature of CRS

[sbueringer]

Not that I'm aware of

[sbueringer]

@rikatz Do you remember if this is about the # of memory allocations or the # of currently allocated objects? (i.e. basically do we have more memory churn or do we actually need exponentially more at the same time)