[Performance] Overview of performance improvements for v1.14

[zach593]

Since v1.13, we have made several performance enhancements. This issue is intended to track the commits that contributed to performance improvements and to quantify the overall performance gain in the upcoming v1.14 release compared to v1.13.

Commits with Clear Performance Gains

• disable deepcopy for list action
  This PR significantly reduces the processing time, CPU usage, and memory consumption of related controllers in karmada-controller-manager.
  Performance comparison before and after the change can be found here:
  disable deepcopy for list action #5813 (comment)

• feat(detector): remove type convert in detector controller by controller-runtime
  By using statically defined values to cache PropagationPolicy/ClusterPropagationPolicy, this PR greatly reduces the execution time, CPU usage, and memory consumption of the resourceDetector in karmada-controller-manager.
  See detailed performance comparison here:
  feat(detector): remove type convert in detector controller by controller-runtime #5802 (comment)

• Add deepequal check for objectWatcher.Update()
  This PR reduces the number of HTTP requests sent from karmada-controller-manager to member clusters, which in turn decreases the workload of the execution controller and lowers CPU usage on member cluster kube-apiserver.
  Performance results:
  Add deepequal check for objectWatcher.Update() #6150 (comment)

• fix(dependenciesdistributor): reduce update trigger
  This PR introduces a deepEqual check on resourceTemplate events in DependenciesDistributor (within karmada-controller-manager) to avoid unnecessary processing.

• [performance] use meta.Accessor() in onResourceBindingUpdate()
  By using meta.Access() to parse rb/crb update events instead of converting them into unstructured.Unstructured, this PR reduces the processing time and CPU usage for handling update events in karmada-scheduler.

Commits with Potential Performance Impact

• unify http ratelimiter for control plane client
  This PR enforces stricter HTTP rate limiting for karmada-controller-manager when accessing the karmada-apiserver, ensuring that no requests bypass the configured rate limits.

• use singleton rate limiter for dynamically created member cluster clients
  This PR enforces stricter HTTP rate limiting when accessing member cluster kube-apiserver, by sharing a singleton rate limiter across components. It prevents any component from exceeding the predefined limits.

• add the ratelimiter options parameter for controllers that are missing it
  This PR adds missing rate limiter configuration support to several controllers in karmada-controller-manager, allowing users to tune the rate limiting behavior for these controllers via unified configuration options. It expands the coverage of existing global settings.

• feat(dependenciesdistributor): remove dependenciesdistributor concurrentreconciles
  This PR updates DependenciesDistributor to use the concurrency value specified by the --concurrent-resourcebinding-syncs flag, instead of the previously hardcoded value of 2. This allows users to better control parallelism and may help improve performance under certain workloads.

If I missed any commits, please let me know and I will add them to the list.

Performance improvements brought by version v1.14 compared to v1.13

karmada-controller-manager configuration:

The test setup includes 5,000 Deployments, each paired with a corresponding PropagationPolicy that schedules it to two member clusters. Each Deployment also has a unique ConfigMap as a dependency, which is distributed along with the Deployment to the same clusters.
These resources were created while karmada-controller-manager was offline, meaning Karmada is syncing them for the first time during the test.

Conclusion

Compared to v1.13.0, v1.14.0-beta.0 delivers significant performance improvements, especially in karmada-controller-manager:

• Cold start time (clear the workqueue) reduced from ~7 minutes to ~4 minutes, a 45% improvement.
• Resource detector: Max value of average processing time reduced from 391ms to 180ms (54% reduction).
• Dependencies distributor: Max value of average processing time reduced from 378ms to 216ms (43% reduction).
• Execution controller: Max value of average processing time reduced from 505ms to 248ms (50% reduction).

In addition to the faster processing speed, karmada-controller-manager shows a notable reduction in resource usage:

• CPU usage reduced from 4–7.5 cores to 1.8–2.4 cores (40%–65% reduction).
• Memory peak usage reduced from 1.9GB to 1.47GB (22% reduction).

Detailed Monitoring Data:

v1.13.0

workqueue depth:

cpu usage:

mem usage:

avg reconciliation processing duration:

v1.14.0-beta.0

workqueue depth:

cpu usage:

mem usage:

avg reconciliation processing duration:

[RainbowMango]

cc release manager here @zhzhuang-zju who might need this to draft the release notes.