ElasticQuota evict pods failed.

[Xdydy]

I want to test the pod evict on v1.6.0, and I install the koordinator with the config below

apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-scheduler-config
  namespace: {{ .Values.installation.namespace }}
data:
  koord-scheduler-config: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: true
      resourceLock: leases
      resourceName: koord-scheduler
      resourceNamespace: {{ .Values.installation.namespace }}
    profiles:
      - pluginConfig:
        ...
        - name: ElasticQuota
          args:
            apiVersion: kubescheduler.config.k8s.io/v1
            kind: ElasticQuotaArgs
            quotaGroupNamespace: {{ .Values.installation.namespace }}
            enableCheckParentQuota: true # Here I set the True
            enableRuntimeQuota: true
            monitorAllQuotas: true
            revokePodInterval: 5s
            delayEvictTime: 5s

And I create 3 quotas below. Here I set the quota-child1's min resources to 0 means that there is no guaranteed for quota-child1. When one pod's quota is set to quota-child1, it can be killed at any time.

apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: ElasticQuota
metadata:
  name: quota-parent
  namespace: default
  labels:
    quota.scheduling.koordinator.sh/is-parent: "true"
spec:
  max:
    cpu: 500m
    memory: 40Mi
  min:
    cpu: 500m
    memory: 40Mi
---
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: ElasticQuota
metadata:
  name: quota-child1
  namespace: default
  labels:
    quota.scheduling.koordinator.sh/is-parent: "false"
    quota.scheduling.koordinator.sh/parent: "quota-parent"
spec:
  max:
    cpu: 500m
    memory: 40Mi
  min:
    cpu: 0
    memory: 0
---
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: ElasticQuota
metadata:
  name: quota-child2
  namespace: default
  annotations:
    quota.scheduling.koordinator.sh/evict-pods-exceed-min: "true" 
  labels:
    quota.scheduling.koordinator.sh/is-parent: "false"
    quota.scheduling.koordinator.sh/parent: "quota-parent"
spec:
  max:
    cpu: 500m
    memory: 40Mi
  min:
    cpu: 500m
    memory: 40Mi

And I first create the pod-example

apiVersion: v1
kind: Pod
metadata:
  name: pod-example
  namespace: default
  labels:
    quota.scheduling.koordinator.sh/name: "quota-child1"
spec:
  schedulerName: koord-scheduler
  containers:
  - command:
    - sleep
    - 5m
    image: busybox
    imagePullPolicy: IfNotPresent
    name: curlimage
    resources:
      limits:
        cpu: 40m
        memory: 30Mi
      requests:
        cpu: 40m
        memory: 30Mi
  restartPolicy: Never

After the pod-example's status is running, I create the pod2-example

apiVersion: v1
kind: Pod
metadata:
  name: pod2-example
  namespace: default
  labels:
    quota.scheduling.koordinator.sh/name: "quota-child2"
spec:
  schedulerName: koord-scheduler
  containers:
  - command:
    - sleep
    - 5m
    image: busybox
    imagePullPolicy: IfNotPresent
    name: curlimage
    resources:
      limits:
        cpu: 40m
        memory: 30Mi
      requests:
        cpu: 40m
        memory: 30Mi
  restartPolicy: Never

Since the memory sum of the two pods(60Mi) is greater than the max of parent-quota(50Mi), pod2-example is at status pending because of the code below.

koordinator/pkg/scheduler/plugins/elasticquota/plugin.go

Line 270 in 0cae469

if g.pluginArgs.EnableCheckParentQuota {

So I set the configmap and reinstall the koordinator

apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-scheduler-config
  namespace: {{ .Values.installation.namespace }}
data:
  koord-scheduler-config: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: true
      resourceLock: leases
      resourceName: koord-scheduler
      resourceNamespace: {{ .Values.installation.namespace }}
    profiles:
      - pluginConfig:
        ...
        - name: ElasticQuota
          args:
            apiVersion: kubescheduler.config.k8s.io/v1
            kind: ElasticQuotaArgs
            quotaGroupNamespace: {{ .Values.installation.namespace }}
            enableCheckParentQuota: false # Here I set the false
            enableRuntimeQuota: true
            monitorAllQuotas: true
            revokePodInterval: 5s
            delayEvictTime: 5s

and repeat the steps above, it will lead to the scene that pod-example and the pod2-example are running, without evicting any of the pods.

And I see the code below and search the logs it reports that the parent-quota's used > runtime.

koordinator/pkg/scheduler/plugins/elasticquota/quota_overuse_revoke.go

Line 61 in 0cae469

func (monitor *QuotaOverUsedGroupMonitor) monitor() bool {

However, it did not evict any pods because the pod is not reference for the parent-quota

koordinator/pkg/scheduler/plugins/elasticquota/core/group_quota_manager.go

Lines 742 to 765 in 0cae469

	func (gqm *GroupQuotaManager) updatePodCacheNoLock(quotaName string, pod *v1.Pod, isAdd bool) {
	quotaInfo := gqm.getQuotaInfoByNameNoLock(quotaName)
	if quotaInfo == nil {
	return
	}
	if isAdd {
	quotaInfo.addPodIfNotPresent(pod)
	} else {
	quotaInfo.removePodIfPresent(pod)
	}
	}
	func (gqm *GroupQuotaManager) UpdatePodIsAssigned(quotaName string, pod *v1.Pod, isAssigned bool) error {
	gqm.hierarchyUpdateLock.RLock()
	defer gqm.hierarchyUpdateLock.RUnlock()
	return gqm.updatePodIsAssignedNoLock(quotaName, pod, isAssigned)
	}
	func (gqm *GroupQuotaManager) updatePodIsAssignedNoLock(quotaName string, pod *v1.Pod, isAssigned bool) error {
	quotaInfo := gqm.getQuotaInfoByNameNoLock(quotaName)
	return quotaInfo.UpdatePodIsAssigned(pod, isAssigned)
	}

Is it a bug? Or is there other code I ingore?

Originally posted by @Xdydy in #2342

[ZiMengSheng]

what about runtime of quota-child1, can you search the related log?

[Xdydy]

what about runtime of quota-child1, can you search the related log?

Here is the logs after i apply the pod-example which related to quota-child1

``` I0609 12:23:41.740704 1 eventhandlers.go:126] "Add event for unscheduled pod" pod="default/pod-example" I0609 12:23:41.740799 1 runtime_quota_calculator.go:500] [UpdateOneGroupRequest finish] quotaName: quota-child1, quotaParentName: quota-parent, IsParent: false, CalculateInfo: {"Max":{"cpu":"500m","memory":"40Mi"},"Min":{"cpu":"0","memory":"0"},"AutoScaleMin":{"cpu":"0","memory":"0"},"Used":{},"NonPreemptibleUsed":{},"Request":{"cpu":"40m","memory":"30Mi"},"NonPreemptibleRequest":{},"ChildRequest":{"cpu":"40m","memory":"30Mi"},"SelfRequest":{"cpu":"40m","memory":"30Mi"},"SelfUsed":{},"SelfNonPreemptibleUsed":{},"SelfNonPreemptibleRequest":{},"SharedWeight":{"cpu":"500m","memory":"40Mi"},"Runtime":{},"Guaranteed":{},"Allocated":{}}, treeName: quota-parent, totalResource: {}, reqLimit: {"quota-child1":{"cpu":"40m","memory":"30Mi"},"quota-child2":{"cpu":"0","memory":"0"}}, refreshedVersion: 8 I0609 12:23:41.740839 1 scheduling_queue.go:537] "Pod moved to an internal scheduling queue" pod="default/pod-example" event="PodAdd" queue="Active" I0609 12:23:41.740729 1 eventhandler_pod.go:91] "Pod missed nodeName" pod="default/pod-example" I0609 12:23:41.740873 1 runtime_quota_calculator.go:500] [UpdateOneGroupRequest finish] quotaName: quota-parent, quotaParentName: koordinator-root-quota, IsParent: true, CalculateInfo: {"Max":{"cpu":"500m","memory":"40Mi"},"Min":{"cpu":"500m","memory":"40Mi"},"AutoScaleMin":{"cpu":"500m","memory":"40Mi"},"Used":{},"NonPreemptibleUsed":{},"Request":{"cpu":"40m","memory":"30Mi"},"NonPreemptibleRequest":{},"ChildRequest":{"cpu":"40m","memory":"30Mi"},"SelfRequest":{},"SelfUsed":{},"SelfNonPreemptibleUsed":{},"SelfNonPreemptibleRequest":{},"SharedWeight":{"cpu":"500m","memory":"40Mi"},"Runtime":{},"Guaranteed":{},"Allocated":{}}, treeName: koordinator-root-quota, totalResource: {"cpu":"11050m","ephemeral-storage":"440613398209","hugepages-1Gi":"0","hugepages-2Mi":"0","kubernetes.io/batch-cpu":"9119","kubernetes.io/batch-memory":"14116897760","kubernetes.io/mid-cpu":"4k","kubernetes.io/mid-memory":"0","memory":"25906232Ki","pods":"110"}, reqLimit: {"koordinator-default-quota":{"cpu":"4950m","memory":"2586Mi"},"quota-parent":{"cpu":"40m","memory":"30Mi"}}, refreshedVersion: 437 I0609 12:23:41.740875 1 schedule_one.go:80] "About to try and schedule pod" pod="default/pod-example" I0609 12:23:41.740892 1 schedule_one.go:93] "Attempting to schedule pod" pod="default/pod-example" I0609 12:23:41.740894 1 pod_handler.go:41] OnPodAddFunc default/pod-example add success, quota: quota-child1, tree: [] I0609 12:23:41.741094 1 runtime_quota_calculator.go:500] [UpdateOneGroupRuntimeQuota finish] quotaName: quota-parent, quotaParentName: koordinator-root-quota, IsParent: true, CalculateInfo: {"Max":{"cpu":"500m","memory":"40Mi"},"Min":{"cpu":"500m","memory":"40Mi"},"AutoScaleMin":{"cpu":"500m","memory":"40Mi"},"Used":{},"NonPreemptibleUsed":{},"Request":{"cpu":"40m","memory":"30Mi"},"NonPreemptibleRequest":{},"ChildRequest":{"cpu":"40m","memory":"30Mi"},"SelfRequest":{},"SelfUsed":{},"SelfNonPreemptibleUsed":{},"SelfNonPreemptibleRequest":{},"SharedWeight":{"cpu":"500m","memory":"40Mi"},"Runtime":{"cpu":"40m","memory":"30Mi"},"Guaranteed":{},"Allocated":{}}, treeName: koordinator-root-quota, totalResource: {"cpu":"11050m","ephemeral-storage":"440613398209","hugepages-1Gi":"0","hugepages-2Mi":"0","kubernetes.io/batch-cpu":"9119","kubernetes.io/batch-memory":"14116897760","kubernetes.io/mid-cpu":"4k","kubernetes.io/mid-memory":"0","memory":"25906232Ki","pods":"110"}, reqLimit: {"koordinator-default-quota":{"cpu":"4950m","memory":"2586Mi"},"quota-parent":{"cpu":"40m","memory":"30Mi"}}, refreshedVersion: 437 I0609 12:23:41.741119 1 runtime_quota_calculator.go:448] setClusterTotalResource, treeName: quota-parent, oldTotalResource: {}, newTotalResource: {"cpu":"40m","memory":"30Mi"}, reqLimit: {"quota-child1":{"cpu":"40m","memory":"30Mi"},"quota-child2":{"cpu":"0","memory":"0"}}, refreshedVersion: 9 I0609 12:23:41.741141 1 scale_minquota_when_over_root_res.go:180] GetScaleMinQuota, parQuota: quota-parent, subQuota: quota-child1, needScaleDimensions: [memory cpu], totalRes: {"cpu":"40m","memory":"30Mi"}, newMinQuota: {"cpu":"0","memory":"0"} I0609 12:23:41.741176 1 runtime_quota_calculator.go:500] [UpdateOneGroupRuntimeQuota finish] quotaName: quota-child1, quotaParentName: quota-parent, IsParent: false, CalculateInfo: {"Max":{"cpu":"500m","memory":"40Mi"},"Min":{"cpu":"0","memory":"0"},"AutoScaleMin":{"cpu":"0","memory":"0"},"Used":{},"NonPreemptibleUsed":{},"Request":{"cpu":"40m","memory":"30Mi"},"NonPreemptibleRequest":{},"ChildRequest":{"cpu":"40m","memory":"30Mi"},"SelfRequest":{"cpu":"40m","memory":"30Mi"},"SelfUsed":{},"SelfNonPreemptibleUsed":{},"SelfNonPreemptibleRequest":{},"SharedWeight":{"cpu":"500m","memory":"40Mi"},"Runtime":{"cpu":"40m","memory":"30Mi"},"Guaranteed":{},"Allocated":{}}, treeName: quota-parent, totalResource: {"cpu":"40m","memory":"30Mi"}, reqLimit: {"quota-child1":{"cpu":"40m","memory":"30Mi"},"quota-child2":{"cpu":"0","memory":"0"}}, refreshedVersion: 9 I0609 12:23:41.742601 1 plugin.go:879] Skip reserve with reservation since there are no matched reservations, pod default/pod-example, node: debian I0609 12:23:41.742659 1 plugin.go:940] Skip the Reservation PreBind since no reservation allocated for the pod default/pod-example on node debian I0609 12:23:41.744421 1 plugin.go:96] "Successfully apply patch for Pod" pod="default/pod-example" I0609 12:23:41.744761 1 default_binder.go:53] "Attempting to bind pod to node" pod="default/pod-example" node="debian" I0609 12:23:41.751012 1 cache.go:404] "Finished binding for pod, can be expired" podKey="c9d94c7c-c56b-4fb5-a7fa-11cc6ea8426f" pod="default/pod-example" I0609 12:23:41.751021 1 eventhandlers.go:171] "Delete event for unscheduled pod" pod="default/pod-example" I0609 12:23:41.751045 1 schedule_one.go:286] "Successfully bound pod to node" pod="default/pod-example" node="debian" evaluatedNodes=1 feasibleNodes=1 I0609 12:23:41.751049 1 group_quota_manager.go:667] updatePodUsed, deltaUsedIsZero and deltaNonPreemptibleUsedIsZero, quotaName: quota-child1, podName: pod-example, podUsed: {"cpu":"40m","memory":"30Mi"}, podNonPreemptibleUsed: null I0609 12:23:41.751056 1 eventhandlers.go:197] "Add event for scheduled pod" pod="default/pod-example" I0609 12:23:41.751068 1 pod_handler.go:69] OnPodUpdateFunc default/pod-example update success, quota:quota-child1, tree: [] I0609 12:23:41.761783 1 eventhandlers.go:218] "Update event for scheduled pod" pod="default/pod-example" I0609 12:23:41.761912 1 group_quota_manager.go:667] updatePodUsed, deltaUsedIsZero and deltaNonPreemptibleUsedIsZero, quotaName: quota-child1, podName: pod-example, podUsed: {"cpu":"40m","memory":"30Mi"}, podNonPreemptibleUsed: null I0609 12:23:41.761952 1 pod_handler.go:69] OnPodUpdateFunc default/pod-example update success, quota:quota-child1, tree: [] I0609 12:23:42.368838 1 controller.go:93] "Skip updating elasticQuota because there are no changes" elasticQuota="quota-child2" I0609 12:23:42.369026 1 controller.go:201] "ElasticQuota changed" elasticQuota="quota-parent" changed="quota.scheduling.koordinator.sh/request changed from cpu:0,memory:0 to cpu:40m,memory:30Mi, quota.scheduling.koordinator.sh/child-request changed from cpu:0,memory:0 to cpu:40m,memory:30Mi, quota.scheduling.koordinator.sh/runtime changed from to cpu:40m,memory:30Mi, used changed from cpu:0,memory:0 to cpu:40m,memory:30Mi" I0609 12:23:42.369059 1 controller.go:99] "Try updating elasticQuota since it has changed" elasticQuota="quota-parent" I0609 12:23:42.383793 1 controller.go:122] "Successfully patch elasticQuota" elasticQuota="quota-parent" I0609 12:23:42.383836 1 controller.go:93] "Skip updating elasticQuota because there are no changes" elasticQuota="koordinator-default-quota" I0609 12:23:42.383860 1 quota_handler.go:75] OnQuotaUpdateFunc update quota: quota-parent I0609 12:23:42.383898 1 controller.go:201] "ElasticQuota changed" elasticQuota="koordinator-root-quota" changed="quota.scheduling.koordinator.sh/request changed from cpu:4950m,memory:2586Mi to cpu:4990m,memory:2616Mi, used changed from cpu:4950m,memory:2586Mi to cpu:4990m,memory:2616Mi" I0609 12:23:42.383908 1 controller.go:99] "Try updating elasticQuota since it has changed" elasticQuota="koordinator-root-quota" I0609 12:23:42.383909 1 quota_handler.go:87] OnQuotaUpdateFunc success: quota-parent, tree: I0609 12:23:42.389278 1 controller.go:122] "Successfully patch elasticQuota" elasticQuota="koordinator-root-quota" I0609 12:23:42.389302 1 controller.go:93] "Skip updating elasticQuota because there are no changes" elasticQuota="koordinator-system-quota" I0609 12:23:42.389376 1 controller.go:201] "ElasticQuota changed" elasticQuota="quota-child1" changed="quota.scheduling.koordinator.sh/runtime changed from to cpu:40m,memory:30Mi, quota.scheduling.koordinator.sh/request changed from cpu:0,memory:0 to cpu:40m,memory:30Mi, quota.scheduling.koordinator.sh/child-request changed from cpu:0,memory:0 to cpu:40m,memory:30Mi, used changed from cpu:0,memory:0 to cpu:40m,memory:30Mi" I0609 12:23:42.389388 1 controller.go:99] "Try updating elasticQuota since it has changed" elasticQuota="quota-child1" I0609 12:23:42.389422 1 quota_handler.go:75] OnQuotaUpdateFunc update quota: koordinator-root-quota I0609 12:23:42.389442 1 quota_handler.go:87] OnQuotaUpdateFunc success: koordinator-root-quota, tree: I0609 12:23:42.398544 1 controller.go:122] "Successfully patch elasticQuota" elasticQuota="quota-child1" I0609 12:23:42.398832 1 quota_handler.go:75] OnQuotaUpdateFunc update quota: quota-child1 I0609 12:23:42.398856 1 quota_handler.go:87] OnQuotaUpdateFunc success: quota-child1, tree: I0609 12:23:42.764953 1 leaderelection.go:281] successfully renewed lease koordinator-system/koord-scheduler I0609 12:23:44.772952 1 leaderelection.go:281] successfully renewed lease koordinator-system/koord-scheduler I0609 12:23:44.955580 1 eventhandlers.go:218] "Update event for scheduled pod" pod="default/pod-example" I0609 12:23:44.955596 1 group_quota_manager.go:667] updatePodUsed, deltaUsedIsZero and deltaNonPreemptibleUsedIsZero, quotaName: quota-child1, podName: pod-example, podUsed: {"cpu":"40m","memory":"30Mi"}, podNonPreemptibleUsed: null I0609 12:23:44.955614 1 pod_handler.go:69] OnPodUpdateFunc default/pod-example update success, quota:quota-child1, tree: [] ```

And there is no more logs about quota-child1 after applying the pod2-example. But I get the quota status about quota-child1

kubectl get eq quota-child1 -o yaml
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: ElasticQuota
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.sigs.k8s.io/v1alpha1","kind":"ElasticQuota","metadata":{"annotations":{},"labels":{"quota.scheduling.koordinator.sh/is-parent":"false","quota.scheduling.koordinator.sh/parent":"quota-parent"},"name":"quota-child1","namespace":"default"},"spec":{"max":{"cpu":"500m","memory":"40Mi"},"min":{"cpu":0,"memory":0}}}
    quota.scheduling.koordinator.sh/child-request: '{"cpu":"40m","memory":"30Mi"}'
    quota.scheduling.koordinator.sh/request: '{"cpu":"40m","memory":"30Mi"}'
    quota.scheduling.koordinator.sh/runtime: '{"cpu":"40m","memory":"30Mi"}'
    quota.scheduling.koordinator.sh/shared-weight: '{"cpu":"500m","memory":"40Mi"}'
  creationTimestamp: "2025-06-02T12:07:42Z"
  generation: 4
  labels:
    quota.scheduling.koordinator.sh/is-parent: "false"
    quota.scheduling.koordinator.sh/parent: quota-parent
  name: quota-child1
  namespace: default
  resourceVersion: "7665463"
  uid: 7795c859-985a-4a1a-9655-0c5f83ff7091
spec:
  max:
    cpu: 500m
    memory: 40Mi
  min:
    cpu: "0"
    memory: "0"
status:
  used:
    cpu: 40m
    memory: 30Mi

[songtao98]

/assign @ZiMengSheng

[ZiMengSheng]

I got it. This is because the runtime of quota-child1 has not been updated. It should be updated regularly.

#2471 This PR seems to have done this. If you are interested, you can review it.