Option to change validating/mutating webhook timeouts

[iambugra]

Hi,

I have a chart I develop, which has the dependency to temporal-operator. When I do an install on my chart, I get an error on temporal-operator side. This is the helm install output with --debug flag used:

W1105 18:13:30.812864   54905 warnings.go:70] tls: failed to parse private key
Error: 1 error occurred:
        * Internal error occurred: failed calling webhook "mtemporalc.kb.io": failed to call webhook: Post "https://peaka-temporal-webhook-service.peaka.svc:443/mutate-temporal-io-v1beta1-temporalcluster?timeout=10s": no endpoints available for service "peaka-temporal-webhook-service"


helm.go:86: 2024-11-05 18:13:32.291759 +0300 +03 m=+15.640909876 [debug] 1 error occurred:
        * Internal error occurred: failed calling webhook "mtemporalc.kb.io": failed to call webhook: Post "https://peaka-temporal-webhook-service.peaka.svc:443/mutate-temporal-io-v1beta1-temporalcluster?timeout=10s": no endpoints available for service "peaka-temporal-webhook-service"

Shortly after this operation fails, I run a helm upgrade --install, and it works as expected. The reason may be the 10s timeout of mutating webhook is short for the creation of pod temporal-controller-manager behind the temporal-webhook-service. Thus, an option to change webhook timeouts may solve the problem. And there is also the tls: failed to parse private key warning at the top of the log, which may be related.

I am installing the chart version 0.5.0 of temporal-operator on RKE2 of version v1.30.5+rke2r1, which is installed on a debian machine.

To reproduce, you can install peaka.

[alexandrevilain]

Hi @iambugra !

I don't think that increasing the timeout is a good option. 10s looks more than enought. Maybe we can try to fix the liveness & readiness probes ?

[iambugra]

Thank you for the response. For the probes, yes, we can give it a try. Do you mean adding a custom health check logic to here and here, instead of healthz.Ping?