Skip to content

Issue cert-manager certificates for etcd-operator #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

ArkaSaha30
Copy link
Contributor

This PR will add a basic capability to issue self-signed certificates for etcd-operator.

  • Prerequisite: cert-manager needs to be installed before deploying the etcd-operator
    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

  • Once etcd-operator is deployed it requests 3 certificates from an issuer in the etcd-operator-system namespace

    • peer certificate
    • client certificate
    • server certificate
deployment.apps/etcd-operator-controller-manager created
certificate.cert-manager.io/etcd-operator-etcd-client-certificate created
certificate.cert-manager.io/etcd-operator-etcd-peer-certificate created
certificate.cert-manager.io/etcd-operator-etcd-server-certificate created
issuer.cert-manager.io/etcd-operator-selfsigned created

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.22.7 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ kubectl get secrets -n etcd-operator-system
NAME              TYPE                DATA   AGE
etcd-client-tls   kubernetes.io/tls   3      33s
etcd-peer-tls     kubernetes.io/tls   3      33s
etcd-server-tls   kubernetes.io/tls   3      34s

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.22.7 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ kubectl -n etcd-operator-system get secret etcd-client-tls -o yaml | grep ca.crt
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMxRENDQWJ5Z0F3SUJBZ0lSQU41cTB2b2tNREsrT3haankvQmZMcE13RFFZSktvWklodmNOQVFFTEJRQXcKQURBZUZ3MHlOREV5TURZeE1USXpNalJhRncweU5UQXpNRFl4TVRJek1qUmFNQUF3Z2dFaU1BMEdDU3FHU0liMwpEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURDVHFVcjZNdnJXLzRYSjJna0Y4cWhhMnFWSnl1L1ozcmhuNENKCkZjQUdPOUlCUmxmbGpiajUvbU5iMW5SUkFJOXNraCtFdWlYd2Y1SFVQbTBNcGpnM3Y0MlRUbzd4MkIyTHloZmUKcDFrQVdBVlFPallOeFZHZjk1ZEFzK3U5MW5Xbi8zM2hHajlodlFTallJT1pORllFOS9xSHJQTlk3SEs4RFRCWQp4ZjZEaVIxSlFUZ1Jmc1FvRkpoZDRCMVhUeDV4eGlMdi85NzRLcmZDbmgrckMyeVVzZWRCMHlEaGlVTm9LVlM3CnU3RUdETC9SOFZUalJGWnphWEM4YXl6OUY1eTlwL1pmbjIzMElrOThqdVBzL0VURUZYUXBuUFFDaGFOTU9JaE0KcEs0ZWIvSWw3ek5lV3p0bnFvUjFJUSttYW1zTWd6bmk0RS9PMGxHbEYxekluMTBwQWdNQkFBR2pTVEJITUE0RwpBMVVkRHdFQi93UUVBd0lGb0RBTUJnTlZIUk1CQWY4RUFqQUFNQ2NHQTFVZEVRRUIvd1FkTUJ1Q0dXVjBZMlF1ClpYUmpaQzF2Y0dWeVlYUnZjaTF6ZVhOMFpXMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQlhhbGJiR1VIbUwKbURCK0piK3pMTzlYT2ZwR21JMGhZRmdPakVtZmxqem4wa0ZRRHo0YmQyc1FlVi95U1BIZC84RXN2Y0NlME1ucwpaRXF1ejNTWUp5cmg2a2g0MjhqVTJPRGI2MXBUMmwreTUvdkU4bi8yVzgxTERYQzlOUnZNYkR2MmlYOVpORFhuCnZndVhnOUJPUXByd3craXlBUVFHdFk4NWYvc2pOdVZnQW8vU1ZQR2RHVUFWME5RbllSNFN1ZmpwOTh5aS9EK1IKRGM5WDNiSUtYVDNNWmcyKzcyU0JYV3YzSjVBZEFyU3lnVlRMUWVqdk5LZzIvbDMyUkN2cFgzR01jYk1IbFBOQgp3NUpkSmU5QWc2QS9mdTZJS1lDREowUlRlQVkwcDlCaEgvUmFncmhsZkdvcVA4U3V5YjZXbjhoekNLT3FPZDVGClJGRlhzTC9Qd1N3PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

@ahrtr
Copy link
Member

ahrtr commented Dec 6, 2024

@ArkaSaha30 please rebase this PR. I just merged #8

@ahrtr
Copy link
Member

ahrtr commented Dec 6, 2024

A couple of high level thoughts:

  • For etcd-cluster's certificate,
    • add the cluster name as the prefix, i.e. cluster1-etcd-client-tls, etc. so that different clusters won't conflict.
    • expose an API for the reconciliation workflow to call to generate certificates for the etcd clusters
      • each member in one cluster should have different certificate

@ahrtr
Copy link
Member

ahrtr commented Dec 6, 2024

cc @hakman

@k8s-ci-robot k8s-ci-robot added size/M and removed size/L labels Dec 7, 2024
@ArkaSaha30 ArkaSaha30 force-pushed the issue-cert branch 2 times, most recently from 3685a85 to f5456d0 Compare December 7, 2024 18:33
@ArkaSaha30
Copy link
Contributor Author

A couple of high level thoughts:

  • For etcd-cluster's certificate,

    • add the cluster name as the prefix, i.e. cluster1-etcd-client-tls, etc. so that different clusters won't conflict.

    • expose an API for the reconciliation workflow to call to generate certificates for the etcd clusters

      • each member in one cluster should have different certificate

Sure, thank you!
Should I update it in this PR or raise a separate as an incremental development?

@ahrtr ahrtr mentioned this pull request Dec 9, 2024
11 tasks
@ahrtr
Copy link
Member

ahrtr commented Dec 9, 2024

Please read #10

@ArkaSaha30 ArkaSaha30 changed the title Issue certifcates for etcd-operator [WIP]Issue certifcates for etcd-operator Jan 12, 2025
@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ArkaSaha30
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ArkaSaha30
Copy link
Contributor Author

Updating the current progress here, making changes to this PR to accommodate certificates prefixed with etcd cluster names as per mentioned in the issue.
I'll remove [WIP] once it's completed and ready for review.

@gdasson
Copy link
Member

gdasson commented Jan 13, 2025

@ArkaSaha30 : Please let me know if you need help in any tasks. Just to avoid overlap, I'll wait for you to confirm if I can pick up some of the tasks. Thanks.

@ArkaSaha30
Copy link
Contributor Author

ArkaSaha30 commented Jan 13, 2025

@ArkaSaha30 : Please let me know if you need help in any tasks. Just to avoid overlap, I'll wait for you to confirm if I can pick up some of the tasks. Thanks.

Sure, thank you!
I will quickly finish this part following which it will be nice if you can help with the next task - Update reconciliation workflow to mount the secrets into etcd POD.

@gdasson
Copy link
Member

gdasson commented Jan 13, 2025

Sure, once you complete this, I will add that the code for secrets mounting in the same PR.

@ArkaSaha30
Copy link
Contributor Author

ArkaSaha30 commented Jan 14, 2025

Updated the PR to align with the certificate management design,

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k apply -k config/samples/                                              
etcdcluster.operator.etcd.io/etcdcluster-sample created

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get etcdcluster -A
NAMESPACE   NAME                 AGE
default     etcdcluster-sample   12s

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get certificates
NAME                                             READY   SECRET                               AGE
etcdcluster-sample-etcd-client-tls-certificate   True    etcdcluster-sample-etcd-client-tls   24s
etcdcluster-sample-etcd-peer-tls-certificate     True    etcdcluster-sample-etcd-peer-tls     24s
etcdcluster-sample-etcd-server-tls-certificate   True    etcdcluster-sample-etcd-server-tls   24s

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get secrets
NAME                                 TYPE                DATA   AGE
etcdcluster-sample-etcd-client-tls   kubernetes.io/tls   3      31s
etcdcluster-sample-etcd-peer-tls     kubernetes.io/tls   3      32s
etcdcluster-sample-etcd-server-tls   kubernetes.io/tls   3      32s

etcd-operator-controller-manager logs:

2025-01-14T23:12:16Z    INFO    Reconciling EtcdCluster {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "spec": {"size":4,"version":"3.5.17","tls":{"member":{"peerSecret":"etcd-peer-tls","serverSecret":"etcd-server-tls"},"operatorSecret":"etcd-client-tls","provider":"cert-manager"}}}
2025-01-14T23:12:16Z    INFO    Reconciling EtcdCluster certificates    {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "tls": {"member":{"peerSecret":"etcd-peer-tls","serverSecret":"etcd-server-tls"},"operatorSecret":"etcd-client-tls","provider":"cert-manager"}}
2025-01-14T23:12:16Z    INFO    Starting reconciliation of Client Certificate   {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "etcdcluster-sample-etcd-client-tls-certificate": "etcdcluster-sample-etcd-client-tls"}
2025-01-14T23:12:16Z    INFO    Starting reconciliation of Peer Certificate     {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "etcdcluster-sample-etcd-peer-tls-certificate": "etcdcluster-sample-etcd-peer-tls"}
2025-01-14T23:12:16Z    INFO    Starting reconciliation of Server Certificate   {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "etcdcluster-sample-etcd-server-tls-certificate": "etcdcluster-sample-etcd-server-tls"}
2025-01-14T23:12:16Z    INFO    Successfully reconciled EtcdCluster certificates        {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "tls": [{"metadata":{"name":"etcdcluster-sample-etcd-client-tls-certificate","namespace":"default","uid":"c0037b03-1ae2-4bc6-b9e0-f72ef03e5fbc","resourceVersion":"808","generation":1,"creationTimestamp":"2025-01-14T23:12:16Z","ownerReferences":[{"apiVersion":"operator.etcd.io/v1alpha1","kind":"EtcdCluster","name":"etcdcluster-sample","uid":"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"cert-manager.io/v1","time":"2025-01-14T23:12:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b\"}":{}}},"f:spec":{".":{},"f:dnsNames":{},"f:issuerRef":{".":{},"f:kind":{},"f:name":{}},"f:secretName":{}}}}]},"spec":{"dnsNames":["etcd.etcd-operator-system"],"secretName":"etcdcluster-sample-etcd-client-tls","issuerRef":{"name":"etcd-operator-selfsigned","kind":"ClusterIssuer"}},"status":{}},{"metadata":{"name":"etcdcluster-sample-etcd-peer-tls-certificate","namespace":"default","uid":"7fc98c31-548b-415c-a582-a8edaa14b9e0","resourceVersion":"809","generation":1,"creationTimestamp":"2025-01-14T23:12:16Z","ownerReferences":[{"apiVersion":"operator.etcd.io/v1alpha1","kind":"EtcdCluster","name":"etcdcluster-sample","uid":"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"cert-manager.io/v1","time":"2025-01-14T23:12:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b\"}":{}}},"f:spec":{".":{},"f:dnsNames":{},"f:issuerRef":{".":{},"f:kind":{},"f:name":{}},"f:secretName":{}}}}]},"spec":{"dnsNames":["etcd.etcd-operator-system"],"secretName":"etcdcluster-sample-etcd-peer-tls","issuerRef":{"name":"etcd-operator-selfsigned","kind":"ClusterIssuer"}},"status":{}},{"metadata":{"name":"etcdcluster-sample-etcd-server-tls-certificate","namespace":"default","uid":"a64b484a-7f3a-4552-b5db-d2b34f23ebfa","resourceVersion":"810","generation":1,"creationTimestamp":"2025-01-14T23:12:16Z","ownerReferences":[{"apiVersion":"operator.etcd.io/v1alpha1","kind":"EtcdCluster","name":"etcdcluster-sample","uid":"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"cert-manager.io/v1","time":"2025-01-14T23:12:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b\"}":{}}},"f:spec":{".":{},"f:dnsNames":{},"f:issuerRef":{".":{},"f:kind":{},"f:name":{}},"f:secretName":{}}}}]},"spec":{"dnsNames":["etcd.etcd-operator-system"],"secretName":"etcdcluster-sample-etcd-server-tls","issuerRef":{"name":"etcd-operator-selfsigned","kind":"ClusterIssuer"}},"status":{}}]}

@ahrtr
Copy link
Member

ahrtr commented Mar 12, 2025

Please rebase this PR and resolve the workflow failure, thx

@ArkaSaha30 ArkaSaha30 force-pushed the issue-cert branch 2 times, most recently from edc0294 to 3cdf84a Compare March 12, 2025 18:01
Signed-off-by: ArkaSaha30 <[email protected]>
This commit will add EnsureCertificateSecret which will
- get invoked only when `GetCertificateConfig` in reconcile logic returns isNotFoundError(i.e. the Certificate is not present)
- check if `issuerName`, `issuerKind` is correctly defined
- call `ValidateCertificateSecret` which will:
  - check if secret is already present
  - check if private key is present and matches with the certificate
- if `ValidateCertificateSecret` returns error, create certificate

Signed-off-by: ArkaSaha30 <[email protected]>

fix

Signed-off-by: ArkaSaha30 <[email protected]>
Signed-off-by: ArkaSaha30 <[email protected]>
@ArkaSaha30
Copy link
Contributor Author

Pre-requisites:

etcd-operator on  issue-cert [!?] via 🐳 colima via 🐹 v1.24.1 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get pods
NAME                   READY   STATUS    RESTARTS   AGE
etcdcluster-sample-0   1/1     Running   0          89s
etcdcluster-sample-1   1/1     Running   0          80s
etcdcluster-sample-2   1/1     Running   0          50s

etcd-operator on  issue-cert [!?] via 🐳 colima via 🐹 v1.24.1 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get certificates
NAME                              READY   SECRET                            AGE
etcdcluster-sample-0-peer-tls     True    etcdcluster-sample-0-peer-tls     87s
etcdcluster-sample-0-server-tls   True    etcdcluster-sample-0-server-tls   87s
etcdcluster-sample-1-peer-tls     True    etcdcluster-sample-1-peer-tls     57s
etcdcluster-sample-1-server-tls   True    etcdcluster-sample-1-server-tls   57s
etcdcluster-sample-2-peer-tls     True    etcdcluster-sample-2-peer-tls     27s
etcdcluster-sample-2-server-tls   True    etcdcluster-sample-2-server-tls   27s
etcdcluster-sample-client-tls     True    etcdcluster-sample-client-tls     96s

etcd-operator on  issue-cert [!?] via 🐳 colima via 🐹 v1.24.1 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get secrets
NAME                              TYPE                DATA   AGE
etcdcluster-sample-0-peer-tls     kubernetes.io/tls   3      93s
etcdcluster-sample-0-server-tls   kubernetes.io/tls   3      93s
etcdcluster-sample-1-peer-tls     kubernetes.io/tls   3      63s
etcdcluster-sample-1-server-tls   kubernetes.io/tls   3      63s
etcdcluster-sample-2-peer-tls     kubernetes.io/tls   3      33s
etcdcluster-sample-2-server-tls   kubernetes.io/tls   3      32s
etcdcluster-sample-client-tls     kubernetes.io/tls   3      102s

@ArkaSaha30 ArkaSaha30 requested review from ivanvc and ahrtr March 12, 2025 21:09
@ivanvc
Copy link
Member

ivanvc commented Mar 14, 2025

@ArkaSaha30, sorry but the tests are still failing.

@ArkaSaha30
Copy link
Contributor Author

@ArkaSaha30, sorry but the tests are still failing.

Yes, I am fixing them.

@ArkaSaha30 ArkaSaha30 force-pushed the issue-cert branch 2 times, most recently from 863bd09 to f20fd10 Compare March 16, 2025 19:40
@ArkaSaha30
Copy link
Contributor Author

cc @ahrtr @ivanvc new commits after the previous review, I will squash them once its good to merge:

Comment on lines +478 to +485
func (r *EtcdClusterReconciler) getStatefulSetPods(sts *appsv1.StatefulSet, ctx context.Context) (*corev1.PodList, error) {
podList := corev1.PodList{}
err := r.Client.List(ctx, &podList, client.InNamespace(sts.Namespace), client.MatchingLabels(sts.Spec.Selector.MatchLabels))
if err != nil {
return nil, err
}
return &podList, nil
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to reuse getStatefulSet instead of adding a new method?

func getStatefulSet(ctx context.Context, c client.Client, name, namespace string) (*appsv1.StatefulSet, error) {
sts := &appsv1.StatefulSet{}
err := c.Get(ctx, client.ObjectKey{Name: name, Namespace: namespace}, sts)
if err != nil {
return nil, err
}
return sts, nil
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you can. The StatefulSet API doesn't have a way to return the pods. I believe the way to get them is by listing pods using the selector, as Arka did it (line 480).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you can. The StatefulSet API doesn't have a way to return the pods. I believe the way to get them is by listing pods using the selector, as Arka did it (line 480).

+1
I was wondering if certificate creation can be independent of pod creation/ready. If so, we can form the certificate/secret name etcdcluster-sample-[n] where n=0, n< statefulset replicas.

@ahrtr
Copy link
Member

ahrtr commented Mar 17, 2025

cc @ivanvc @jmhbnz @hakman

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants