inter-mtls-setup issue

[michalschott]

I'm struggling bringing up cluster with inter-mtls-setup, followed this guide

Manifests:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: rabbitmq
  namespace: rabbitmq
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rabbitmq-selfsigned-ca
  namespace: rabbitmq
spec:
  isCA: true
  commonName: rabbitmq-selfsigned-ca
  secretName: rabbitmq-selfsigned-ca-root-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: rabbitmq
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: rabbitmq-ca-issuer
  namespace: rabbitmq
spec:
  ca:
    secretName: rabbitmq-selfsigned-ca-root-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rabbitmq-tls-secret
  namespace: rabbitmq
spec:
  dnsNames:
    - "*.rabbitmq-nodes.rabbitmq.svc.cluster.local"
  secretName: rabbitmq-tls-secret
  issuerRef:
    kind: Issuer
    name: rabbitmq
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rabbitmq-mtls-inter-node-nodes-tls
  namespace: rabbitmq
spec:
  secretName: rabbitmq-mtls-inter-node-nodes-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - RabbitMQ
  commonName: mtls-inter-node
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
  - rabbitmq-server-0.rabbitmq-server-nodes.rabbitmq
  - rabbitmq-server-1.rabbitmq-server-nodes.rabbitmq
  - rabbitmq-server-2.rabbitmq-server-nodes.rabbitmq
  issuerRef:
    name: rabbitmq-ca-issuer
    kind: Issuer
    group: cert-manager.io
---
apiVersion: v1
data:
  inter_node_tls.config: |
    [
      {server, [
        {cacertfile, "/etc/rabbitmq/certs/ca.crt"},
        {certfile,   "/etc/rabbitmq/certs/tls.crt"},
        {keyfile,    "/etc/rabbitmq/certs/tls.key"},
        {secure_renegotiate, true},
        {fail_if_no_peer_cert, true},
        {verify, verify_peer},
        {customize_hostname_check, [
          {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
        ]}
      ]},
      {client, [
        {cacertfile, "/etc/rabbitmq/certs/ca.crt"},
        {certfile,   "/etc/rabbitmq/certs/tls.crt"},
        {keyfile,    "/etc/rabbitmq/certs/tls.key"},
        {secure_renegotiate, true},
        {fail_if_no_peer_cert, true},
        {verify, verify_peer},
        {customize_hostname_check, [
          {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
        ]}
      ]}
    ].
kind: ConfigMap
metadata:
  name: mtls-inter-node-tls-config
  namespace: rabbitmq
---
---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  labels:
    app: rabbitmq
  name: rabbitmq
  namespace: rabbitmq
spec:
  replicas: 3
  tls:
    secretName: rabbitmq-tls-secret
    caSecretName: rabbitmq-selfsigned-ca-root-secret
  resources:
    requests:
      cpu: 2
      memory: 2Gi
    limits:
      cpu: 2
      memory: 2Gi
  rabbitmq:
    additionalConfig: |
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000
    envConfig: |
      SERVER_ADDITIONAL_ERL_ARGS="-proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter-node-tls.config"
      RABBITMQ_CTL_ERL_ARGS="-proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter-node-tls.config"
  persistence:
    storage: "10Gi"
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
              - rabbitmq
        topologyKey: kubernetes.io/hostname
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: rabbitmq
              imagePullPolicy: Always
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
                privileged: false
                readOnlyRootFilesystem: false
                runAsNonRoot: true
              volumeMounts:
              - mountPath: /etc/rabbitmq/certs
                name: mtls-inter-node-nodes-tls
              - mountPath: /etc/rabbitmq/inter-node-tls.config
                name: inter-node-config
                subPath: inter_node_tls.config
            volumes:
            - configMap:
                defaultMode: 420
                name: mtls-inter-node-tls-config
              name: inter-node-config
            - name: mtls-inter-node-nodes-tls
              secret:
                secretName: rabbitmq-mtls-inter-node-nodes-tls
                items:
                - key: ca.crt
                  mode: 416
                  path: ca.crt
                - key: tls.crt
                  mode: 416
                  path: tls.crt
                - key: tls.key
                  mode: 416
                  path: tls.key
            initContainers:
            - name: setup-container
              imagePullPolicy: Always
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
                privileged: false
                readOnlyRootFilesystem: true
                runAsNonRoot: true

Logs:

+ rabbitmq-server-2 › rabbitmq
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.02
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.02 Welcome to the Bitnami rabbitmq container
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.02 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.02 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.02
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.02 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.04 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.06 INFO  ==> Initializing RabbitMQ...
rabbitmq-server-2 rabbitmq rabbitmq 16:16:06.10 INFO  ==> Starting RabbitMQ in background...
rabbitmq-server-2 rabbitmq rabbitmq 16:18:54.62 ERROR ==> Couldn't start RabbitMQ in background.
rabbitmq-server-2 rabbitmq rabbitmq 16:18:54.62 INFO  ==> Stopping RabbitMQ...
- rabbitmq-server-2 › rabbitmq
+ rabbitmq-server-2 › rabbitmq
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.26
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.26 Welcome to the Bitnami rabbitmq container
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.26 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.27 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.27
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.27 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.29 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.31 INFO  ==> Initializing RabbitMQ...
rabbitmq-server-2 rabbitmq rabbitmq 16:18:58.34 INFO  ==> Starting RabbitMQ in background...

Once I comment out these cluster starts:

    envConfig: |
      SERVER_ADDITIONAL_ERL_ARGS="-proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter-node-tls.config"
      RABBITMQ_CTL_ERL_ARGS="-proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter-node-tls.config"

+ rabbitmq-server-2 › rabbitmq
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.85
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.85 Welcome to the Bitnami rabbitmq container
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.85 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.86 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.86
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.87 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.90 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.92 INFO  ==> Initializing RabbitMQ...
rabbitmq-server-2 rabbitmq rabbitmq 16:22:04.96 INFO  ==> Starting RabbitMQ in background...
rabbitmq-server-2 rabbitmq rabbitmq 16:22:19.01 INFO  ==> Stopping RabbitMQ...
rabbitmq-server-2 rabbitmq rabbitmq 16:22:22.44 INFO  ==> ** RabbitMQ setup finished! **

Not a clue where to even start debugging. Hopefully I've set certificates properly ;) Thanks for any advice.

Using rabbitmq-operator: docker.io/bitnami/rabbitmq-cluster-operator:1.10.0-scratch-r1

[lukebakken]

Could you please try setting up this cluster outside of K8S using the same configuration and certs? This way you can verify that the cert files are in place and readable by the rabbitmq user. You can also use openssl s_client to verify that you can make TLS connections to port 25672 - https://www.rabbitmq.com/troubleshooting-ssl.html

[michalschott]

I actually don't think this is permission issue as:

Defaulted container "rabbitmq" out of: rabbitmq, setup-container (init)
I have no name!@rabbitmq-server-1:/$ ls -la /etc/rabbitmq/certs/
total 0
drwxrwsrwt 3 root  999 140 Nov 12 08:14 .
drwxrwxr-x 1 root root 131 Nov 12 08:17 ..
drwxr-sr-x 2 root  999 100 Nov 12 08:14 ..2021_11_12_08_14_28.267187890
lrwxrwxrwx 1 root root  13 Nov 12 08:14 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root  31 Nov 12 08:14 ..data -> ..2021_11_12_08_14_28.267187890
lrwxrwxrwx 1 root root  14 Nov 12 08:14 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root root  14 Nov 12 08:14 tls.key -> ..data/tls.key
I have no name!@rabbitmq-server-1:/$ ls -la /etc/rabbitmq/certs/..data/
total 12
drwxr-sr-x 2 root 999  100 Nov 12 08:14 .
drwxrwsrwt 3 root 999  140 Nov 12 08:14 ..
-rw-r----- 1 root 999 1326 Nov 12 08:14 ca.crt
-rw-r----- 1 root 999 1326 Nov 12 08:14 tls.crt
-rw-r----- 1 root 999 1679 Nov 12 08:14 tls.key
I have no name!@rabbitmq-server-1:/$ id
uid=999 gid=0(root) groups=0(root),999
I have no name!@rabbitmq-server-1:/$ head /etc/rabbitmq/certs/..data/tls.crt
-----BEGIN CERTIFICATE-----
MIIDpzCCAo+gAwIBAgIQTwepZNLICJ96k2YFSUN6nzANBgkqhkiG9w0BAQsFADAt
I have no name!@rabbitmq-server-1:/$

[michalschott]

Turns out I had a typo there:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rabbitmq-mtls-inter-node-nodes-tls
  namespace: rabbitmq
spec:
...
  dnsNames:
  - rabbitmq-server-0.rabbitmq-server-nodes.rabbitmq
  - rabbitmq-server-1.rabbitmq-server-nodes.rabbitmq
  - rabbitmq-server-2.rabbitmq-server-nodes.rabbitmq
...

Corrected:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rabbitmq-mtls-inter-node-nodes-tls
  namespace: rabbitmq
spec:
...
  dnsNames:
  - rabbitmq-server-0.rabbitmq-nodes.rabbitmq
  - rabbitmq-server-1.rabbitmq-nodes.rabbitmq
  - rabbitmq-server-2.rabbitmq-nodes.rabbitmq
...

I also had to bump rabbitmq image from 3.8 to 3.9.