Schema Registry SSL authentication with SchemaRegistryClient class

[andreyolv]

Description

When trying to authenticate in kafka using the SchemaRegistryClient class I get the error (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

I am using kafka in kubernetes through the Strimzi operator, and schema registry certificates created by the Strimzi KafkaUser.

How to reproduce

KafkaUser for Schema Registry:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: user-schema-registry
  namespace: kafka-strimzi
  labels:
    strimzi.io/cluster: foobar
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
    - resource:
        type: topic
        name: _schemas2
        patternType: literal
      operation: All
    - resource:
        type: group
        name: schema-registry
        patternType: prefix
      operation: All

Converting certificates do .jks

#!/bin/bash

set -e

KAFKA-CLUSTER=foobar
USERNAME=user-schema-registry

kubectl get secret $KAFKA-CLUSTER-cluster-ca-cert -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl get secret $KAFKA-CLUSTER-cluster-ca-cert -o=jsonpath='{.data.ca\.password}' | base64 -d > ca.password

kubectl get secret $USERNAME -o=jsonpath='{.data.user\.crt}' | base64 -d > user.crt
kubectl get secret $USERNAME -o=jsonpath='{.data.user\.key}' | base64 -d > user.key
kubectl get secret $USERNAME -o=jsonpath='{.data.user\.password}' | base64 -d > user.password
kubectl get secret $USERNAME -o=jsonpath='{.data.user\.p12}' | base64 -d > user.p12

keytool -import -trustcacerts \
    -file ca.crt \
    -storepass $(cat ca.password) \
    -keystore truststore.jks \
    -noprompt

echo 'Truststore created!'

keytool -importkeystore \
    -srckeystore user.p12 \
    -srcstoretype PKCS12 \
    -srcstorepass $(cat user.password) \
    -destkeystore $USERNAME-keystore.jks \
    -deststorepass $(cat user.password)

echo 'Keystore created!'

rm user.p12

Configuring Schema Registry:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: schema-registry-cp-schema-registry2
  namespace: kafka-strimzi
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: schema-registry-cp-schema-registry2
  template:
    metadata:
      labels:
        app: schema-registry-cp-schema-registry2
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "5556"
        cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    spec:
      imagePullSecrets:
      - name: acr-secret
      containers:
      - name: schema-registry-cp-schema-registry2
        image: cp-schema-registry:7.2.1
        ports:
        - name: schema-registry
          containerPort: 8081
          protocol: TCP
        - name: jmx
          containerPort: 5555
        env:
        # Kafka TLS
        - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
          value: SSL://foobar-kafka-bootstrap:9093
        - name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL
          value: SSL
        - name: SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_LOCATION
          value: /ssl/truststore.jks
        - name: SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_PASSWORD
          valueFrom:
           secretKeyRef:
             name: user-schema-registry-jks-passwords
             key: TRUSTSTORE_PASSWORD
        - name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_LOCATION
          value: /ssl/user-schema-registry-keystore.jks
        - name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_PASSWORD
          valueFrom:
           secretKeyRef:
             name: user-schema-registry-jks-passwords
             key: KEYSTORE_PASSWORD
        - name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEY_PASSWORD
          valueFrom:
           secretKeyRef:
             name: user-schema-registry-jks-passwords
             key: KEYSTORE_PASSWORD
        - name: SCHEMA_REGISTRY_HOST_NAME
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

         Schema Registry TLS  
        - name: SCHEMA_REGISTRY_INTER_INSTANCE_PROTOCOL
          value: https
        - name: SCHEMA_REGISTRY_LISTENERS
          value: https://0.0.0.0:8081
        - name: SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION
          value: /ssl/truststore.jks
        - name: SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD
          valueFrom:
           secretKeyRef:
             name: user-schema-registry-jks-passwords
             key: TRUSTSTORE_PASSWORD
        - name: SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION
          value: /ssl/user-schema-registry-keystore.jks
        - name: SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD
          valueFrom:
           secretKeyRef:
             name: user-schema-registry-jks-passwords
             key: KEYSTORE_PASSWORD
        - name: SCHEMA_REGISTRY_SSL_KEY_PASSWORD
          valueFrom:
           secretKeyRef:
             name: user-schema-registry-jks-passwords
             key: KEYSTORE_PASSWORD

        # Others configs
        - name: SCHEMA_REGISTRY_LOG4J_ROOT_LOGLEVEL
          value: INFO
        - name: SCHEMA_REGISTRY_KAFKASTORE_TOPIC
          value: _schemas2
        - name: SCHEMA_REGISTRY_HEAP_OPTS
          value: "-Xms512M -Xmx512M"
        - name: JMX_PORT
          value: "5555"
        resources:
          requests:
            cpu: "0.1"
            memory: 300Mi
          limits:
            cpu: "0.5"
            memory: 1000Mi
        volumeMounts:
        - name: jks-files
          mountPath: /ssl
      - name: prometheus-jmx-exporter
        image: kafka-prometheus-jmx-exporter
        command:
        - java
        - -XX:+UnlockExperimentalVMOptions
        - -XX:+UseContainerSupport
        - -XX:MaxRAMFraction=1
        - -XshowSettings:vm
        - -jar
        - jmx_prometheus_httpserver.jar
        - "5556"
        - /etc/jmx-schema-registry/jmx-schema-registry-prometheus.yml
        ports:
        - containerPort: 5556
        resources:
          requests:
            cpu: "0.1"
            memory: 300Mi
          limits:
            cpu: "0.5"
            memory: 1000Mi
        volumeMounts:
        - name: jmx-config
          mountPath: /etc/jmx-schema-registry
      volumes:
      - name: jks-files
        configMap:
          name: user-schema-registry-jks-files
      - name: jmx-config
        configMap:
          name: schema-registry-jmx-configmap2

Try to connecto do schema registry using SchemaRegistryClient class with SSL:

import json
from confluent_kafka.schema_registry import Schema
from confluent_kafka.schema_registry import SchemaRegistryClient

schema_registry_conf = {
    'url': 'https://schema-registry-cp-schema-registry2.kafka-strimzi.svc.cluster.local',
    'ssl.ca.location': 'certs-sr/ca.crt',
    'ssl.certificate.location': 'certs-sr/user.crt',
    'ssl.key.location': 'certs-sr/user.key',
}

schema_registry_client = SchemaRegistryClient(schema_registry_conf)

schema_subject = 'user-schema3'
schema_type = 'AVRO'
schema_str = """
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "user_id", "type": "int"},
    {"name": "user_name", "type": "string"},
    {"name": "email", "type": "string"},
    {"name": "is_active", "type": "boolean"}
  ]
}
"""
schema = Schema(schema_str, schema_type)
schema_id = schema_registry_client.register_schema(schema_subject, schema)

Error:

SSLError: HTTPSConnectionPool(host='schema-registry-cp-schema-registry2.kafka-strimzi.svc.cluster.local', port=8081): Max retries exceeded with url: /subjects/user-schema3/versions?normalize=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

Versions:

• confluent-kafka-python version: 2.5.3
• Strimzi operator version: 0.43.0
• Apache Kafka broker version: 3.8.0

[allamiro]

If your Schema Registry certificate is signed by an intermediate CA, ensure that intermediate CA’s certificate is included along with the root CA in the trust stores or certificates ca.crt provided .