Feature: Cluster web configuration wizard (#3241)

* created wizard

* Create wizard form schema

* Wizard kafka cluster form (#3245)

* created wizard Kafka Cluster form

* created error message

Co-authored-by: davitbejanyan <[email protected]>

* Update schema.ts

* Wizard authentication (#3268)

* created authentication form

* changed SaslType.tsx switch case

* remove console.log

* commented unused variables

* auth validation

* auth Security Protocol

* changed schema.ts username, password

* added Delegation tokens validation schema

* changed auth form

---------

Co-authored-by: davitbejanyan <[email protected]>

* New Dashboard flow. Add an ability to configure clusters

* wizard kafka cluster validate (#3294)

* kafka cluster validate

* fixed  bootstrap server uncontrolled input  warning error

---------

Co-authored-by: davitbejanyan <[email protected]>

* Wizard schema registry (#3286)

* created schema registry

* unused variables

* Prevent Default on click

---------

Co-authored-by: davitbejanyan <[email protected]>

* feat: cleanup

* Application config API (#3242)

* wip

* wip

* wip

* wip

* OAuthProperties added to dynamic config api

* wip

* files upload endpoint added

* rbac conf api added

* rbac conf api improvements

* App configuration validation endpoint (#3264)

Co-authored-by: iliax <[email protected]>

---------

Co-authored-by: iliax <[email protected]>
Co-authored-by: Oleg Shur <[email protected]>

* add app config api client

* refactor cluster section

* refactor cluster section

* linting

* refactor Registry Form (#3311)

* refactor Registry Form

* refactor Registry

---------

Co-authored-by: davitbejanyan <[email protected]>

* auth form improvements

* refactoring

* linting

* file upload API changes

* Auth

* Start connecting to schema & payload

* Auth

* fileupload

* Wizard JMX Metrics form (#3303)

* created JMX Metrics form

* refactor JMXMetrics.tsx styles

* added cursor on checkbox, changed styles submit button

* refactor Metrics

* refactoring

* uncomment schema connect validation

---------

Co-authored-by: davitbejanyan <[email protected]>

* validate api

* refactor

* Wizard Kafka Connect form (#3300)

* created Kafka Connect form

* renaming functions and variables

* refactor

* changed button name

* refactoring kafka connect

* made handler function, reset replace with set value,

* refactoring

* uncomment schema metrics validation

---------

Co-authored-by: davitbejanyan <[email protected]>

* fixing AdminClient validation

* fixing AdminClient validation

* refactor kafka connect

* refactor metrics

* Per-cluster SSL verification settings (#3336)

* ssl configuration moved to app & cluster level

* documentations changes

* trust all removed, global app settings removed

* extracting ssl properties settings to SslPropertiesUtil

* wip

* documentation fix

---------

Co-authored-by: iliax <[email protected]>
Co-authored-by: Roman Zabaluev <[email protected]>

* SSL properties NPE fixes

* api integration

* custom fields for existing auth config

* OffsetsResetServiceTest fix

* cluster.properties structure flattening added

* kafka-ssl.yml: ssl properties moved to separate section, producer ssl properties copy added

* custom auth

* error messaging

* form submit

* feedback

* 1. defaulting metrics type to JMX
2. AdminClient id generation made uniq

* checkstyle fix

* checkstyle fix

* refactoring

* feedback

* feedback

* feedback

* feedback

* feedback

* feedback

* Wizard: Application info API (#3391)

* Application info API added, cluster features enum renamed to `ClusterFeature`

* show config for specific envs only

* refactor widget

* Cluster connection validation err msg improved

* KSQL DB section

* Refactor + deps upgrade

* experiment: get rid of babel

* BE validations refactoring

* Update kafka-ui.yaml

fixed to string type param

* fixes #3397

* linting

* #3399 adjust size of port input

* disable selects for disabled form

* Wizard: Keystore separation (#3425)

* wip

* wip

* compose fix

* dto structure fix

---------

Co-authored-by: iliax <[email protected]>

* dynamic ops enablement properties improvements

* own keystore for each section

* linting

* fix keystore submit

* fix keystore submit

* feedback

* feedback

* refactoring

* Connect config userName field renamed

* metrics configs mapping fix

* feedback

* Wizard: Jmx ssl (#3448)

JMX SSL implementation. Added ability to set specific ssl keystore for each cluster when connection to jmx endpoint.

* Review fixes

* upd compareVersionsOperation qase id

* add toBeAutomated into manual suite

* DYNAMIC_CONFIG_ENABLED property description added

* Resolve conflicts

* Fix issue with 400 error

* fix SR edit form

---------

Co-authored-by: davitbejanyan <[email protected]>
Co-authored-by: Alexander Krivonosov <[email protected]>
Co-authored-by: Oleg Shur <[email protected]>
Co-authored-by: Ilya Kuramshin <[email protected]>
Co-authored-by: iliax <[email protected]>
Co-authored-by: Roman Zabaluev <[email protected]>
Co-authored-by: bkhakimov <[email protected]>
Co-authored-by: Mgrdich <[email protected]>
Co-authored-by: VladSenyuta <[email protected]>

Author: 9 people

README.md

@@ -185,16 +185,17 @@ For example, if you want to use an environment variable to set the `name` parame
 |`KAFKA_CLUSTERS_0_KSQLDBSERVERAUTH_PASSWORD` 	| KSQL DB server's basic authentication password
 |`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTORELOCATION`   	|Path to the JKS keystore to communicate to KSQL DB
 |`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTOREPASSWORD`   	|Password of the JKS keystore for KSQL DB
-|`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTORELOCATION`   	|Path to the JKS truststore to communicate to KSQL DB
-|`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTOREPASSWORD`   	|Password of the JKS truststore for KSQL DB
 |`KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL` 	|Security protocol to connect to the brokers. For SSL connection use "SSL", for plaintext connection don't set this environment variable
 |`KAFKA_CLUSTERS_0_SCHEMAREGISTRY`   	|SchemaRegistry's address
 |`KAFKA_CLUSTERS_0_SCHEMAREGISTRYAUTH_USERNAME`   	|SchemaRegistry's basic authentication username
 |`KAFKA_CLUSTERS_0_SCHEMAREGISTRYAUTH_PASSWORD`   	|SchemaRegistry's basic authentication password
 |`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTORELOCATION`   	|Path to the JKS keystore to communicate to SchemaRegistry
 |`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTOREPASSWORD`   	|Password of the JKS keystore for SchemaRegistry
-|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTORELOCATION`   	|Path to the JKS truststore to communicate to SchemaRegistry
-|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTOREPASSWORD`   	|Password of the JKS truststore for SchemaRegistry
+|`KAFKA_CLUSTERS_0_METRICS_SSL`          |Enable SSL for Metrics (for PROMETHEUS metrics type). Default: false.
+|`KAFKA_CLUSTERS_0_METRICS_USERNAME` |Username for Metrics authentication
+|`KAFKA_CLUSTERS_0_METRICS_PASSWORD` |Password for Metrics authentication
+|`KAFKA_CLUSTERS_0_METRICS_KEYSTORELOCATION` |Path to the JKS keystore to communicate to metrics source (JMX/PROMETHEUS). For advanced setup, see `kafka-ui-jmx-secured.yml`
+|`KAFKA_CLUSTERS_0_METRICS_KEYSTOREPASSWORD` |Password of the JKS metrics keystore
 |`KAFKA_CLUSTERS_0_SCHEMANAMETEMPLATE` |How keys are saved to schemaRegistry
 |`KAFKA_CLUSTERS_0_METRICS_PORT`        	 |Open metrics port of a broker
 |`KAFKA_CLUSTERS_0_METRICS_TYPE`        	 |Type of metrics retriever to use. Valid values are JMX (default) or PROMETHEUS. If Prometheus, then metrics are read from prometheus-jmx-exporter instead of jmx
@@ -205,11 +206,9 @@ For example, if you want to use an environment variable to set the `name` parame
 |`KAFKA_CLUSTERS_0_KAFKACONNECT_0_PASSWORD`| Kafka Connect cluster's basic authentication password
 |`KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTORELOCATION`| Path to the JKS keystore to communicate to Kafka Connect
 |`KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTOREPASSWORD`| Password of the JKS keystore for Kafka Connect
-|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTORELOCATION`| Path to the JKS truststore to communicate to Kafka Connect
-|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTOREPASSWORD`| Password of the JKS truststore for Kafka Connect
-|`KAFKA_CLUSTERS_0_METRICS_SSL`          |Enable SSL for Metrics? `true` or `false`. For advanced setup, see `kafka-ui-jmx-secured.yml`
-|`KAFKA_CLUSTERS_0_METRICS_USERNAME` |Username for Metrics authentication
-|`KAFKA_CLUSTERS_0_METRICS_PASSWORD` |Password for Metrics authentication
 |`KAFKA_CLUSTERS_0_POLLING_THROTTLE_RATE` |Max traffic rate (bytes/sec) that kafka-ui allowed to reach when polling messages from the cluster. Default: 0 (not limited)
+|`KAFKA_CLUSTERS_0_SSL_TRUSTSTORELOCATION`| Path to the JKS truststore to communicate to Kafka Connect, SchemaRegistry, KSQL, Metrics
+|`KAFKA_CLUSTERS_0_SSL_TRUSTSTOREPASSWORD`| Password of the JKS truststore for Kafka Connect, SchemaRegistry, KSQL, Metrics
 |`TOPIC_RECREATE_DELAY_SECONDS` |Time delay between topic deletion and topic creation attempts for topic recreate functionality. Default: 1
 |`TOPIC_RECREATE_MAXRETRIES`  |Number of attempts of topic creation after topic deletion for topic recreate functionality. Default: 15
+|`DYNAMIC_CONFIG_ENABLED`|Allow to change application config in runtime. Default: false.

documentation/compose/jaas/client.properties

@@ -11,4 +11,4 @@ KafkaClient {
     user_admin="admin-secret";
 };
 
-Client {};
+Client {};

documentation/compose/jaas/kafka_connect.jaas

@@ -15,27 +15,25 @@ services:
       KAFKA_CLUSTERS_0_NAME: local
       KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
       KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka0:29092 # SSL LISTENER!
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /kafka.truststore.jks
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: secret
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION: /kafka.keystore.jks
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: secret
       KAFKA_CLUSTERS_0_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: '' # DISABLE COMMON NAME VERIFICATION
+
       KAFKA_CLUSTERS_0_SCHEMAREGISTRY: https://schemaregistry0:8085
       KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTORELOCATION: /kafka.keystore.jks
       KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTOREPASSWORD: "secret"
-      KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTORELOCATION: /kafka.truststore.jks
-      KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTOREPASSWORD: "secret"
+
       KAFKA_CLUSTERS_0_KSQLDBSERVER: https://ksqldb0:8088
       KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTORELOCATION: /kafka.keystore.jks
       KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTOREPASSWORD: "secret"
-      KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTORELOCATION: /kafka.truststore.jks
-      KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTOREPASSWORD: "secret"
+
       KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: local
       KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: https://kafka-connect0:8083
       KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTORELOCATION: /kafka.keystore.jks
       KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTOREPASSWORD: "secret"
-      KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTORELOCATION: /kafka.truststore.jks
-      KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTOREPASSWORD: "secret"
+
+      KAFKA_CLUSTERS_0_SSL_TRUSTSTORELOCATION: /kafka.truststore.jks
+      KAFKA_CLUSTERS_0_SSL_TRUSTSTOREPASSWORD: "secret"
+      DYNAMIC_CONFIG_ENABLED: 'true'  # not necessary for ssl, added for tests
+
     volumes:
       - ./ssl/kafka.truststore.jks:/kafka.truststore.jks
       - ./ssl/kafka.keystore.jks:/kafka.keystore.jks

documentation/compose/jaas/kafka_connect.password

@@ -11,11 +11,11 @@ services:
     environment:
       KAFKA_CLUSTERS_0_NAME: local
       KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
-      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092 # SSL LISTENER!
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /kafka.truststore.jks
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: secret
       KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION: /kafka.keystore.jks
-      KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: secret
+      KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: "secret"
+      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092 # SSL LISTENER!
+      KAFKA_CLUSTERS_0_SSL_TRUSTSTORELOCATION: /kafka.truststore.jks
+      KAFKA_CLUSTERS_0_SSL_TRUSTSTOREPASSWORD: "secret"
       KAFKA_CLUSTERS_0_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: '' # DISABLE COMMON NAME VERIFICATION
     volumes:
       - ./ssl/kafka.truststore.jks:/kafka.truststore.jks
@@ -60,4 +60,4 @@ services:
       - ./ssl/creds:/etc/kafka/secrets/creds
       - ./ssl/kafka.truststore.jks:/etc/kafka/secrets/kafka.truststore.jks
       - ./ssl/kafka.keystore.jks:/etc/kafka/secrets/kafka.keystore.jks
-    command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"
+    command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"

documentation/compose/jaas/kafka_server.conf

@@ -19,6 +19,7 @@ services:
       KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry0:8085
       KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: first
       KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: http://kafka-connect0:8083
+      DYNAMIC_CONFIG_ENABLED: 'true'  # not necessary, added for tests
 
   kafka0:
     image: confluentinc/cp-kafka:7.2.1.arm64