Support for running with Kubernetes as quorum manager instead of ZooKeeper

[soenkeliebau]

NiFi now supports running with Kubernetes as quorum manager instead of ZooKeeper. We should implement this and offer it as an option.

In principle this is just a few settings that could be overriden, but an entry needs to be written in an xml file, which we currently hard-code and effectively kill these overrides.

[nightkr]

Looks like this has two components:

1. Clustering coordination (https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-clustering)

This just requires us to set nifi.cluster.leader.election.implementation, and give NiFi permission to manage Lease objects:

---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  # [snip]
  nodes:
    configOverrides:
      nifi.properties:
        nifi.cluster.leader.election.implementation: KubernetesLeaderElectionManager
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nifi-extra
rules:
  # https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-clustering
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - create
      - get
      - update
      # undocumented but required
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nifi-extra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nifi-extra
subjects:
  - kind: ServiceAccount
    name: simple-nifi-serviceaccount

(Though obviously we should provide proper support, this is just illustrative of the scope of change we'd need to make.)

2. Cluster state storage (https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-configmap-cluster-state-provider)

This is configured by the XML file Sönke refers to in the OP. In theory the change scope should be similarly small, but this is indeed hard-coded and would require an operator change.