I am trying to connect some Nifi Nodes (1.13.2) to an external Zookeeper to be able to clusterize the service, but I am facing some errors while the NiFi tries to do the election using zookeeper. The error that I found on nifi-app.log was:
2021-07-12 12:32:24,673 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@448ca30f Connection State changed to RECONNECTED
2021-07-12 12:32:24,673 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2021-07-12 12:32:24,673 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
2021-07-12 12:32:24,774 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-07-12 12:32:24,774 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@448ca30f Connection State changed to SUSPENDED
2021-07-12 12:32:24,893 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
2021-07-12 12:32:24,893 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@448ca30f Connection State changed to RECONNECTED
2021-07-12 12:32:24,894 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-07-12 12:32:24,894 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@448ca30f Connection State changed to SUSPENDED
2021-07-12 12:32:24,894 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:647)
at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152)
at org.apache.curator.framework.imps.FindAndDeleteProtectedNodeInBackground$2.processResult(FindAndDeleteProtectedNodeInBackground.java:104)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:630)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
My nifi.properties configs are:
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-node01
nifi.cluster.node.protocol.port=9999
nifi.cluster.node.protocol.threads=20
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=10 sec
nifi.cluster.node.read.timeout=10 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=
nifi.zookeeper.connect.string=zookeeper-node0.fqdn:2181,zookeeper-node1.fqdn:2181,zookeeper-node2.fqdn:2181
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi-prod
nifi.zookeeper.client.secure=false
nifi.zookeeper.security.keystore=
nifi.zookeeper.security.keystoreType=
nifi.zookeeper.security.keystorePasswd=
nifi.zookeeper.security.truststore=
nifi.zookeeper.security.truststoreType=
nifi.zookeeper.security.truststorePasswd=
nifi.zookeeper.auth.type=default
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=
The state-management configs for zk-provider are:
<cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Connect String"> zookeeper-node0.fqdn:2181, zookeeper-node1.fqdn:2181, zookeeper-node2.fqdn:2181</property>
<property name="Root Node">/nifi-prod</property>
<property name="Session Timeout">10 seconds</property>
<property name="Access Control">Open</property>
</cluster-provider>
The zookeeper security configs on CM (CDH-6.3.3) are:
And the znode security on nifi-prod is:
[zk: zookeeper-node0:2181(CONNECTED) 1] getAcl /nifi-prod
'world,'anyone
: cdrwa
** I got this acl connecting to zookeeper using this command on the nifi node: /opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/zookeeper/bin/zkCli.sh -server zookeeper-node0:2181
Can this error occurs because I did not put the security configs? Even with the zkCli command without authentication working from the nifi node? The zookeeper log does not print any error