NiFi Cluster Docker Load Balancing configuration

user_9908137 0
Viewed 231

I would like to configure Load Balancing in docker-compose.yml file for NiFi cluster deployed via Docker containers. Current docker-compose parameters for LB are as follows (for each of three NiFi nodes):

      # load balancing
      - NIFI_CLUSTER_LOAD_BALANCE_PORT=6342
      - NIFI_CLUSTER_LOAD_BALANCE_HOST=node.name
      - NIFI_CLUSTER_LOAD_BALANCE_CONNECTIONS_PER_NODE=4
      - NIFI_CLUSTER_LOAD_BALANCE_MAX_THREADS=8

But, when I try to use load balancing in queues, I can choose all the parameters there, and do not have any error, but LB is not working, everything is done on the primary node (because I used GetSFTP on the primary node only, but want to then process data on all 3 nodes). Also, NiFi cluster is configured to work with SSL.

Thanks in advance!

apache-nifi docker-compose docker load-balancing
1 Answers

I had opened load balance port on my docker file. Also I had to specify hostname for each node's compose file

here is my docker file for basic clustering

version: "3.3"
services:
  nifi_service:
    container_name: "nifi_service"
    image: "apache/nifi:1.11.4"
    hostname: "APPTHLP7"
    environment:
        - TZ=Europe/Istanbul
        - NIFI_CLUSTER_IS_NODE=true
        - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8088
        - NIFI_ZK_CONNECT_STRING=172.16.2.238:2181,172.16.2.240:2181,172.16.2.241:2181
    ports:
        - "8080:8080"
        - "8088:8088"
        - "6342:6342" 
    volumes:
        - /home/my/nifi-conf:/opt/nifi/nifi-current/conf
    networks:
      - my_network
    restart: unless-stopped    
networks:
  my_network:
    external: true

please not that you have to configure load balance strategy on the downstream connection in your flow.

user_154794430
Related