How to expose kafka using istio ingress?

Viewed 675

I use istio-ingress gateway and virtualservice to expose different microservices. So far all of them have been http services, so it was straight-forward to follow istio's documentation.

But with kafka I am facing some issues. I am using bitnami/kafka helm chart for kafka installation. Here's the values.yaml used for it:

global:
  storageClass: "kafka-sc"

replicaCount: 3
deleteTopicEnable: true

resources:
  requests:
    memory: 1024Mi
    cpu: 500m
  limits:
    memory: 2048Mi
    cpu: 1000m

zookeeper:
  replicaCount: 3
  resources:
    requests:
      memory: 1024Mi
      cpu: 500m
    limits:
      memory: 2048Mi
      cpu: 1000m

This deployment exposes kafka on this endpoint: my-kafka.kafka.svc.cluster.local:9092

I want this endpoint to be accessible via internet using ingress controller. Therefore, I applied following kubernetes manifests -->

A. kafka-ingress-gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kafka-ingress-gateway
  namespace: kafka
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 9092
      name: tcp
      protocol: TCP
    hosts:
    - "kafka.<public_domain>"

B. kafka-ingress-virtualservice.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kafka-ingress-virtualservice
  namespace: kafka
spec:
  hosts:
  - "kafka.<public_domain>"
  gateways:
  - kafka/kafka-ingress-gateway
  tcp:
  - match:
    - port: 9092
    route:
    - destination:
        host: my-kafka.kafka.svc.cluster.local
        port:
          number: 9092

To verify whether this works, I am using following approach:

  1. Create a kafka-client pod and login to it in two different terminals
  2. In first terminal, I produce in a topic called test using this command: kafka-console-producer.sh --broker-list my-kafka-0.my-kafka-headless.kafka.svc.cluster.local:9092 --topic test
  3. In second terminal, I consume in test topic using this command.

In here, this works: kafka-console-consumer.sh --bootstrap-server my-kafka.kafka.svc.cluster.local:9092 --topic test --from-beginning

This does not work: kafka-console-consumer.sh --bootstrap-server kafka.<public_domain>:9092 --topic test --from-beginning

I am getting this error: WARN [Consumer clientId=consumer-console-consumer-89304-1, groupId=console-consumer-89304] Bootstrap broker kafka.<public_domain>:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

I am new to kafka, so not sure what else is required to expose the consumer endpoint. From similar questions on stackoverflow, I noticed we are supposed to define "advertisedListeners" in kafka config, but not sure what value to put there.

Please let me know if I am missing any details here.

2 Answers

edit your istio-ingressgateway and add 9092 for tcp port

kubectl edit svc -nistio-system istio-ingressgateway

add

- name: kafka-broker
  port: 9092
  protocol: TCP
  targetPort: 9092

You would need to get the load balancer DNS name or IP address from the Kubernetes service for instance, you should find it in the status section there. Then use it to configure the advertised address in the Kafka broker you want to expose. I'm not sure if you need to expose a bootstrap-server endpoint or not though.

Related