Whitelisting ips in azure aks haproxy controller

I am having a difficult time for some reason, getting expected behavior when whitelisting ips inside Ingress resource when haproxy controller is using in Azure AKS. I am following this documentation: https://github.com/haproxytech/kubernetes-ingress/blob/master/documentation/annotations.md#whitelist

My environment:

Azure AKS version: 1.17.11 Haproxy controller version: 1.8.3

Current behavior: When including whitelist annotation on Ingress seems to block any ip. If we have only blacklist in Ingress annotation this seems to have not block anything at all. When having both whitelist and blacklist annotation seems that whitelist takes precedence.

Here's example yaml of Ingress that I have:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: someIngressName
  namespace: someNamespaceName
  annotations:
      haproxy.org/whitelist: "x.x.x.x"
      ingress.class: "haproxy"
spec:
  rules:
  - host: some-host.com
    http:
      paths:
        - path: /
          backend:
            serviceName: someDeployment
            servicePort: 8080

Yes, I replaced actual values with fake values and x.x.x.x was replaced with actual ip. I am trying to whitelist ip of some Azure vms that we have.

I have tried different things like trying with different ips say what Azure reports as public ip may be different than what internet reports as its public ip and I tried both with no improved result. I tried internal ip also. I tried different prefixes before the whitelist part and still nothing.

Furthermore, I got a shell into one of the haproxy ingress pods and made sure that ip in the value gets applied to internal haproxy configuration. This much was true.

I am not sure what else I am doing wrong at this point. Could it be possible that aks version is too old for this functionality to work correctly?

Any help, suggestions, criticism is welcome.

Thanks.