Keycloak Docker log contains IP of Reverse Proxy

I have the following docker-compose file:

version: '2.3'

services:
  postgres:
    image: postgres
    restart: unless-stopped
    volumes:
      - ./postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=keycloak
      - POSTGRES_USER=keycloak
      - POSTGRES_PASSWORD=password
  keycloak:
    image: jboss/keycloak
    restart: unless-stopped
    command: -b 0.0.0.0 -Djboss.bind.address.private=127.0.0.1
    environment:
      - DB_VENDOR=POSTGRES
      - DB_ADDR=postgres
      - DB_DATABASE=keycloak
      - DB_USER=keycloak
      - DB_SCHEMA=public
      - DB_PASSWORD=password
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=password
      - PROXY_ADDRESS_FORWARDING=true
    ports:
      - "2600:8443"
    depends_on:
      - postgres

networks:
  default:
    driver: bridge
    enable_ipv6: true
    ipam:
      config:
        - subnet: "172.26.0.0/24"
        - subnet: "fc00:2600::/96"

And the following nginx config

server {
  listen 80;
  server_name sso.domain.de;

  location / {
    return 301 https://sso.domain.de$request_uri;
  }

  location /.well-known/acme-challenge/ {
    root /var/www/certbot/sso.domain.de/;
  }
}

server {
  listen 443 ssl;
  server_name sso.domain.de;

  ssl_certificate /etc/letsencrypt/live/sso.domain.de/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/sso.domain.de/privkey.pem;

  include /etc/nginx/ssl/options-ssl-nginx.conf;
  ssl_dhparam /etc/nginx/ssl/ssl-dhparams.pem;

  client_max_body_size 200M;

  location / {
    proxy_pass https://keycloak-server:2600/;

    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains;";

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Nginx-Proxy true;

    proxy_redirect off;
  }
}

The config works fine for normal operations. To find wrong login tries with fail2ban I need the correct IP address of the client. But what I get in the Keycloak Log is still the IP of the loadbalancer (it seems to ignore PROXY_ADDRESS_FORWARDING=true)

keycloak_1  | 14:46:54,726 WARN  [org.keycloak.events] (default task-3) type=LOGIN_ERROR, realmId=myrealm, clientId=frontend, userId=d25b66b4-5b9e-45bd-94dc-14a447f47cc2, ipAddress=loadbalancer-ip, error=invalid_user_credentials, auth_method=openid-connect, auth_type=code, redirect_uri=https://frontend.domain.de/sso/login, code_id=07fca7fd-4f5b-415a-b7e1-a4d358693a88, username=badman, authSessionParentId=07fca7fd-4f5b-415a-b7e1-a4d358693a88, authSessionTabId=EIUTSY8Mxwk
keycloak_1  | 14:46:54,732 WARN  [org.keycloak.services] (Brute Force Protector) KC-SERVICES0053: login failure for user d25b66b4-5b9e-45bd-94dc-14a447f47cc2 from ip loadbalancer-ip

Any ideas?