We had some hardware issue in VMWare and all VMs were shutdown and restarted.
For some storage reason the some of coreOS are starting with the wrong IP address 172.x.x.x The DHCP is on an RHEL configured to 10.128.0.0/14 and hardcoded with a specific 10.x Ip address using a mac address.
After a couple of days of debugging! it turns out that the helper Node with DHCP server also had a docker installed. We found a docker was serving 172.16 DHCP requests to some of your nodes
Docker uses the default 172.17.0.0/16 subnet for container networking
Stopping/disabling docker fixed the issue with nodes coming up on the wrong network.
Hope this helps someone in the future
Reference: Docker network explained
