Kubespray - Kubectl localhost AND external load balancer

I have an external load balancer configured via loadbalancer_apiserver but on the kube_control_nodes kubectl tries to connect to localhost:8080 and fails. I assume this is because there is no internal load balancer? Is there anyway I could make the control nodes connect via localhost?