Load balancer scaling

I am quite new to the load balancing scenarios expecially in cloud. We are using load balancers to avoid load to a particular server and to avoid single point of failure . But then how does the load balancer itself scale . Isn't it a single point of failure ? Say, We have 1000 connections from client and we distribute it to to 4 servers using load balancer equally ,Then each server handle 250 connections . But the load balancer need to handle all 1000. Then, how do we avoid single point of failure or distribute load across machines?. I know that load balancer may not require to do as much processing required by the application server, since it does not do any functional processing. But still, there are modern load balancers which does lot of functionalities like SSL offload, intelligent routing etc. How does this work in Modern Highly scalable scenarios where we need to handle millions of connections?.