There are many different approaches to this, but they all boil down to "divide and conquer" techniques.
These techniques would divide the network into "zones", where each "zone" is in charge of a subset of the data/network.
These "zones" could be further subdivided into smaller "zones" allowing for further scaling as needs change.
Each leaf node on this "zone" tree might hold a whole stack of machines (load balancers, caching (i.e., memcache), databases or API endpoint), depending on the requirements for the app.
Part of the reason AWS is popular is because it provides tools that do a lot of this, making scalability easier to manage.
This depends on your use-case, but it makes sense to divide zones geographically (consider how CDNs work).
For example, an application might be divided into a zone per continent, i.e., Asia, Europe, America, Africa, Oceania, etc'.
Each of these zones might be divided further by country / state.
Some states might require multiple (sub) zones due to heavy traffic (a third layer of routing / redirection).
Each of these sub-zones will hold it's own (sub) domain, a load balancer and a number of machines that manage both the connections and the data for that zone.
This example of geographical division could (theoretically) minimize network traffic and latency by geographically targeting DNS responses. This will allow (for example) for NYC traffic to remain in NYC unless "cross-border" communications are required.
"Cross-border" communication is sometimes delayed and aggregated to minimize both traffic and database transactions (something you wouldn't do for real-time applications).
This, obviously, has a number of complications. For example: what if a USA user is traveling in the UK? do we move his data to a different zone or do we route the connection to a USA server with all the latency that entails? ...
However, in the end, there's no choice but to "divide and conquer". Even databases can't hold all the data on a single machine and the data has to be divided into sub-sets on different databases.
Also, some data (like the zone assignments) is very hard to divide. Some data insists on being "global" (for example, how do we know which zone a user belongs to?)... This is often where hash values are used.
When using Hash values to calculate distribution, it makes it harder to later re-distribute the data. It's still possible, especially when using logarithmic zone division, but that just one approach out of a number of possible solutions.
