Parallel file download

user_651174 0
Viewed 558

This is a basic question but I'd like to have someone with some more networking experience provide a more comprehensive answer.

Let's say I have 3 files on an external server that are 1GB each. And to download them I would do:

$ wget https://server.com/file1.mov

In terms of doing the three items in parallel (in three separate tabs/shells/threads, for example), or in doing then in series, such as:

$  wget https://server.com/file1.mov \
&& wget https://server.com/file2.mov \
&& wget https://server.com/file3.mov \

Under the following circumstances:

  1. I have a very fast local download connection and the server is much slower (for example, let's say it takes the server 15s to generate each file).
  2. My local download connection is slow and the server is much faster with upload bandwidth.
  3. The two are relatively the same.

For the first case, it seems obvious that we'd want to use parallel downloads, if there is going to be what amounts to a fixed overhead cost on the external server, but what about for the other two scenarios, why would one be better than the other?

wget bandwidth networking download performance
3 Answers

For all three cases the actual results depend on a number of limiting factors. You’ve listed download and upload bandwidth, but the bottleneck can be also in disk I/O, CPU and RAM as well as data transfer protocol and number of others.

In general segmented file transfer is a preferable way to obtain data as it reduces an impact of TCP Congestion Control (assume we use TCP protocol) and heterogeneous environment. As described in this “Applied Techniques for High Bandwidth Data Transfers across Wide Area Networks” (Jason Lee, Dan Gunter, Brian Tierney) paper:

“TCP probes the available bandwidth of the connection by continuously increasing the window size until a packet is lost, at which point it cuts the window in half and starts “ramping up” the connection again. The higher the bandwidth-delay product, the longer this ramp up will take, and less of the available bandwidth will be used during its duration.”

And

“In order to improve this situation where the network becomes the bottleneck, parallel streams can be used. This technique is implemented by dividing the data to be transferred into N portions and transferring each portion with a separate TCP connection. The effect of N parallel streams is to reduce the bandwidth-delay product experienced by a single stream by a factor of N because they all share the single-stream bandwidth (u). Random packet losses for reasonable values of q (<0.001) will usually occur in one stream at a time, therefore their effect on the aggregate throughput will be reduced by a factor of N. When competing with connections over a congested link, each of the parallel streams will be less likely to be selected for having their packets dropped, and therefore the aggregate amount of potential bandwidth which must go through premature congestion avoidance or slow start is reduced.”

Worth mentioning that congestion algorithms play a significant role in bandwidth allocation but they still need to be set according to the network “class”. Broadband, satellite, 3G, WiFi - they all have features dictated by the physical environment and CWND implementations perform differently.

Another thing to consider is a behavior of parallel vs concurrent data transfer in congested networks. Theoretically, the more data you transfer the higher chance to clog the network and activate policies on the ISP side that will start dropping or shaping connections. However, even in this case a probability to exchange data is higher with several small parallel links rather than with one big connection.

Again, this is a very generic explanation and connection throughput may vary significantly depending upon a variety of factors. It’s also possible to find yourself in a situation when a single connection will perform on par with multi-threaded and at the same time won’t require extra efforts and code for implementation.

user_129909480

tl;dr : TCP congestion control

Explanation

The real answer is that it's almost doesn't matter!

Once you create a TCP Session, TCP will handle your bandwidth and will try to reach maximum speed using the congestion control mechanism:

Transmission Control Protocol (TCP) uses a network congestion-avoidance algorithm that includes various aspects of an additive increase/multiplicative decrease (AIMD) scheme, along with other schemes including slow start and congestion window, to achieve congestion avoidance.

Congestion avoidance can be both "real" congestion between two packets on the line- or can also be a packet loss since your CPU/NIC/something else couldn't handle the packet bandwidth.

Why is it ALMOST doesn't matter?

That's because in some scenarios we can have a congestion because of two parallel sessions.

Here is a stupid example that will explain why it can matter:

Computer with bandwidth of 8bit/s NIC ability trying to create a parallel session. Once he will send "SYN" in parallel there will be a congestion. Since 8bit is the minimum bits this specific computer can send in a packet the congestion will not allow the computer to use the internet.

user_62617170

Depending on your TCP windowing function / method it can be faster to always use parallel downloads.

also if some intermediate node (access point / switch / router / frirewall / proxy) goes to saturation. it has to drop some packets.

The simplest way they do this is by tail dropping. but firewalls and routers also can have complexer traffic shapers, e.g.

if you try to parallel download, and the bandwidth is shared among other traffic, you generate more packets, or more connections (for per connection traffic shaping), so you can gain some speed

user_10818210
Related