Musing:How Long Does It Take to Transfer Data ? Rules of Thumb

When it comes trans­fer­ring large files or data sets ( say big­ger than 10 Gigabytes), many people for­get that the trans­fer time becomes sig­ni­fic­ant.

Transferring of a 20 GB file

Let’s say that you are trans­fer­ring a 20 Gigabyte file over an Internet or intersite con­nec­tion with band­width that is less than Gigabit. (If you are meas­ur­ing inter­act­ive traffic flows you need another technique).

20 000 Megabytes * 8 bits per byte * 1 000 000 bits per Megabyte= 160 000 000 000 bits

When we pack this raw data into a TCP/​IP packet (e,.g L3 packet head­ers and L2 fram­ing) the data set will expand by about 40% (that is my rule of thumb).

160 000 000 000 bits plus 40% encod­ing over­head = 224 000 000 000 bits of net­work data.

If you can trans­fer this data (without encryp­tion) at, say, 10 mega­bits per second:

224 000 000 000 bits /​ 10 000 000 bits per second = 22400 seconds = 373 minutes.

So we can expect the trans­fer at 10 Mbps to take around six hours (give or take a bit). Transferring the file at 100Mbps will take about 35 minutes.

Impact of Latency

This cal­cu­la­tion is a best case and assumes that:

• there is no packet loss, and thus no retransmissions
• that there is no sig­ni­fic­ant latency in the trans­mis­sion path

Application Latency is the time taken for a data to pass from the applic­a­tion, propag­ate from A to B, and then be acknow­ledged from B to A, and be handed to Application to release the API data buffer.

Latency Rules of Thumb

My rule of thumb for cir­cuits between 1 to 10 Megabits per second is to reduce the through­put by another 10 to 15% for intra-​​continental traffic, and 20 to 25% for Major eco­nom­ies in the north­ern hemi­sphere( UK, Europe, US), add more for the old Eastern Bloc and emer­ging coun­tries (such as Turkey). Think hard and make your own choices for longer dis­tances such as Australia, Japan, China etc.

Encryption Inflation

Remember that adding encryp­tion (IPSec or SSL) will fur­ther inflate the amount of trans­ferred data. The actual impact will depend a lot on aver­age packet size. IPSec typ­ic­ally add 40 bytes to the header, which has less impact for a large data trans­fer with an MTU of 1350 bytes that for an inter­act­ive ses­sion exchan­ging pack­ets with an MTU of of 400 bytes.

Often for­got­ten is the encryp­tion and decryp­tion delay that makes teh trans­fer more latent.

Rule of Thumb: add another 10 to 15% if you choose to encrypt the transfer.

Worst Case

So com­ing back to our ori­ginal plan.

224 000 000 000 bits /​ 10 000 000 bits per second = 22400 seconds = 373 minutes.

Add 20% for the UK to US data transfer.

Add 10% for Encrypting into IPsec tunnel.

373 minutes * 20% * 10% = 492 minutes.

Now our chart looks more like this:

Wrapping it up

These rules of thumb are import­ant because these are what you use in a design meet­ing before the idea turns into a pro­ject. You can save days of work by know­ing rough num­bers in advance. When the pony-​​tailed Unix Admin at the meet­ing cas­u­ally throws out “and trans­fer the 20GB tile images using the VPN to China” you can do some quick num­bers and instantly ask for another £250K for WAN upgrades (which of course will put that bright idea into the bin).

Near enough is usu­ally good enough — Etherealmind.

Postscript

Not that these cal­cu­la­tions are for file trans­fers which is easier to define. Trying cal­cu­late 20 GB of inter­act­ive traffic is much more dif­fi­cult because the under­ly­ing con­di­tion is inde­term­in­ate. Beware.

The best pos­sible net­work per­form­ance is achieved when the net­work pipe between the sender and the receiver is kept full of data, that is, the ‘in-​​flight bytes’ is set to the max­imum that a sender and receiver can buf­fer before an acknow­ledge­ment is issued. This ‘should’ occur when trans­fer­ring a large file and thus issues such as BDP and TCP Windowing are not really relevant.

Please rate this post:

(No Ratings Yet)

 Loading ...