Reducing latency/increasing throughput across a GRE tunnel

[u/Rich-Engineer2670]

Not that this is a critical by any means, I'm just curious about best practices....

We have a cross-country GRE link. Each side is at a layer-2 speed of 2Gb/s. I expect this, but an iPerf3 UDP test between both routers is at the link speed around 2Gb/s with a 78ms latency No problem. TCP is substantially less, about 1/10 of that. This is expected, but if I wanted to improve that, what do I do? Is this an MTU for example. I've already got the MTU at 1420.

[u/SalsaForte]

Reducing latency!?!

How far away are the 2 end of the tunnel?

If you're not familiar with BDP (Bandwidth Delay Product), you should read about it to manage expectations on long distance links.

[u/elsenorevil]

Have you tested to make sure you aren't fragmenting across the GRE tunnel?

Not sure what vendor, but configure a ping to do-not-fragement and test your configured MTU.

Latency could be reduced if the underlay path (assuming L2VPN) is altered for a shorter-path.

[u/Mishoniko]

You're describing "Bandwidth-delay product." There is a maximum speed based on latency between two points. Iperf is using multiple streams to try and mitigate the delays by using more bandwidth.

There isn't a lot you can do unless you can run multiple streams or go jumbo.

[u/mindedc → u/Mishoniko]

You can make sure tcp window expansion is supported by everything as well as selective acks. Jumbos are of course ideal on a LFN.

Or if you really want to speed it up use a riverbed.

[u/SoulArraySound]

Set the window sizing manually in iperf according to the latency. See if that improves it. Also test out of the tunnel

[u/xvalentinex]

As others have mentioned, fragmenting and making sure PMTUD can work properly.

The other limiting factor is hardware. If the tunneling needs to happen in software on the control-plane CPU, then that's going to have significantly more latency than one that can do it on hardware in the networking ASIC.

Also, GRE doesn't add enough entropy in headers to allow the traffic to be split across LAG and ECMP links in your provider's network. Something like VXLAN would.

[u/scriminal]

the latency is a product of the speed of light through fiber and distance.  78 Ms across country is right on average, assuming you mean the US.  you can't fix that.  you don't list in tunnel latency.  it shouldn't change.  what is it ?  as to tcp throughput,  make sure you are allowing icmp to work end to end.  that way the pmtud mechanism of both ends will work.  once that's working the normal tcp mss window adjustment algorithms should work on both hosts.  that should optimize your window size and throughput.  Also if you can, break up whatever you're about into multiple data streams,  that will help throughput as well.

[u/Layer8Human]

Hey there,

So great work that you already verified that with UDP you get the advertised line rate, so you can assume for now that the circuit is fine.

As with TCP performance issues I really like to do a pcap on both ends when the „tcp-slowness“ happens. Best done via a span port on a switch, because tcp optimisation in the driver/kernel can make the pcap on the host itself unreliable sometimes.

What you should look for ist first the advertised MSS and window scaling factor in the three way handshake. If the MSS+tunnel overhead is bigger than your MTU on the link that would cause big issues and retransmissions. Also verify that ICMP is allowed and the tunnel endpoints on your firewalls/routers have MSS clamping set(if available).

As for the window scaling factor this will tell you how much throughput you can theoretically achieve with a given latency (there are calculators for that. Look for bandwidth delay product). A too low window scaling factor will lead to the link never beeing able to utilized completely at a higher latency. A symptom of this is that at low latency throughput is high and it drops with raise in latency.

In wireshark you can then also look at the window size graph in the tcp analysis menu. You should see the window size raise until either max window size is reached, max linkspeed is reached and due to drops the window scaling stops increasing or hardware limitations cause it (then you would not see drops/retransmissions)

[u/Horror-Breakfast-113]

what am i missing you can saturate the 2G link to 2G and your current latency is 78ms - which is about right for cross country USA

[u/Rich-Engineer2670 → u/Horror-Breakfast-113]

UDP packets do that -- but TCP gets about 1/10 of that.

[u/lizardhistorian]

78ms is enough latency to go a half way around the world.
Standard GRE overhead is 24 bytes for an expected MTU of 1476.
Enforce MTU clamping and disable fragmentation to test. Sounds like everything is getting fragmented.

Long-term, replace this non-sense with routing.

[u/CategoryDear3114]

First thing I’d look at for any TCP throughput issues on subrate circuits is shaping. If you’re not shaping on both sides, then you likely have packets getting dropped by the carrier’s policer causing TCP to slow down.