TCP over TCP is a bad idea (2000)
Using TCP over TCP can cause delays and connection failures due to differing timeout settings, leading to unnecessary retransmissions and a "meltdown" effect; UDP is a more effective alternative.
Read original article
Using TCP over TCP is problematic due to inherent inefficiencies in how TCP manages data transmission. When encapsulating protocols like PPP over a TCP connection, issues arise from TCP's retransmission algorithms, which are designed for reliability but can lead to significant delays and connection failures. TCP segments are sent as individual IP datagrams, each with sequence and acknowledgment numbers to manage data integrity. However, when one TCP connection is layered over another, the differing timeout settings can cause a cascading failure. If the lower layer experiences packet loss and increases its timeout, the upper layer may not receive timely acknowledgments, leading to unnecessary retransmissions and further congestion. This situation can result in a "meltdown" effect, where the upper layer TCP connection stalls due to excessive retransmissions, despite the lower layer guaranteeing delivery. Practical experiences, such as those from the CIPE project, have shown that using a datagram carrier like UDP is more effective, as it aligns better with the characteristics of IP, avoiding the complications that arise from TCP's reliability mechanisms.
- TCP over TCP can lead to significant delays and connection failures.
- Different timeout settings between stacked TCP connections can cause cascading failures.
- Unnecessary retransmissions occur when the upper layer TCP does not receive timely acknowledgments.
- The "meltdown" effect can stall upper layer connections due to excessive retransmissions.
- Using a datagram carrier like UDP is often more effective for encapsulating protocols.
Related
Timeliness without datagrams using QUIC
The debate between TCP and UDP for internet applications emphasizes reliability and timeliness. UDP suits real-time scenarios like video streaming, while QUIC with congestion control mechanisms ensures efficient media delivery.
Beyond bufferbloat: End-to-end congestion control cannot avoid latency spikes
End-to-end congestion control methods like TCP and QUIC face challenges in preventing latency spikes, especially in dynamic networks like Wi-Fi and 5G. Suggestions include anticipating capacity changes and prioritizing latency-sensitive traffic for a reliable low-latency internet.
Comparing TCP and QUIC (2022)
Geoff Huston compares TCP and QUIC protocols in the October 2022 ISP Column. QUIC is seen as a transformative protocol with enhanced privacy, speed, and flexibility, potentially replacing TCP on the Internet. QUIC offers improved performance for encrypted traffic and independent transport control for applications.
Linux Network Performance Optimization: Tips for Optimizing Network Latency
Optimizing network performance in Linux involves using eBPF and XDP for efficient packet processing, adjusting MTU and TCP settings, and utilizing tools like irqbalance and ethtool to enhance throughput and reduce latency.
Falsehoods programmers believe about TCP
The discussion highlights the effectiveness of NetworkManager versus `wpa_supplicant` for wireless connections, misconceptions about TCP reliability, and the complexities of achieving consensus over unreliable links, impacting issues like buffer bloat.
8 commentsOver the last month, I've been attempting to set up a fast Wireguard VPN tunnel between AWS and OVH. AWS killed all internet access on the instance with zero warning and sent us an email indicating that they suspected the instance was compromised and being used as part of a DDOS attack. OVH randomly performs "DDOS mitigation" anytime the tunnel is under any load. In both cases we were able to talk to someone and have the issue addressed, but I wanna stress: this is one stream between two IPs -- there's nothing that makes this anything close to looking like a DDOS. Even after getting everything properly blessed, OVH drops all UDP traffic over 1 Gbps. It took them a month of back-and-forth troubleshooting to tell us this.
The really terrible part is "TCP over TCP is bad" is now so prevalent there's basically no good VPN options for it if you need it. Wireguard won't do it directly, but there's hacks involving udp2raw. I tried it, and wasn't able to achieve more than 100 Mbps. OpenVPN can do it, but is single-threaded and won't reasonably do more than 1 Gbps without hardware acceleration, which didn't appear to work on EC2 instances. strongSwan cannot be configured to do unencapsulated ESP anymore -- they removed the option -- so it's UDP encapsulated only. Their reasoning is UDP is necessary for NAT traversal, and of course everybody needs that. It's also thread-per-SA so also not fast. The only solution I've found than can do something not UDP is Libreswan, which can still do unencapsulated ESP (IP Protocol 50) if you ask nicely. It's also thread-per-SA, but I've managed to wring 2 - 3 Gbps out of a single core after tinkering with the configuration.
For the love of all that's good in the world, just add performant TCP support to Wireguard. I do not care about what happens in non-optimal conditions.
/rant
Why TCP over TCP is a bad idea (2001) - https://news.ycombinator.com/item?id=25080693 - Nov 2020 (68 comments)
Why TCP Over TCP Is a Bad Idea (2001) - https://news.ycombinator.com/item?id=9281954 - March 2015 (43 comments)
Why TCP Over TCP Is A Bad Idea - https://news.ycombinator.com/item?id=2409090 - April 2011 (26 comments)
[0] https://web.archive.org/web/20000310230940/http://sites.inka...
Related
Timeliness without datagrams using QUIC
The debate between TCP and UDP for internet applications emphasizes reliability and timeliness. UDP suits real-time scenarios like video streaming, while QUIC with congestion control mechanisms ensures efficient media delivery.
Beyond bufferbloat: End-to-end congestion control cannot avoid latency spikes
End-to-end congestion control methods like TCP and QUIC face challenges in preventing latency spikes, especially in dynamic networks like Wi-Fi and 5G. Suggestions include anticipating capacity changes and prioritizing latency-sensitive traffic for a reliable low-latency internet.
Comparing TCP and QUIC (2022)
Geoff Huston compares TCP and QUIC protocols in the October 2022 ISP Column. QUIC is seen as a transformative protocol with enhanced privacy, speed, and flexibility, potentially replacing TCP on the Internet. QUIC offers improved performance for encrypted traffic and independent transport control for applications.
Linux Network Performance Optimization: Tips for Optimizing Network Latency
Optimizing network performance in Linux involves using eBPF and XDP for efficient packet processing, adjusting MTU and TCP settings, and utilizing tools like irqbalance and ethtool to enhance throughput and reduce latency.
Falsehoods programmers believe about TCP
The discussion highlights the effectiveness of NetworkManager versus `wpa_supplicant` for wireless connections, misconceptions about TCP reliability, and the complexities of achieving consensus over unreliable links, impacting issues like buffer bloat.