The Impact of Socket Buffers on TCP Performance

The socket buffer has a significant impact on TCP performance, and many articles suggest increasing the socket buffer size. But how much should we adjust it? This article skips the testing process and directly discusses the result—how to configure the sizes of tcp_rmem and tcp_wmem.

Checking the tcp_rmem and tcp_wmem files

The files contain three values:

1
2
3
4

cat /proc/sys/net/ipv4/tcp_rmem
4096 4096 16777216
cat /proc/sys/net/ipv4/tcp_wmem
4096 4096 16777216

The three values represent: min, default, and max.

• min: Defines the minimum size of the TCP socket buffer.
• default: Defines the default size.
• max: Defines the maximum size. For a TCP connection, the buffer size will vary between min and max. The primary factor influencing this variation is the current memory pressure. If setsockopt is used to set a specific buffer size, this value will be ignored, effectively disabling dynamic buffer adjustment.

Estimating the Bandwidth-Delay Product (BDP)

To calculate the correct receive buffer size, you can estimate the Bandwidth-Delay Product (BDP).

• If the send buffer exceeds the BDP, the excess cannot be effectively transmitted over the network, causing network congestion and packet loss.
• If the send buffer is smaller than the BDP, it won’t fully utilize the network transmission efficiency.

Thus, the send buffer size should ideally be close to the BDP.

So, how is the BDP value calculated?

It’s simply the Bandwidth x RTT = BDP.

Example Calculation

Let’s take a G-port at Alibaba Hong Kong as an example. First, we use ping to measure the RTT time:

1
2
3
4
5
6
7
8
9
10
11
12

PING 8.210.67.1 (8.210.67.1): 56 data bytes
64 bytes from 8.210.67.1: icmp_seq=0 ttl=48 time=42.566 ms
64 bytes from 8.210.67.1: icmp_seq=1 ttl=48 time=44.087 ms
64 bytes from 8.210.67.1: icmp_seq=2 ttl=48 time=43.466 ms
64 bytes from 8.210.67.1: icmp_seq=3 ttl=48 time=44.988 ms
64 bytes from 8.210.67.1: icmp_seq=4 ttl=48 time=41.912 ms
64 bytes from 8.210.67.1: icmp_seq=5 ttl=48 time=43.440 ms
64 bytes from 8.210.67.1: icmp_seq=6 ttl=48 time=42.994 ms
64 bytes from 8.210.67.1: icmp_seq=7 ttl=48 time=46.686 ms
64 bytes from 8.210.67.1: icmp_seq=8 ttl=48 time=42.769 ms
64 bytes from 8.210.67.1: icmp_seq=9 ttl=48 time=45.627 ms
64 bytes from 8.210.67.1: icmp_seq=10 ttl=48 time=41.631 ms

As we can see, the RTT time is less than 50ms, so:

BDP = (Bandwidth of G-port) 128 MBps x (50ms) 0.05 s = 6.4 MB

Next, we convert 6.4 MB into bytes:

6.4 MB = 6710886.4 bytes

This value corresponds to the tcp_wmem of the sender. The receiver’s tcp_rmem can be set to twice the sender’s value, but the TCP window scaling factor also affects this value:

• net.ipv4.tcp_adv_win_scale in recent kernels defaults to 1, meaning half of the buffer is used for application data, and the other half is the available receive window.
• In kernels before version 3.10, the default value was 2, meaning a quarter of the buffer was for application data, and three-quarters was for the available receive window.
• If net.ipv4.tcp_adv_win_scale is set to -2, three-quarters of the buffer is used for application data, and the available receive window is one-quarter.

If the default value is 1, the BDP value should be adjusted by multiplying by 4 and dividing by 2:

6.4 x 4 / 2 = 12.8 MB

However, it is recommended to set tcp_adv_win_scale to -2 to increase the tolerance space for configuring tcp_wmem. For example, for Alibaba Hong Kong:

6.4 x 4 / 1 = 25.6 MB

Configuration

You can directly write the following configuration to /etc/sysctl.conf:

1
2
3

net.ipv4.tcp_adv_win_scale = -2
net.ipv4.tcp_rmem = 8192 262144 53687091
net.ipv4.tcp_wmem = 4096 16384 26843545

For a West US G-port, with 128 MBps bandwidth, 160ms RTT, and a scaling factor of 4:

BDP = 128 MBps x 0.16 s x 4 = 81 MB