More and more network card devices support the offload feature to improve network transmission and reception performance. Offload is to put some data packet processing originally done by the operating system (such as fragmentation, reassembly, etc.) into the network card hardware, reducing system CPU consumption while improving processing performance; including LSO/LRO, GSO/GRO, TSO/UFO, etc.
The Linux system and surrounding hardware have some solutions for network acceleration, and the main idea is still to reduce the time the CPU spends processing network data packets. The core problem of network transmission is that the time the CPU can spend processing each network packet is shortened, so reducing the time the CPU spends processing network data packets is the most straightforward solution. With the development of network technologies such as SDN/DPDK, there are also various other solutions for acceleration and performance enhancement at the network level, such as large-page Cache caching and bypassing memory, etc. (this part requires reading relevant DPDK books);
In the operation and maintenance process, receiving offload feature configuration in cloud platform virtual network mode may also produce some unknown problems, and it is necessary to have a clear understanding of the role and characteristics of related features to quickly locate the problem;
Related documentation:
《Segmentation Offloads in the Linux Networking Stack》
《Checksum Offloads in the Linux Networking Stack》
《Common Network Acceleration Technologies》
《Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in VMware Environment》
《Packet fragmentation and segmentation offload in UDP and VXLAN》
Practical operation
The Linux server can use the command ethtool to view network card configuration options.
View configuration items: ethtool -k [network card device name]
Note that if the physical machine network card has a bond group, when checking and modifying the configuration, you need to perform operations on the bond, eth0, eth1, etc.;
View configuration
ethtool -k eth0 | grep tcp-segmentation-offload # tso
ethtool -k eth0 | grep generic-segmentation-offload # gso
ethtool -k eth0 | grep udp-fragmentation-offload # ufo
ethtool -k eth0 | grep generic-receive-offload # gro
ethtool -k eth0 | grep large-receive-offload # lroModify configuration
ethtool -K eth1 gro off
ethtool -K eth1 lro on
ethtool -K eth1 tso on
Introduction to TCP/IP protocol stack
When a user needs to send data over the network, the user actually completes this task through an application program. The application program writes data to a file descriptor that describes the connection to the peer.
Then, the TCP/IP protocol stack in the operating system kernel receives the data from the file descriptor, completes the TCP segmentation (if it is a TCP connection), adds TCP, IP, and Ethernet headers. When adding these headers, some content calculations are involved, such as checksums and sequence numbers.
Finally, the operating system kernel informs the network card driver to send the data, and the data is sent to the peer through the network card. The network card adds some other data to ensure the reliability of the transmission. Finally, the network data is sent out from the network card through the network cable (if it is a wired connection) and reaches the peer through various network forwarding devices.
The receiving end, which is the peer of the network data, has a similar process, but in the reverse direction. The network card receives the data from the network cable, notifies the system kernel to retrieve the data, and the TCP/IP protocol stack in the system kernel completes the verification, removes the TCP, IP, and Ethernet headers, and reassembles the data. Finally, the complete data is passed to the application program or the final user. The user program still reads the data through a file descriptor.
Taking Linux as an example, the traditional network device driver package handles the following actions:
- The data packet arrives at the network card device.
- The network card device performs DMA operations according to the configuration.
- The network card sends an interrupt to wake up the processor.
- The driver software fills in the read and write buffer zone data structure.
- The data packet reaches the kernel protocol stack and performs high-level processing.
- If the final application is in user mode, the data is moved from the kernel to the user mode.
- If the final application is in kernel mode, it continues to be processed in the kernel.
So, the entire process of network transmission within an operating system can be divided into three parts:
- User area: applications send and receive data
- Kernel area: TCP/IP protocol stack and system kernel encapsulate and decapsulate user data
- Device area: network cards actually send and receive network data
From the previous description, it can be seen that the work of User area and Device area is relatively simple, while the handling of complex network protocols is mainly done in Kernel area. Any processing in Kernel area requires CPU completion. Obviously, if more data needs to be transmitted in a unit of time, the CPU needs to perform more calculations.
Network bandwidth has increased significantly in recent years, with Ethernet evolving from 10M to 100G, a ten-thousand-fold increase. Although CPUs have also developed significantly, the frequency of single-core CPUs has not increased as much. Some people may say that the number of CPU cores has increased significantly, but assigning a network data stream to multiple CPU cores for processing is challenging in itself. On the other hand, the tasks that computers need to process are becoming increasingly complex, especially with the introduction of virtualization, where computers not only run applications but also need to run containers and virtual machines, which may already be a heavy burden on the CPU.
The increase in Ethernet speed is greater than the increase in CPU computing speed, resulting in less time for the CPU to process individual network packets. If the CPU cannot process network data in a timely manner, it will inevitably affect network transmission latency and throughput. Therefore, some technologies/schemes are needed to reduce the time it takes for the CPU to process individual network packets.
DMA
DMA stands for Direct Memory Access. DMA can be applied to both network data sending and receiving. DMA is a general-purpose technology that has an independent DMA controller from the CPU. When copying data, the CPU only needs to tell the DMA controller the starting address of the data, the data length, and then hand over the bus control to the DMA controller, which can complete the data copy without CPU intervention.
Using DMA, when the network card copies data from memory (sending) and the network card copies data to memory (receiving), only minimal CPU intervention is required.
RSS
RSS stands for Receive Side Scaling, which can be inferred from its name, is an acceleration technology that only takes effect when receiving network data. Network cards with RSS capabilities have multiple receive queues, and the network card can use different receive queues to receive different network flows, and then assign these queues to different CPU cores for processing, making full use of the multi-core processor's capabilities, dispersing the network data reception load, and thus improving network transmission efficiency.
Although RSS can better utilize multi-core CPUs, on the one hand, network data distribution needs to consider TCP connections, NUMA, and other factors, which are relatively complex. On the other hand, it increases the impact of network transmission on the CPU. As mentioned earlier, computers have their own computing tasks and cannot be used solely for network transmission. In actual use, RSS is usually limited to a limited number of CPU cores to isolate the CPU impact of network transmission.
NAPI
As network interface bandwidth has advanced from gigabits to tens of gigabits, the original per-packet interrupt has become a significant overhead, and frequent interrupts due to a large amount of data can cause the system to become unable to handle it. Therefore, someone introduced the NAPI mechanism in the Linux kernel, which uses a polling approach to process multiple data packets at once after being interrupted, until the network is idle again and reverts to interrupt waiting. The NAPI strategy is used in high-throughput scenarios and has significantly improved efficiency.
NAPI stands for New API, which is an optimization for network reception in Linux systems. Hardware I/O and CPU interaction generally have two methods: interrupts and polling. Interrupts have a high CPU cost but good real-time performance and do not require the CPU to constantly monitor. Polling requires the CPU to periodically query I/O and requires the CPU to constantly monitor, which is not truly real-time. For network cards, a busy network will trigger an interrupt for each network data packet, which can affect the overall system efficiency. For a network with small traffic, if polling is used, it will lead to increased latency and low CPU efficiency.
NAPI adopts different methods for CPU and network card interaction based on different scenarios. In scenarios with large network traffic, it uses polling to read network card data. In scenarios with small network traffic, it uses interrupts, thus improving CPU efficiency.
Checksum offload
Many network protocols, such as IP, TCP, and UDP, have their own checksums. Traditionally, checksum calculations (sending data packets) and verifications (receiving data packets) are performed by the CPU. This has a significant impact on the CPU, as checksums require participation from every byte of data. For a 100G bandwidth network, the CPU needs to calculate approximately 12G of data per second.
To alleviate this impact, current network cards support checksum calculations and verifications. The system kernel can skip checksums when encapsulating network data packets. The network card calculates the checksum according to the network protocol rules after receiving the network data packet and fills in the corresponding position.
Due to the existence of Checksum offload, when using packet capture analysis tools like tcpdump, it is sometimes found that the captured packets indicate checksum errors. The network packets captured by tcpdump are the packets sent by the system kernel to the network card. If the checksum is calculated by the network card, at the time the packet is captured, the checksum has not been calculated yet, so the value seen is incorrect.
[root@network-test ~]# ethtool -k eth0 | grep -i Checksum
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
Scatter/Gather
This acceleration can only be used for sending network data. Scatter/Gather itself is also a general technology in operating systems, also known as vector addressing. Simply put, during data transmission, the data recipient does not need to read data from a continuous block of memory, but can read data from multiple discrete memory addresses. For example, when the system kernel receives raw data from an application, it can keep the data unchanged. Then, in another block of memory, it calculates the headers of various protocol layers. Finally, it notifies the network card driver to copy the data from these two blocks of memory. SG can reduce unnecessary memory copy operations.
SG requires Checksum offload support, because now that the data is discrete, it is not easy for the system kernel to calculate the Checksum.
[root@network-test ~]# ethtool -k eth0 | grep scatter
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
[root@network-test ~]#
TSO
TSO is short for TCP Segmentation Offload, which can only be used for sending network data. As the name suggests, it is a method closely related to the TCP protocol.
Applications can pass data of any length to TCP. TCP, located in the transport layer, will not directly pass the entire user data to the lower-layer protocol for transmission. Because TCP is a reliable transmission protocol, while the lower-layer protocols, IP/Ethernet, are not reliable, data may be lost during transmission. TCP not only needs to ensure the reliability of transmission but also needs to maximize the success rate of transmission to ensure efficiency. TCP's approach is to divide the data into smaller segments and transmit them separately.
Before proceeding with the description, let's first clarify two similar and easily confused terms. One is Segmentation, and the other is Fragmentation. The TCP protocol will divide large data into smaller segments based on the Maximum Segment Size (MSS) before passing it to the IP layer. This process is called Segmentation, and the resulting data is called Segments. The IP protocol, due to the limitation of the Maximum Transmission Unit (MTU), will divide the data received from the upper layer into smaller fragments if it exceeds the MTU. This process is called Fragmentation, and the resulting data is called Fragments. Both processes involve dividing large data into smaller blocks, but the difference lies in that one is completed at the TCP layer (L4) and the other at the IP layer (L3).
Let's continue with the description. If TCP directly transmits the entire data to the lower-layer protocol, assuming it's 15,000 bytes of user data, and the MTU of the network card is 1,500, considering the header, the IP layer will divide the data into 11 IP fragments for transmission over the network. For simplicity, let's assume it's divided into 10 IP fragments. Assuming the transmission success rate of each IP packet is 90%, because TCP has its own checksum, the receiving end of the IP protocol must receive the complete 15,000 bytes of user data and assemble it before passing it to TCP, which is considered a successful reception. In this case, the probability of successful transmission in one attempt is (90%)^10 = 34%. If the TCP receiving end does not successfully receive the data, the sending end needs to retransmit the entire 15,000 bytes of data. Assuming it takes four attempts, which means a total of 60,000 bytes are transmitted, the transmission success rate can be increased to 80%.
What if the TCP protocol itself divides the data into smaller segments for transmission? As mentioned earlier, TCP performs Segmentation based on MSS, usually calculated based on MTU to ensure that a TCP segment does not need to be fragmented at the IP protocol layer. For simplicity, let's still ignore the headers of network protocols. Now, TCP divides the 15,000 bytes of application-layer data into 10 segments at its own layer. Each segment corresponds to one IP packet, with a success rate of 90%. If a segment fails to send, TCP only needs to retransmit the current segment, and previously successfully sent TCP segments do not need to be retransmitted. In this way, for each segment, sending it twice can achieve a success rate of 99%. Assuming each segment is sent twice, the corresponding application-layer data is sent twice as well, which is 30,000 bytes, and the transmission success rate can reach (99%)^10 = 90%. That is to say, after TCP Segmentation and then transmission, less data needs to be sent, and the success rate is actually higher. Of course, in reality, due to TCP Segmentation, each TCP segment will increase the TCP header, resulting in slightly more data being transmitted. However, this small amount of data does not affect the analysis result. Therefore, TCP Segmentation is necessary for the reliability of TCP.
However, it also has its own disadvantages. After TCP Segmentation, it is equivalent to dividing a piece of data into several TCP segments, each with its own TCP header, which requires the CPU to calculate checksums, sequences, etc. At the same time, each TCP segment will also have its own IP protocol header, which requires the CPU to calculate the contents of the IP protocol header. Therefore, it can be foreseen that after TCP Segmentation, the CPU burden increases significantly.
TSO offloads the work of TCP Segmentation to the network card. With TSO, the operating system only needs to pass a large TCP data packet (of course, packaged in Ethernet Header and IP Header, and not exceeding 64K) to the hardware network card. The network card will replace the TCP/IP protocol stack to complete TCP Segmentation. This eliminates the CPU burden brought by TCP Segmentation.
Another benefit lies in DMA. Although each DMA operation does not require much CPU intervention, the CPU still needs to configure the DMA controller. The characteristic of DMA is that regardless of the length of the data being transmitted, the configuration workload is the same. If the system kernel itself completes TCP Segmentation, there will be several TCP segments that need to be transmitted to the network card via DMA. By adopting TSO, because a large piece of data is being transmitted, only one DMA configuration is needed to copy the data to the network card. This also reduces the CPU burden to some extent.
Network cards that support TSO will still generate and send network data packets according to the TCP/IP protocol. For external systems, the existence of TSO is imperceptible.
The improvements brought by TSO are significant. On the one hand, more CPUs are released to complete other tasks. On the other hand, network throughput is not affected by CPU load. Without TSO, when CPU performance is poor or the CPU itself is already heavily loaded, the CPU cannot process enough network data in time, resulting in decreased network throughput and increased latency.
TSO requires the support of SG and Checksum offload. Because the TCP/IP protocol stack does not know what the final network data packet looks like, it naturally cannot complete the checksum calculation.
# 检查命令：
ethtool -k eth1 | grep tcp-segmentation-offload
# 关闭命令：
ethtool -K eth1 tso off
Potential Issues with TSO Configuration
In actual operation, potential issues may arise due to TSO configuration being incompatible with the business gateway's traffic forwarding mode:
TSO enables the network protocol stack to push data exceeding the PMTU to the network card, which then performs the segmentation work. When TSO is enabled, the TCP layer gradually increases the MSS value. However, in some network models, such as SDN virtual networks using GRE or IPIP tunnel modes, the encapsulated packets may not allow fragmentation due to the inner layer encapsulation. As a result, when TSO is enabled, the increased MSS value can lead to the generation of large tunnel data packets, which may be discarded at the network card if they exceed the MTU, causing disconnections or high latency issues.
Jumbo Frames
When Ethernet was first introduced, it was designed with a Maximum Transmission Unit (MTU) of 1500 bytes, which is the maximum payload size of an Ethernet frame. The reason for this choice is a trade-off between efficiency and reliability. Longer packets result in higher efficiency but also increase the likelihood of packet loss. On the other hand, shorter packets reduce efficiency due to the lower proportion of effective data in the overall network data but also decrease the likelihood of packet loss. Therefore, the IEEE 802.3 standard specifies an MTU of 1500 for Ethernet.
During network transmission, the MTU must match. For example, a machine with an MTU of 1500 can receive data from a machine with an MTU of 9000 without issues. However, if a machine with an MTU of 9000 sends data to a machine with an MTU of 1500, the data may be too long and get discarded due to the MTU mismatch. As a result, the MTU of the sending and receiving ends must match. Additionally, the internet has been built based on the IEEE 802.3 standard's MTU of 1500 to ensure uniformity and compatibility.
However, modern network devices have significantly improved reliability and can transmit larger network packets stably. Jumbo Frames refer to Ethernet frames with an MTU of 9000 bytes. With Jumbo Frames, each network packet has a higher proportion of effective data due to the fixed length of the network protocol header. For example, in a 10G network, an MTU of 1500 requires the CPU to process over 800,000 network packets per second, while an MTU of 9000 only requires processing 140,000 packets per second. As a result, the CPU has more time to process each packet under an MTU of 9000.
Supporting Jumbo Frames requires corresponding hardware, which is widely available in modern devices and can be enabled with simple configuration. However, Jumbo Frames have limitations in actual use due to the existing internet infrastructure built on an MTU of 1500. Since the MTU must match, it is impractical to modify the entire network. Therefore, Jumbo Frames are typically used only in internal data center networks, such as internal storage networks. The MTU for internet connections is usually set to 1500.
GSO
GSO stands for Generic Segmentation Offload, which is only effective during network data transmission. According to Herbert Xu, the author of GSO, "If we can't use a larger MTU, we can go for the next-best thing: pretend that we're using a larger MTU." Since the internet is built on an MTU of 1500, network packets transmitted over the internet must comply with this standard. GSO aims to delay IP fragmentation as much as possible in the operating system, creating a "path" in the TCP/IP protocol stack where network data with a payload exceeding 1500 bytes can be transmitted. This approach reduces the number of network packets the CPU needs to process, allowing for more time to handle each packet.
The idea behind GSO is similar to TSO, which avoids TCP segmentation and IP fragmentation between the user program and the network card, allowing for larger packets up to 64K. However, TSO only supports TCP protocols and requires hardware support from the network card, whereas GSO is designed for other scenarios. Since most network data fragmentation occurs at the IP layer for non-TCP protocols, GSO is not entirely accurate in its naming.
GSO chooses to split large packets into smaller ones just before sending them to the network card driver, as it does not rely on hardware support. Although the network card still receives multiple small network packets, there is a "path" in the TCP/IP protocol stack where the CPU processes fewer large packets.
Since GSO completes the segmentation just before sending the packets to the network card driver, it can serve as a fallback for TSO. When sending packets to the network card driver, GSO checks if the network card supports TSO. If it does, the large packets are sent directly to the network card driver. If not, GSO is performed.
According to the Linux Foundation's documentation, using GSO can increase network throughput by 17.5% when the MTU is 1500.
LRO
LRO stands for Large Receive Offload, also known as RSC (Receive Side Coalescing). As the name suggests, it is only effective when receiving network data. LRO is the inverse implementation of TSO, where the network card combines TCP segments of the same TCP connection into a larger TCP packet before passing it to the operating system. This reduces the CPU's processing time and the number of network packets that need to be processed in the TCP/IP protocol stack. Like TSO, LRO requires support from the network card.
However, unlike TSO, LRO is not as useful. Because TSO occurs on the sending side, the sender has control over the entire network data and can manage the sending process accordingly. LRO, on the other hand, occurs on the receiving side and is asynchronous to the sender, so it can only make decisions based on the limited data and information available, making it more challenging. This is like disassembling something is easy, but reassembling it is difficult.
LRO may lose important data, such as fields added to the header by the sender to distinguish between different network packets. Merging packets may cause these fields to be lost, as there is only one header after merging. Moreover, when the operating system needs to forward data, the merged packet may need to be re-segmented, and the original header's distinguishing fields will be lost. Due to LRO's limitations, it has been removed from some of the latest network cards.
GRO
GRO stands for Generic Receive Offload, which is the counterpart to GSO on the receiving end. GRO's author is the same as GSO's, Herbert Xu. Unlike GSO, which is a supplement to TSO, GRO has gradually replaced LRO. Because GRO runs in the system kernel, it has more information and can use stricter rules to merge network packets. This allows it to avoid losing critical information. On the other hand, in cases where forwarding is required, GRO can utilize GSO's code to re-segment packets.
Other advantages of GRO include its greater generality, as it is not dependent on hardware devices and supports protocols beyond TCP.
UFO
UFO stands for UDP fragmentation offload. As the name suggests, it is an optimization for UDP. Unlike TCP, UDP does not have a segmentation process, and user programs send data to UDP, which is then passed to the IP layer. The IP layer performs fragmentation based on the MTU. UFO enables network devices, such as network cards, to divide a long UDP data segment (exceeding the MTU) into multiple IPv4 fragments. This reduces the CPU's processing time.
However, in the latest Linux kernel, UFO has been deprecated. Since most offload processes, except for TSO, perform fragmentation at the IP layer, UFO is no longer necessary and has been merged with GSO.
tx-udp_tnl-segmentation
Overlay networks, such as VxLAN, are becoming increasingly popular. Overlay networks allow users to create, configure, and manage virtual network connections without being limited by physical networks. They also enable multiple tenants to share a physical network, increasing network utilization. There are many types of overlay networks, but VxLAN is the most representative. VxLAN is a MAC in UDP design, with the following format:
From VxLAN's format, it can be seen that overlay networks, represented by VxLAN, have two performance issues. One is the increase in overhead, as VxLAN adds an extra layer of Ethernet+IP+UDP+VXLAN to the original Ethernet frame, resulting in an additional 50 bytes of data being transmitted. Therefore, it can be expected that the efficiency of overlay networks will be lower than that of underlay networks. The other issue is more severe than the additional 50 bytes, which is the need to process these extra bytes. These 50 bytes include four headers, each involving copying and calculation, which consumes CPU resources. Our current problem is that the CPU has less time to process each network packet.
Firstly, VxLAN's 50 bytes are unavoidable. Secondly, the only option is to reduce their impact. Here, the idea of Jumbo Frames can still be applied, as the 50 bytes are fixed, and the larger the network packet, the smaller the relative impact of the 50 bytes.
Let's take a look at the network connection diagram of the virtual machine. The virtual machine connects to the TAP device on the host machine through QEMU, and then passes through the virtual machine switch to the VTEP (VxLAN Tunnel EndPoint), encapsulates the VxLAN format, and sends it to the host machine's network card.
In an ideal situation, a large segment of VxLAN data is directly transmitted to the network card, and the network card completes the remaining fragmentation, segmentation, and encapsulation of VxLAN for each small network packet, as well as checksum calculation and other tasks. This way, the impact of VxLAN on virtual machine networks can be minimized. In reality, this is possible, but it requires a series of preconditions.
First, the virtual machine needs to send large network packets to the host machine. Since the virtual machine also runs an operating system and has its own TCP/IP protocol stack, the virtual machine is fully capable of splitting large network packets into smaller ones on its own. From the introduction above, only TSO can truly send a large network packet to the network card. GSO splits the large network packet into smaller network data packets before entering the driver. Therefore, the following conditions are required: the virtual machine's network card supports TSO (Virtio defaults to support), and TSO is enabled (default enabled), and the virtual machine sends TCP data.
After passing through QEMU, virtual machine switching, and VTEP encapsulation, the large TCP data is encapsulated in VxLAN format. 50 bytes of VxLAN data are added to this large TCP data. The problem arises here: this was originally a TCP data, but because it has been encapsulated in VxLAN, it now looks like a UDP data. If the operating system does not perform any processing, according to the introduction above, it should go through GSO to perform IP fragmentation and split it into smaller packets before sending it to the network card. In this way, if the network card originally supported TSO, it can no longer be used. Moreover, the more serious problem is that TCP segmentation is also lost.
For modern network cards, in addition to TSO and GSO offload options, there is another option tx-udp_tnl-segmentation. If this option is enabled, the operating system itself will recognize that the UDP data encapsulated in VxLAN is tunnel data, and the operating system will directly send this large VxLAN data to the network card for processing. In the network card, the network card will perform TCP segmentation on the inner TCP data. Then, it will add VxLAN encapsulation (50 bytes) to each TCP segment. In this way, the impact of VxLAN on virtual machine networks is minimized.
From the above description, to achieve the above effect, the host machine's network card needs to support both TSO and tx-udp_tnl-segmentation. If either of these is not supported, the system kernel will call GSO to split the large TCP data encapsulated in VxLAN format into smaller packets before sending it to the network card driver, and add VxLAN encapsulation to each TCP segment.
————————————————
Copyright statement: This article is an original work by CSDN blogger "海渊_haiyuan", following the CC 4.0 BY-SA copyright protocol. For reprinting, please attach the original article link and this statement.
Original article link: https://blog.csdn.net/LL845876425/article/details/107499529
若无法加载请检查网络环境。
若无法加载请检查网络环境，或切回 Disqus 稍后再试。