What is SRD
SRD (Scalable Reliable Datagram), meaning scalable reliable datagram, is a high-throughput, low-latency network transmission protocol implemented by Amazon on AWS Nitro cards for HPC/ML applications.
SRD Features

  • Does not preserve packet order, leaving it to the upper-layer message passing layer to handle
  • Sends packets through as many network paths as possible, utilizing the ECMP standard, and controls packet encapsulation at the sending end to control ECMP path selection, achieving multi-path load balancing
  • Has its own congestion control algorithm, based on dynamic rate limiting for each connection, combined with RTT (Round Trip Time) to detect congestion, allowing for rapid recovery from packet loss or link failures
  • Due to unordered packet sending and lack of support for segmentation, the number of QPs (queue pairs) required for SRD transmission is significantly reduced
    Why not use TCP
    TCP is the primary means of reliable data transmission in IP networks, but it is not suitable for latency-sensitive applications: in data centers, the ideal round-trip latency for TCP is around 25us, but if congestion or link failures occur, TCP's waiting time can increase to 50ms. The main reason for this delay is TCP's retransmission mechanism after packet loss.
    Why not use RoCE
    RoCE (RDMA over Converged Ethernet) is an Ethernet implementation of IB, allowing InfiniBand transmission to run on Ethernet, supporting kernel bypass and transmission offload. The disadvantage of RoCE is its poor scalability, as RoCEv2 requires priority flow control (PFC), which is difficult to achieve in large-scale networks, causing head-of-line blocking, congestion spreading, and occasional deadlocks; even with PFC, RoCE still suffers from ECMP conflicts under congestion (similar to TCP) and suboptimal congestion control.
    SRD and RDMA similarities and differences