Everything You Should Know About RoCE

RoCE is a technology that meets the needs of high-bandwidth and high-performance applications. The traditional TCP/IP protocol has high latency and inefficient bandwidth utilization and cannot meet the growing network needs, so RoCE was developed. It integrates remote memory access into Ethernet through RDMA technology, significantly improving data transmission efficiency and reducing latency. This article will take you to a detailed understanding of what RoCE is, so that you can have a deep understanding of it.

What is RoCE Network

RoCE (RDMA over Converged Ethernet) is an Ethernet-based remote direct memory access (RDMA) technology designed to achieve low-latency and high-throughput data transmission over lossless Ethernet. RoCE technology introduces RDMA technology in high-performance computing and networks into standard Ethernet infrastructure, giving traditional Ethernet RDMA capabilities and high-performance computing properties. Therefore, RoCE technology has the powerful advantages of both Ethernet and RDMA.

RoCE technology allows high performance similar to InfiniBand to be achieved on existing Ethernet networks. It reduces latency and increases bandwidth by optimizing Ethernet protocols. Compared with traditional Ethernet: RoCE provides lower latency and higher throughput while maintaining close compatibility with existing Ethernet devices.

Advantages of RoCE: The main advantages of RoCE are its cost-effectiveness and compatibility. Since it is based on Ethernet standards, it can use mature network equipment and cables on the market, is not subject to single brand and regional restrictions, and has good supply cycle and after-sales service. Key is that it also has excellent compatibility with Ethernet and its equipment, greatly improving flexibility and reducing costs. Therefore, RoCE technology is increasingly widely used in data centers, HPC high-performance storage, artificial intelligence and other fields, becoming a cost-effective interconnection technology in GPU server interconnection. QSFPTEK provides L3/L3+ switches that support RoCE2.

Development and Types of RoCE

RoCE v1

In April 2010, IBTA released RoCE. This standard was released as an add-on to the Infiniband Architecture Specification, so it is also called IBoE (InfiniBand over Ethernet). At this time, the RoCE standard replaced the TCP/IP network layer with the IB network layer on the Ethernet link layer, so it does not support IP routing, so it is called RoCE v1.

RoCE v1 (Layer 2) operates on the Ethernet Link Layer (Layer 2), so the typeID of the RoCE v1 protocol at the Ethernet layer is 0x8915, so the normal Frame size is 1500 bytes, and the Jumbo Frame is 9000 bytes.

RoCE v1 is an RDMA protocol implemented based on the Ethernet link layer (the switch needs to support flow control technologies such as PFC to ensure reliable transmission at the physical layer), allowing two hosts in the same VLAN to communicate.

RoCE v2

Since the data frame of RoCE v1 does not carry an IP header, it can only communicate within the L2 subnet. To solve this problem, IBTA proposed RoCE V2 in 2014. RoCE v2 is an extension of RoCE v1, replacing the GRH (Global Routing Header) with a UDP header + IP header. RoCE v2 overcomes the limitation of RoCE v1 being bound to a single VLAN. By changing the packet encapsulation, including the IP and UDP headers, RoCE v2 can be used across L2 and L3 networks. The extended frame structure is shown in the figure below.

RoCE v2 (Layer 3) operates on UDP/IPv4 or UDP/IPv6 (Layer 3) and uses UDP Port 4791 for transmission. Because RoCE v2 packets are routable on Layer 3, RoCE v2 can also be called Routable RoCE or RRoCE for short.

RoCE v2 is an evolution and improvement of the previous generation RoCE v1. By introducing a series of improvements, it effectively solves the original limitations and comprehensively improves performance. The second-generation version RoCE v2 stands out with its significant performance improvement and greater flexibility, and has become the mainstream application in the market.

Implementation of RoCE

In order to achieve RoCE interconnection, the network card device must install a driver that supports RoCE. All Ethernet NICs (Network Interface Cards) must support RoCE network adapter cards (i.e. RoCE network cards). RoCE drivers are available in Red Hat, Linux, Windows, and other common operating systems. Therefore, three methods are required to implement RoCE networks:

a. For servers or host devices, you need to use RoCE network cards to connect.

b. For network switches, choose to use switches that support PFC operating systems. PFC, or priority flow control, is one of the necessary means to build lossless Ethernet. It can provide priority-based flow control hop by hop. This individual flow control method allows the network to remain smooth when some traffic is congested, while preventing interference with other traffic, thereby ensuring large bandwidth and low latency.

c. Build an intelligent lossless Ethernet architecture.

RoCE NIC

RoCE NIC is a network card that supports the RoCE protocol. It uses RDMA technology and integrates necessary hardware features. It can free the CPU from heavy tasks, thereby significantly reducing the latency of data transmission and effectively improving the operating performance and bandwidth of the entire system. 

High-performance and High-bandwidth Switch

Due to the demand for large bandwidth, RoCE often requires the support of high-performance and high-bandwidth switches. The core foundation for building high-performance and high-bandwidth network switches lies in the forwarding chip technology of the switch.

Intelligent and lossless Ethernet Architecture

In addition to the above hardware resources, in the RoCE network, we also need to build an intelligent lossless Ethernet architecture to ensure that there is no packet loss during network transmission.

Intelligent lossless networks use flow control technology and congestion control technology to improve the overall network throughput and reduce network latency. On the other hand, they use technologies such as intelligent lossless storage networks to achieve network and application system integration optimization. According to the correspondence between intelligent lossless network technology and TCP/IP protocol stack, the following is the technical architecture diagram of intelligent lossless network:

Comparison Between IB and RoCE v2 Networks

Technical Comparison

From a technical perspective, InfiniBand uses a lot of technologies to improve network forwarding performance, reduce fault recovery time, improve scalability, and reduce operation and maintenance complexity.

 

Business Scenario Comparison

Business performance: Since the end-to-end latency of InfiniBand is lower than that of RoCE v2, the network built on InfiniBand has an advantage in application layer business performance. However, the performance of RoCE v2 can also meet the business performance requirements of most intelligent computing scenarios.

Business scale: InfiniBand can support a single cluster of 10,000 GPU cards and ensure that the overall performance does not decrease, and there are many commercial practice cases in the industry. RoCE v2 network can support a scale of 1,000 GPU cards in a single cluster without much reduction in overall network performance.

Business operation and maintenance: InfiniBand is more mature than RoCE v2, including multi-tenant isolation capabilities, operation and maintenance diagnostic capabilities, etc.

Business cost: The cost of InfiniBand is higher than RoCE v2, mainly because the cost of InfiniBand switches and modules is higher than that of Ethernet switches.

Conclusion

RoCE bridges the gap between high-performance computing and Ethernet-based networks by integrating RDMA functions into Ethernet infrastructure. It provides low latency, high bandwidth and cost-effectiveness, making it an ideal choice for applications such as data centers and cloud computing. Compared with InfiniBand, it has wider compatibility and lower costs. Although RoCE is slightly inferior in latency and multi-tenant management, it is a highly competitive alternative that breaks the position of InfiniBand.