There is a lot of talk about the value of Infiniband as a storage protocol to replace FibreChannel with several SSD vendors offering Infiniband options. Most likely this is necessary to allow servers to get enough network speed but mostly to reduce latency and CPU consumption. Good Infiniband networks have latency measured in hundreds of nanoseconds and much lower impact on system CPU because Infiniband uses RDMA to transfer data. RDMA ( Remote Direct Memory Access) means that data is transferred from memory location to memory location thus removing the encapsulation overhead of Ethernet and IP (that’s as short as I can make that description).
Infiniband works especially well for server area networks because the scale is relatively small. A few hundred servers is a good size for a Infiniband switch network. In
VMware has demonstrated some testing that proves this point. This RDMA on vSphere: Update and Future Directions shows testing by VMware on the enormous performance benefits for vMotion when using RoCEE – RDMA over Converged Enhanced Ethernet. By using the benefits of RDMA to reduce the protocol encoding overhead the performance can be dramatically improved.
I’ve taken these images from testing report (see previous) performed by VMware (CTO Office) and they show a striking performance improvement. If VMware moves to adopt RoCEE into the core hypervisor then these tests suggests that we could massive performance improvements, especially in CPU consumption, in the data centre without the hassle of using 40Gbe and 10GbE Ethernet.
In my view, the reduction in CPU has serious implications for improving guest mobility in large scale systems that have high GuestOS density.Imagine a server with fifty GuestOS using 80% less CPU during vMotion ? Excited ? You bet.
There is some serious impacts about how I look at Ethernet networks after reading this document.