Here’s an interesting article (but maybe you’ve seen it, but anyway): http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/

1 Million IOPS using nothing but software iSCSI initiator on Windows Server 2008 SR2 (and, of course, Intel Xeon 5500).

- Lossless Ethernet
- A similar error recovery logic like FCP SRR, simple and effective
- Transport over IP

Since the lossless Ethernet standard is coming, maybe it is time to consider retiring TCP from iSCSI fammily.

FCoE assumes that Ethernet does not drop packets. The application will retransmit as needed but if error rates exceed 1 in 10^12 then performance is severely impacted and can lead to loss of service.

PS – will check the follow-up comments – thanks for the tip.

iSCSI does not use FC, so there’s no FC error recovery mechanisms available.

> …FCoE is an example of exactly that…

If I understand it correctly, FCoE does not guarantee delivery by virtue of E. It does it by the virtue of FC. There is no FC in iSCSI.

> Storage industry has no ves≠ted interest in using cheaper and sim≠pler tech≠no≠logy

Storage industry (as any other out there) has vested interest in selling as many units as possible. If a younger competitor starts offering solutions comparable in features and reliability at much lower price point, larger players usually have no choice but to adopt.

(All my personal opinion and understanding, of course).

P.S. Looks like the “Notify me of follow-up comments by email” option on your blog engine does not work – I never got a notification of your reply.

this is so true. Hopefully Sun’s efforts shake things up a little

Second, protocols that guarantee delivery can be built and FCoE is an example of exactly that. but why guarantee delivery using Ethernet ? Why not use similar techniques for IP ?

The 802.1aq QoS mechanisms will work eqaully as well for iSCSI as for FCoE, and could easily be reconfigured for an IP Storage protocol.

The use TOE HBA’s is also highly recommended, however, the Storage industry has no vested interest in using cheaper and simpler technology and comprehensively dismissed these technologies as toys or not significant.

SCSI protocol is not designed to handle lost frames, or blocks. It expects them to arrive, no matter if they’re damaged (have CRC errors). IP or Ethernet will not deliver a frame which has CRC errors; behavior which is not compatible with what SCSI expects. This is why iSCSI uses TCP.

The way to improve performance of iSCSI is to use HBAs (Ethernet NIC) with TCP offload, which do all the necessary calculations on an embedded ASIC, significantly reducing the processing latency. Majority of storage arrays with iSCSI already make use of TCP offload, so some of the performance problems that you’re seeing are quite likely caused by a poor choice of Ethernet NIC in the server (or incorrect/badly configured NIC drivers).

The reliability of the FCoE is brought about by the enhanced end to end congestion avoidance mechanisms at the Ethernet level (DCE/CEE extensions), which are simply not available everywhere where you would use iSCSI, such as over a L3-only link such as IP VPN, for example.