I am working on a Data Centre that has more than 500 servers and we are considering what our storage strategy should be. This is why I have been looking at FCoE and iSCSI and establishing the benefits (or not).

My next few posts will be a “stream of consciousness” on possible designs for an iSCSI infrastructure.

Administrative Concerns

In my mind, I have concerns that many networking people do not take enough care when configuring their switches. A short interruption in network access is often not noticed, but for an iSCSI connection an outage of more than a few seconds will cause problems. For certain types of software (databases) any interruption will cause corruption and server outages.

Servers Intensity

In simple terms, I classify servers as low intensity and high intensity.

Low intensity would refer to servers that :

1) use limited resources (CPU, Memory, Disk) and thus require a low level of performance

2) are not critical.

Typical servers in this category are email, fax gateways, WSUS, network management systems.

High intensity would refer to servers that are high resource and / or business critical.

1) require CPU / Memory / Disk in large volumes, or to be able to spike to large volumes to meet performance (either real or perceived).

2) are mission critical and would severely impact the business if an outage occurred.

3) if the OS or Server platform suffers a failure, then the restoration process would require rebuilding e.g. database application where a corruption of the database would require hours to rebuild.

iSCSI initiator performance

When implementing iSCSI in your server operating system you have two choices, software or hardware (using Host Block Adapter).

A software initiator is identical to what happens in your server today. The TCP/IP packet is generated by the operating system using your core CPU. This includes some quite intensive operations such generating the CRC for the TCP Packet, sequence calculations and tracking, session initiation and teardown, sliding window calculations and so on. The operating system kernel then pushes the prepared packet to the NIC for transmission.

A Host Bus Adapter is a PCI card that performs the TCP processing. This is known as TCP Offload and the cards are generally known as TCP Offload Engines or TOE.

TOE basically means that there a dedicated chip on the network card that performs the TCP functions. TOE cards are conceptually similar to GPU’s for graphics processing. And just like GPU cards, the quality of the TOE cards varies according to the manufacturer and price. TOE Network Drivers are also important (in the same way the Nvidia drivers for graphics are important.

I would suggest that you can use iSCSI software initiators for low intensity servers and the performance will be acceptable. You should certainly consider iSCSI HBA for high intensity server.

While prices for HBA are relatively expensive, they offer significant performance gains. Take time to ensure that you purchase a quality product as the software drivers will be vital to your success.

iSCSI Security

FCoE has made a lot of noise about their encryption feature. If thats important to you then you should remember that iSCSI has had a IPSec in the RFC since April 2004.

It seems to be agreed that practical implementations of IPSec have not been possible because of performance concerns. However, I suspect that HBA will soon (if they don’t already) include silicon to perform the IPSec processing. Conceptually I can envisage standard IPsec encryption silicon from the major chip fabs to appear on iSCSI HBAs without too much problem.(3)

I like the idea of IPSec, I already know about it and can use my ‘off the shelf’ knowledge to set it up.

Network Requirements

In my reading, many articles point out that Storage Area Networks have a requirement for low latency and low jitter.

So my first method of addressing this is to use the following design of a single layer Ethernet switch to connect servers and iSCSI storage.

There a a number of features about this design that I like.

Conclusion

In some ways, this looks like a simple Fibrechannel design, without the overhead of a Fibrechannel switch system. The maximum density of the system is determined by the number of ports on the switch, and they must all be physically located near the switch.

Another limitation is that bandwidth is limited to 1Gb/s (unless you are purchasing a 10GB Ethernet switch and NIC’s, but there are not many 10GB HBA around).

So the next step will be to grow this design to encompass a larger number of servers, and to increase the bandwidth.

Feedback

As I am learning from textbooks and various Internet sources, please feel free to contact me or comment. I would value some input. myetherealmind (at} gmail.com.

Footnotes

(1) Stream of consciousness: phrase used by William James in 1890 to describe the unbroken flow of thought and awareness of the waking mind

(2) Note that Fibrechannel has exactly the same issues as FC uses the same concepts to improve performance.

(3) Let me know if these already exist and I will update.