iSCSI Network Designs: Part 3 — Server Side — iSCSI Host Bus Adapters and IP Performance

I have been research­ing iSCSI impe­ment­a­tions on the server to try and under­stand the dffer­ence between them and to come to grips with how they work. This art­icle looks to com­pare the vari­ous meth­ods of con­nect­ing to a iSCSI network.

It seems that many people do not know or under­stand that the gen­er­a­tion and trans­mis­sion of IP pack­ets is CPU intens­ive pro­cess. In some oper­at­ing sys­tems, it can also be very lat­ent since there are many trans­fers across the memory bus and the PCI bus before the data is actu­ally transmitted.

When man­aging IP the server OS must man­age the following:

• ses­sion state for every TCP connection
• buf­fer for each session
• memory man­age­ment and con­trol for every IP session
• data trans­fer on and off the PCI bus

And when using TCP the fol­low­ing items must be handled by the driver.

• TCP Window Size
• TCP Window Scale
• TCP Timestamps
• TCP Delayed ACKs
• TCP Selective ACK
• 802.1x Flow Control /​ Ethernet Pause
• Maximum Segment Size (MSS) – Jumbo Frames

iSCSI Implementations

So you can imple­ment an iSCSI con­nec­tion on the com­puter using three basic modes:

1. Software ini­ti­at­ors
2. TCP Offoad Engines (TOE)
3. Host Bus Adapters

Software Initiators

This is the most com­mon imple­ment­a­tion. You can down­load the Microsoft iSCSI Initiator and get star­ted imme­di­ately. It seems that some other vendors also offer Initiators, e.g. DelL.

There are sev­eral per­form­ance issues with this approach:

1. you are using a gen­eral pur­pose CPU to per­form the data transformation
2. you are per­form­ing mul­tiple data copy func­tions across the internal bus of your computer.
3. it is not optim­ised for performance

For a desktop or low intens­ity server this might work OK. But for VM plat­forms and high intens­ity serv­ers, spend­ing a lot of CPU cycles gen­er­at­ing iSCSI pack­ets will impact performance.

TCP Offload Engines (TOE)

For a while I thought the TOE cards and HBA were the same thing but this is not true. The TOE cards are able to improve the TCP per­form­ance of a server. Since iSCSI uses TCP for data trans­fer, this will improve stor­age per­form­ance and reduce latency.

Many serv­ers now ship with TOE cards as stand­ard but most drivers have the TOE fea­ture dis­abled. Also, device driver qual­ity seems to be import­ant, so make sure you get latest ver­sions and use qual­ity vendors.

For serv­ers that exchange a lot of data, enabling TOE will improve per­form­ance and reduce the server CPU utilisation.

Host Bus Adapters

It is a gen­eric term for con­nect­ing the I/​O bus of your server to an external sys­tem such as Ethernet or FC. Thus an eth­er­net adapter is also a HBA. Note that both FC and iSCSI people use the term HBA.

Header and Data Digests were added by the iSCSI Working Group as a more robust mech­an­ism for ensur­ing data integ­rity com­pared to TCP check­sums. However, iSCSI Header and Data Digest cal­cu­la­tions are very CPU intensive.

Only a full iSCSI off­load HBA has the logic built into the ASIC to accel­er­ate these cal­cu­la­tions. General pur­pose NICs and TOEs do not have this innate cap­ab­il­ity; there­fore, the cal­cu­la­tions must be per­formed by the host CPU (if desired). If these cal­cu­la­tions are per­formed by the host CPU, both through­put and IOPS per­form­ance will fur­ther degrade, poten­tially slow­ing applic­a­tion per­form­ance to an unac­cept­able level

Full iSCSI off­load HBAs offer SAN Administrators what they need in a stor­age adapter, including:

• Full iSCSI off­load HBAs con­sist­ently have low CPU
  overhead
• offer iSCSI digest
  reli­ab­il­ity at line speeds without impact­ing the per­form­ance of host applications
• tools that show the capa­city and per­form­ance of your iSCSI connection.

It is the man­age­ment tools that are par­tic­u­larly inter­est­ing. The abil­ity to have exten­ded report­ing on the iSCSI data flows provides a real bene­fit in terms of loc­at­ing per­form­ance or net­work prob­lems, provide stats on packet drops and con­nec­tion failues and thus let you know that a prob­lem exists and give you tools to resolve that problem.

Conclusion

After research­ing HBA, it seems clear that you are more likely to be suc­cess­ful if you pur­chase iSCSI HBA for your serv­ers. It seems that many of the fea­tures that make Fibrechannel pop­u­lar are actu­ally derived from the FC HBA and not from an inher­ent superi­or­ity at some other layer.

When imple­ment­ing an iSCSI back­bone you should ensure that get iSCSI HBA for your serv­ers. You will improve your server per­form­ance, and get bet­ter vis­ib­il­ity into your iSCSI ser­vice and net­work overlay.

Please rate this post:

(2 votes, average: 10.00 out of 10)

 Loading ...