iSCSI Network Designs: Part 5 — iSCSI Multipathing, Host Bus Adapters, High Availability and Redundancy

In iSCSI Part 3 — Server Side — iSCSI Host Bus Adapters and IP Performance I looked at how server side issues would affect the traffic gen­er­ated on a per server basis. I recom­men­ded that you use iSCSI HBAs for high intens­ity serv­ers to meet the high levels of performance.

The next level is eval­u­ate how the server should con­nect to the net­work, spe­cific­ally, this means how many eth­er­net ports you need, and what con­fig­ur­a­tion is needed to sup­port them to deliver high avail­ab­il­ity /​ redund­ancy and increased bandwidth.

I feel that is import­ant to under­stand how the adapters will integ­rate with the switch­ing infra­struc­ture so that I can ensure that the net­work delivers.

To sum­mar­ise the iSCSI ini­ti­ator on the server — I like this pic­ture because it speaks to the ISO seven layer model (which is embed­ded in my design think­ing) (1):

This nicely presents how the per­form­ance can be achieved. Particularly you can see the import­ance of the soft­ware drivers since they will be key in terms of the features.

Redundancy

To be able to design the net­work for iSCSI redund­ancy we need to under­stand how iSCSI imple­ments its HA fea­tures. There are three ways to achieve this:

1. Link Aggregation — LACP
2. iSCSI Multipathing — Active /​ Standby
3. iSCSI Multipathing — Active /​ Active

The fol­low­ing dia­gram shows the rela­tion­ships between the dif­fer­ent solu­tions using an approx­im­a­tion of the ISO model.

Link Aggregation — Etherchannel /​ LACP

The most simple of the choices but requires the most con­fig­ur­a­tion and setup. You must have the fol­low­ing criteria:

• two net­work cards in the servers
• the two net­work cards must con­nect to the same Cisco switch (2)
• net­work adapters and drivers must have sup­port for LACP
• at least one switch with sup­ports LACP (some­times know as Etherchannel)
• you must be able to con­fig­ure both the server drivers and switch configuration

Storage data only

I recom­mend that the only IP traffic on this inter­face is iSCSI. This will ensure that you do not impact stor­age per­form­ance by over­run­ning the inter­face with data. I hope to post a future art­icle show­ing how to con­fig­ure QoS in the net­work if you must mix IP stor­age and IP data traffic on the same net­work adapter.

Network Adapters

Not all adapters sup­port LACP, and some vendors have a license that you pur­chase as an upgrade to enable it. It is a hard­ware fea­ture and soft­ware drivers will not make LACP avail­able. Check your net­work adapters sup­ports LACP.

LACP load balancing

There are cer­tain cir­cum­stances when LACP load bal­an­cing algorithm does not work optim­ally. Typically the driver makes a hash of the source and des­tin­a­tion IP address and then sends all data down one port of the LACP bundle. Normally there are enough con­ver­sa­tions to make this bal­ance work sat­is­fact­or­ily, that is, there are enough unique source/​destination pairs to evenly bal­ance the load between the two inter­faces, but if all your data from your Server to Storage is a single source/​destination then, by default, only one gig­abit eth­er­net port will be used. A key server might need more than one gig­abit of bandwidth.

You can con­fig­ure the Switch LACP AND the Server LACP (since each can only load bal­ance the out­bound frames) to use SRC IP /​ Port and DST IP /​ Port to make the full LACP band­width avail­able to a single iSCSI ini­ti­ator /​ tar­get pair.

iSCSI Multipathing Active /​ Standby

This is defined in the iSCSI RFC as the method for achiev­ing high avail­ab­il­ity. The iSCSI ini­ti­ator will ini­ti­ate at least two TCP con­nec­tions to the iSCSI tar­get. Data will flow down the primary con­nec­tion until a fail­ure is detec­ted and data will then be diver­ted to the second connection.

Since the con­nec­tion is fully estab­lished, fail­over should be fast.

Note that you can still use LACP to improve band­width. e.g. if you need two gig­abits of stor­age band­width, you would need 4 GB ports in two bundles of two to achieve this. This is a reas­on­able option for large VMware ESX serv­ers which might host thirty or forty guest serv­ers using the the iSCSI stor­age and you want to have max­imum availability.

Dell has an example of how to con­fig­ure this.

iSCSI Multipathing Active /​ Active

This solu­tion is identical to the iSCSI Active /​ Standby, how­ever, the iSCSI ini­ti­ator and tar­get include addi­tional driver soft­ware that allow data to be trans­ferred over either connection.

As before, LACP can be used to improve band­width by cre­at­ing multi-​​gigabit connections.

Conclusion

We have looked at the three options for redund­ant iSCSI con­nec­tions. Note that each of these solu­tions is pos­sible with nat­ive net­work adapters, TOE or HBA (as covered in iSCSI Part 3 — Server Side — iSCSI Host Bus Adapters and IP Performance so that you can choose which per­form­ance level suits the require­ments for the server.

From this inform­a­tion, it would seem that iSCSI mul­tipath­ing Active/​Standy is most effect­ive for most people since it is simple to con­fig­ure and oper­ate. Choosing iSCSI Multipathing Active/​Active would be a good choice for a server /​ stor­age com­bin­a­tion where you can spend time test­ing the drivers and func­tion­al­ity. I am not sure but I think that both the stor­age and server would need to sup­port this fea­ture (can any­one con­firm ? Leave a com­ment below if you know).

You could achieve redund­ancy using LACP on the server and stor­age (and also a per­form­ance improve­ment), but you must be able co-​​ordinate the switch con­fig­ur­a­tion and the server con­fig­ur­a­tion, plus modify the LACP load bal­an­cing type.

You could do both of course, using the fol­low­ing dia­gram which show using Active Standby iSCSI Multipathing AND LACP to cre­ate a two gig­abits per second con­nec­tion. Note that your server archi­tec­ture, iSCSI driver and/​or HBA would need to be able to gen­er­ate this volume of traffic for it to be use­ful (and of course, the stor­age unit would also need to have the cap­ab­il­ity to move this volume of data).

With this type of imple­ment­a­tion the phys­ical con­nec­tions should look like this:

This would require a total of four eth­er­net ports per server, with NICs that sup­port LACP, and two switches (LACP sup­port). This would ensure redund­ant paths and bandwidth.

Whats Next

Now that we have fin­ished the con­sid­er­a­tions for the Server con­nec­tions to the net­work, we can return to design­ing the net­work and con­sid­er­ing band­width, eth­er­net per­form­ance and QoS requirements.

Footnotes

(1) iSCSI Performance Options — NetApp Whitepaper June 2005.

(2) Nortel has Multi Link Trunking which allows for LACP to span mul­tiple switches. I do not recom­mend using Nortel switches to access this fea­ture as my exper­i­ence of Nortel is less than excel­lent. I under­stand that Nortel owns the pat­ent and does not make it avail­able to other companies.

Please rate this post:

(1 votes, average: 7.00 out of 10)

 Loading ...