An article I wrote on concerns about FCoE adoption has been posted at The Future of Storage and reviews seven reasons why FCoE might not achieve critical mass in the marketplace.
Feedback appreciated. Is anyone considering using FCoE in real life ?
I feel that is important to understand how the adapters will integrate with the switching infrastructure so that I can ensure that the network delivers.
To summarise the iSCSI initiator on the server – I like this picture because it speaks to the ISO seven layer model (which is embedded in my design thinking) (1):
This nicely presents how the performance can be achieved. Particularly you can see the importance of the software drivers since they will be key in terms of the features.
To be able to design the network for iSCSI redundancy we need to understand how iSCSI implements its HA features. There are three ways to achieve this:
The following diagram shows the relationships between the different solutions using an approximation of the ISO model.
The most simple of the choices but requires the most configuration and setup. You must have the following criteria:
I recommend that the only IP traffic on this interface is iSCSI. This will ensure that you do not impact storage performance by overrunning the interface with data. I hope to post a future article showing how to configure QoS in the network if you must mix IP storage and IP data traffic on the same network adapter.
Not all adapters support LACP, and some vendors have a license that you purchase as an upgrade to enable it. It is a hardware feature and software drivers will not make LACP available. Check your network adapters supports LACP.
There are certain circumstances when LACP load balancing algorithm does not work optimally. Typically the driver makes a hash of the source and destination IP address and then sends all data down one port of the LACP bundle. Normally there are enough conversations to make this balance work satisfactorily, that is, there are enough unique source/destination pairs to evenly balance the load between the two interfaces, but if all your data from your Server to Storage is a single source/destination then, by default, only one gigabit ethernet port will be used. A key server might need more than one gigabit of bandwidth.
You can configure the Switch LACP AND the Server LACP (since each can only load balance the outbound frames) to use SRC IP / Port and DST IP / Port to make the full LACP bandwidth available to a single iSCSI initiator / target pair.
This is defined in the iSCSI RFC as the method for achieving high availability. The iSCSI initiator will initiate at least two TCP connections to the iSCSI target. Data will flow down the primary connection until a failure is detected and data will then be diverted to the second connection.
Since the connection is fully established, failover should be fast.
Note that you can still use LACP to improve bandwidth. e.g. if you need two gigabits of storage bandwidth, you would need 4 GB ports in two bundles of two to achieve this. This is a reasonable option for large VMware ESX servers which might host thirty or forty guest servers using the the iSCSI storage and you want to have maximum availability.
Dell has an example of how to configure this.
This solution is identical to the iSCSI Active / Standby, however, the iSCSI initiator and target include additional driver software that allow data to be transferred over either connection.
As before, LACP can be used to improve bandwidth by creating multi-gigabit connections.
We have looked at the three options for redundant iSCSI connections. Note that each of these solutions is possible with native network adapters, TOE or HBA (as covered in iSCSI Part 3 – Server Side – iSCSI Host Bus Adapters and IP Performance so that you can choose which performance level suits the requirements for the server.
From this information, it would seem that iSCSI multipathing Active/Standy is most effective for most people since it is simple to configure and operate. Choosing iSCSI Multipathing Active/Active would be a good choice for a server / storage combination where you can spend time testing the drivers and functionality. I am not sure but I think that both the storage and server would need to support this feature (can anyone confirm ? Leave a comment below if you know).
You could achieve redundancy using LACP on the server and storage (and also a performance improvement), but you must be able co-ordinate the switch configuration and the server configuration, plus modify the LACP load balancing type.
You could do both of course, using the following diagram which show using Active Standby iSCSI Multipathing AND LACP to create a two gigabits per second connection. Note that your server architecture, iSCSI driver and/or HBA would need to be able to generate this volume of traffic for it to be useful (and of course, the storage unit would also need to have the capability to move this volume of data).
With this type of implementation the physical connections should look like this:
This would require a total of four ethernet ports per server, with NICs that support LACP, and two switches (LACP support). This would ensure redundant paths and bandwidth.
Now that we have finished the considerations for the Server connections to the network, we can return to designing the network and considering bandwidth, ethernet performance and QoS requirements.
(1) iSCSI Performance Options – NetApp Whitepaper June 2005.
(2) Nortel has Multi Link Trunking which allows for LACP to span multiple switches. I do not recommend using Nortel switches to access this feature as my experience of Nortel is less than excellent. I understand that Nortel owns the patent and does not make it available to other companies.
ASAi – (ass-i) – The correct plural term for more than one Cisco ASA firewall appliances. [Read more…]
Put differently if you unplug one, the overall bandwidth does not drop to 12Gb/s etc. it will disconnect a single HBA port on a number of servers and force them to failover to the other path and FC-VC module.
It does not do any dynamic load balancing or anything like that – it is literally a physical port concentrator which is why it needs NPIV to pass through the WWNís from the physical blade HBAs.
What does that mean ?
Unplugging an FC cable from bay 3, port 4 will disconnect one of the HBA imageconnections to all of the blades in bays 4,8,10 and 14 and force the bladeís host OS to handle a failover to its secondary path via the FC-VC module in bay 4.
A key take away from this is that your blade hosts still need to run some kind of multi-pathing software, like MPIO or EMC PowerPath to handle the failover between paths – the FC-VC modules donít handle this for you.
See, easy isn’t it. Give me an IP connection any day, it much easier than this technical masterpiece – NOT.