Why did someone bother to create FCoE?
The requirement for Fibre Channel over Ethernet)FCoE is primarily to create an “opportunity” for existing Fibrechannel (FC) storage to connect to existing Ethernet and IP networks without leaving the comfort of FibreChannel. Customers have recognised that having two data networks, one Ethernet/IP, and the other Fibrechannel is duplication of cost and effort and are looking to rationalise to a single network.
The argument goes something like this: There is more than $55 Billion invested in legacy FC storage networks today, therefore we must provide a migration or interconnection strategy that allows for the integration of the existing storage platforms.
However, FCoE supporters now claim that FCoE is a better solution for block storage access over the network than alternative solutions such as iSCSI or NFS et al. Oft quoted reasons are backward compatibility to access legacy FC storage and ease of use, reuse skillsets and so on.
FCoE supporters don’t discuss FC/IP, possibly because Storage people don’t want to see their networks converge.
A bit of Fibrechannel History
Fibrechannel was designed as a block transport protocol to support lossless and low latency transfer of data across a data network.
When storage over IP was first proposed by the vendors, they approached the IEEE and IETF but were unhappy with time frames and integration requirements because people were insisting on compatibility with IP or Ethernet which didn’t suit the vendors proprietary viewpoint. There was much hue and cry over the development of a second standard when existing mechanisms already existed aka why not use IP ?
So they took their ideas to ANSI who allowed to them to create their own standard. Indeed, the physical and data link layers of the protocol are 100% compatible with Gigabit Ethernet, all that is different is the format of the frames. And customers got it marketed to them as the new Emperors clothes.
Who was it that said “the nice thing about standards is that there are so many to choose from” ?
Why is it a Layer 2 technology ? Layer 2 doesn’t scale.
To improve performance FC uses a Layer 2 type packet format. It takes much less CPU to encapsulate a payload in a Layer 2 format, than to add an abstraction layer, such as IP. However, when you make the choice to use Layer 2, you lose scale. This Fibrechannel networks are typically not viable beyond a fifty or so switch elements. Careful design and operational awareness might stretch this to a hundred or so, but you can’t get much more.
Why is Fibrechannel still important ?
Because it’s there. Fibrechannel isn’t important in the longer term future because iSCSI or some other form of Storage over IP will be dominant. All parties seem to agree on this, its just a matter of time (three years to ten years ?) until the technology builds up to support it.
FC solved the limitations of silicon and design in 1995, FCoE addresses the limitations of silicon, software and design in 2008.
But as Cisco will tell you, it’s the legacy investment that matters. There is literally thousands of tons of FC kit moldering away in todays data centres with more being installed every day in their own proprietary little clusters and isolated from sharing and integration unless you have that FC HBA.
Is Fibrechannel Dead ?
The short answer is yes. The long answer: FC isn’t going to disappear in the next three to five years given it’s market momentum. But there won’t be many new developments or speed increases because the cost of manufacturing FC is so high. The limited size of the FC deployments, mean that equipment is always going to be more expensive than Ethernet.
In practical terms FC as a technology has probably reached its mature stage, and successor technologies such as FCoE / iSCSI when joined with Converged Enhanced Ethernet mean that vendors will no longer make significant investments in FC.
Proof: Brocade, the biggest booster for FC, has bought Foundry to get access to the Ethernet switches. If FC has a future as a standalone entity, then why buy an Ethernet company ?
So What DOES FCoE do ?
The current marketing pitch looks something like this:
- Servers have too many cables
- cables and interfaces cost money in switch ports, FC ports and labour
- lets use a 10GbE to carry a 4Gb FC connection AND an IP Data connection
- then my server will only need two 10GbE ports
- I need new 10GbE switches
- I need Ethernet switches that have FC software and integration
- I might need switches that are lossless, low latency and a LOT OF BANDWIDTH
- I need a Quality of Service for Ethernet to guarantee service for shared service flows
So the big idea is that a single 10 Gigabit Ethernet cable will be able to support all your data AND storage needs. A magic trick indeed.
Storage versus Networks
It seems to me, that the motivation of the storage vendors to create and use FC was to create a proprietary protocol that only they supported. Say hello to Brocade – technology lock-in can be a viable business for a while.
OK, so some people will say FC switches are unique, and have special features for low loss blah blah blah. The reality is that this could equally have been done with Ethernet or IP. FC suited Brocade / MDS / McData so that they could stop Cisco / Nortel from owning this market at the time. Of course, Cisco eventually bought MDS and it visiting its revenge in Brocade. Laugh? oh yeah.
CIO’s have gradually realised that they have two networking teams. One supporting the IP/Ethernet network, and one supporting the FC network. Why are we building two networks, with two skill-sets and a significant amount of overheads ?
The joining of these two teams is one of the key stumbling blocks to FCoE adoption. Of course, iSCSI doesn’t have this problem ((note that iSCSI does create a problem with building lossless, low latency IP networks that have very high up times. This will require a change in data networking))
Why not choose Ethernet or IP at that time ?
Why indeed ? Cisco did choose iSCSI back in 2001 with their SN5420 iSCSI to FC router, and were a massive supporter of the technology as was Microsoft. It took a few years for Microsoft to release the software iSCSI initiator which then validated iSCSI as a technology. However, storage companies did not like the idea of losing control of their customers (and the revenue) and saw the creation of a new technology as a way to control their customers and make more revenue by selling additional products.
FC has some advantages of course, and at that time, Ethernet/IP networks were not quite ready to offer lossless & low latency networking. As we now know, these problems can be solved.
Comparing FCoE with iSCSI and NFS
The claimed advantages of FCoE are many, including a number of ShamWow! moments, however they don’t often explain that these are “marketing” advantages that can easily be mitigated with technology, design or just good old thinking.
FCoE has higher throughput
By encapsulating data in Ethernet frames instead of IP packets you have less overhead. The TCP/IP frame adds 80 odd bytes per packet which is lost bandwidth. This is of course true, and a 10Gb Ethernet connection will deliver 9 or so gigabits per second of storage bandwidth. They claim that this means “bandwidth is wasted”.
Wasted bandwidth has never been a problem. The Internet itself is testimony to this.
Insofar that IP is not an efficient protocol, this is true. But I don’t see people claiming that the Internet is massively inefficient. No, we just get some more bandwidth and get on with it. And this is exactly what is happening. 40Gb and 100Gb Ethernet is under development and will arrive shortly. By the time our servers are able to use 10Gb connections for storage and data, we should have even faster connections so that this so-called limitation doesn’t matter.
So, who cares about throughput ?
FCoE has lower latency
In the same way that encapsulating in Ethernet is faster that encapsulating in Ethernet/IP, then, yes, FCoE will have lower latency. And the network does not need to route IP packets, but forward Ethernet frames which is inherently quicker and less latent.
FCoE proponents would argue that these few milliseconds per transaction are significant. And for very high performance computing platforms, this argument would make sense. And for a few specific enterprise applications, this argument would also make sense.
But how many people, really need that sort of performance ? If you want a super computer, then go and buy Infiniband which has guaranteed latencies of less than two or three milliseconds. Or what about PCI-IOV ?
FCoE uses less CPU than IP encapsulations
This is also true. The segmentation of data and encapsulation of data into Ethernet frames requires less CPU than the creation of an TCP packet AND an Ethernet packet.
But thinking through this more completely, we realise the FCoE will only reach it’s full performance and features through the use of Converged Network Adapters (CNA). A CNA is a lot like a graphics card with dedicated processors performing the FC signalling and transfer functions, frame creation and a raft of other network functions that remove load from the CPU and Bus of the server.
Of course, when too much would be barely enough, the CNA does a lot of other FC functions such as virtualisation, management and monitoring as well.
But there are many hardware acceleration options for iSCSI in the market, and more are being developed. Because iSCSI is currently perceived as a lesser technology, there is less marketing dollars to promote these products. iSCSI Host Bus Adapters (HBAs) are being purchased is vast quantities by small to medium businesses who need to improve the performance of iSCSI and this means that scale of manufacturing is in favour of iSCSI HBAs in the longer term.
Don’t get confused with software iSCSI initators
Many people think that iSCSI software initators are the only implementation. This is because Microsoft and VMware released software products. As someone pointed out the other day, iSCSI is the ONLY storage technology offering a low cost starting point and can scale up.
If you haven’t looked at hardware acceleration for iSCSI, then you should immediately do this. The throughput and latency improvements are huge, and CPU on your servers will reduce enormously. In fact, iSCSI HBA’s are probably the key to its final success – and cost less than equivalent FCoE CNA because they are sold in much larger volumes.
You still need IP for Remote Copy, Synchronisation and Backup
If you need to perform offsite backup or replication, you will most likely use IP. Unless you can create a L2VPN …… yeah, make it easy for yourself. If you need IP anyway for management / control / administration and for backups, why not just use IP for everything.
iSCSI is well supported
If you are of the opinion that iSCSI is not supported at the high end, its time to think again. All of the major storage vendors have implemented iSCSI and many enhanced features.
For sure, there are several pieces of the iSCSI puzzle that need to happen before it can become mainstream. For example, high performance Host Bus Adapters, Converged Enhanced Ethernet standards for the Data Centre (the same ones as used for FCoE). Deeper adoption by organisations.
As someone pointed out to me this week, iSCSI is the ONLY technology that has a cheap option by using the software initiator. This means that many people will use it by default.
The most important aspect of iSCSI, is that it is the cheaper technology. If there is one thing that I have learned in twenty years of networking and IT, the cheap technology always wins in the end. Whether Beta vs VHS, Ethernet vs Token Ring / FDDI. It a big reason why I believe in iSCSI.
FCoE lacks key standards
At time of writing, the only FCoE implementations come from Cisco and you require a full networking implementation from Cisco. Minimum cost = $$$$SHEDLOADS. This is because the key Ethernet standards that support FCoE are still moving through the IEEE. Collectively these are known as Converged Enhanced Ethernet (CEE) and Cisco is already developing their own interim versions called Data Centre Ethernet™ (DCE). Cisco claims that they will support all standard via software upgrade but colour me skeptical. I have been caught out before where a hardware upgrade was needed.
These standards are also not going well. Recently we saw TRILL initiated at the IETF which is directly competing against the 802.1aq standard which appears to have stalled. This may be a sign of major problems with agreements on the standards.
Secondly, FCoE may not be a Cisco-only effort, but the line-up of Emulex, Intel, Qlogic, Vmware, EMC and Netapp is hardly an industry-wide adoption. What about IBM, HP, Sun, Dell? What about HDS? A new protocol with new HBA cards is precisely in the best interests of Emulex, Intel and Qlogic as they tag along on Cisco marketing efforts. Its a free ride for them.
Remember that HP bought Left Hand networks and DELL bought Equal Logic, for multibillions, to get iSCSI technology into their portfolio and their engineers.
FCoE requires special, new and expensive, hardware to do ANYTHING
FCoE is still Fibrechannel, therefore you still need a FC controller or switch to configure all the usual FC functions like ISL, management. For Cisco this means buying a Cisco Nexus 5000, and possibly Nexus 2000 as edge devices.
Compare this to iSCSI where you CAN use your existing Ethernet infrastructure and progressively plan an upgrade to new hardware, possibly around the Cisco Nexus 7000 as a core switch and the upgrade path is much more organic, and fit into existing budgeting and project processes.
FC has historical software support, but now iSCSI is equally important
Early VMware adopters will know that VMotion, VMDR and a raft other features were supported on FC first because that is what their customers expected. But more recently, as their customer base has expanded into the wider marketplace and SMB, these features are also supported on iSCSI, NFS etc. Take the time to research why many people are FC biased, and you will often find that its historical. That is, “that’s how its always been done around here“. But that doesn’t always mean that’s the way it will be in future.
FCoE doesn’t scale
One of the most hidden, least realised and unknown shortcomings of FCoE is the lack of scale. Encapsulating FC in Ethernet is OK, but the requirements for the Ethernet signalling for CEE (or DCE) are not better or any improvement on the current standards. That is, we cannot create data centres using Ethernet VLANs with more than fifty to a hundred switches before we lose stability. ((Note this is also true for Fibrechannel, however, data centres with more than fifty FC switches in a single network is exceptional…and unlikely))
Does this really matter ?
Evidence: Cisco has introduced what they call Fabric Extenders (Nexus 2000 & 2100) which create “distributed switches”. That is, the Nexus 5000 is the core and the Nexus 2000/2100 act as “blades” of the Nexus. There are two possibilities here, either its a low cost way of scaling the Nexus 5000 to meet customer requirements or its an admission that FCoE cannot scale and the need to keep L2 switch domains very small is high priority.
This might be less of a problem if we can use very large switches, but building lossless, low latency, Fibrechannel-capable is a really expensive proposition. Therefore, just like Fibrechannel, Ethernet storage networks will mostly consist of small switches for storage connections only.
Evidence: 802.1aq/TRILL does not scale. Implementations have a limit of about 100 switches, which isn’t a lot for good sized data centre and certainly not enough for Cloud Computing.
The path to FCoE is not certain. Cisco has managed to force competitor Brocade into delivering Storage over Ethernet (FCoE) and making massive marketing push with its Data Centre 3.0 strategy but don’t be blinded by this ostentatious display of wealth. Be smart and make your own choices by considering what will work in your Data Centre.
FCoE is by no means the default or only choice for storage networking, and I believe it will be optional for the vast majority of the computer rooms because of late standards, high cost and significant learning curve. Look at iSCSI (use HBA’s for acceleration) and NFS carefully because of their lower cost.
This may mean that FCoE will only ever be a niche technology for legacy FC platform integration and not as a standalone high performance technology.