The case against FCoE AND Fibrechannel – a reasonably complete view

Why did someone bother to create FCoE?

The requirement for Fibre Channel over Ethernet)FCoE is primarily to create an “opportunity” for existing Fibrechannel (FC) storage to connect to existing Ethernet and IP networks without leaving the comfort of FibreChannel. Customers have recognised that having two data networks, one Ethernet/IP, and the other Fibrechannel is duplication of cost and effort and are looking to rationalise to a single network.

The argument goes something like this: There is more than $55 Billion invested in legacy FC storage networks today, therefore we must provide a migration or interconnection strategy that allows for the integration of the existing storage platforms.

However, FCoE supporters now claim that FCoE is a better solution for block storage access over the network than alternative solutions such as iSCSI or NFS et al. Oft quoted reasons are backward compatibility to access legacy FC storage and ease of use, reuse skillsets and so on.

FCoE supporters don’t discuss FC/IP, possibly because Storage people don’t want to see their networks converge.

A bit of Fibrechannel History

Fibrechannel was designed as a block transport protocol to support lossless and low latency transfer of data across a data network.

When storage over IP was first proposed by the vendors, they approached the IEEE and IETF but were unhappy with time frames and integration requirements because people were insisting on compatibility with IP or Ethernet which didn’t suit the vendors proprietary viewpoint. There was much hue and cry over the development of a second standard when existing mechanisms already existed aka why not use IP ?

So they took their ideas to ANSI who allowed to them to create their own standard. Indeed, the physical and data link layers of the protocol are 100% compatible with Gigabit Ethernet, all that is different is the format of the frames. And customers got it marketed to them as the new Emperors clothes.

Who was it that said “the nice thing about standards is that there are so many to choose from” ?

Why is it a Layer 2 technology ? Layer 2 doesn’t scale.

To improve performance FC uses a Layer 2 type packet format. It takes much less CPU to encapsulate a payload in a Layer 2 format, than to add an abstraction layer, such as IP. However, when you make the choice to use Layer 2, you lose scale. This Fibrechannel networks are typically not viable beyond a fifty or so switch elements. Careful design and operational awareness might stretch this to a hundred or so, but you can’t get much more.

Why is Fibrechannel still important ?

Because it’s there. Fibrechannel isn’t important in the longer term future because iSCSI or some other form of Storage over IP will be dominant. All parties seem to agree on this, its just a matter of time (three years to ten years ?) until the technology builds up to support it.

FC solved the limitations of silicon and design in 1995, FCoE addresses the limitations of silicon, software and design in 2008.

fcoe-case-against-1a.png

But as Cisco will tell you, it’s the legacy investment that matters. There is literally thousands of tons of FC kit moldering away in todays data centres with more being installed every day in their own proprietary little clusters and isolated from sharing and integration unless you have that FC HBA.

Is Fibrechannel Dead ?

The short answer is yes. The long answer: FC isn’t going to disappear in the next three to five years given it’s market momentum. But there won’t be many new developments or speed increases because the cost of manufacturing FC is so high. The limited size of the FC deployments, mean that equipment is always going to be more expensive than Ethernet.

In practical terms FC as a technology has probably reached its mature stage, and successor technologies such as FCoE / iSCSI when joined with Converged Enhanced Ethernet mean that vendors will no longer make significant investments in FC.

Proof: Brocade, the biggest booster for FC, has bought Foundry to get access to the Ethernet switches. If FC has a future as a standalone entity, then why buy an Ethernet company ? tweet

So What DOES FCoE do ?

The current marketing pitch looks something like this:

  • Servers have too many cables
  • cables and interfaces cost money in switch ports, FC ports and labour
  • lets use a 10GbE to carry a 4Gb FC connection AND an IP Data connection
  • then my server will only need two 10GbE ports

However:

  • I need new 10GbE switches
  • I need Ethernet switches that have FC software and integration
  • I might need switches that are lossless, low latency and a LOT OF BANDWIDTH
  • I need a Quality of Service for Ethernet to guarantee service for shared service flows

So the big idea is that a single 10 Gigabit Ethernet cable will be able to support all your data AND storage needs. A magic trick indeed.

Storage versus Networks

It seems to me, that the motivation of the storage vendors to create and use FC was to create a proprietary protocol that only they supported. Say hello to Brocade – technology lock-in can be a viable business for a while.

OK, so some people will say FC switches are unique, and have special features for low loss blah blah blah. The reality is that this could equally have been done with Ethernet or IP. FC suited Brocade / MDS / McData so that they could stop Cisco / Nortel from owning this market at the time. Of course, Cisco eventually bought MDS and it visiting its revenge in Brocade. Laugh? oh yeah. tweet

CIO’s have gradually realised that they have two networking teams. One supporting the IP/Ethernet network, and one supporting the FC network. Why are we building two networks, with two skill-sets and a significant amount of overheads ?

The joining of these two teams is one of the key stumbling blocks to FCoE adoption. Of course, iSCSI doesn’t have this problem ((note that iSCSI does create a problem with building lossless, low latency IP networks that have very high up times. This will require a change in data networking))

Why not choose Ethernet or IP at that time ?

Why indeed ? Cisco did choose iSCSI back in 2001 with their SN5420 iSCSI to FC router, and were a massive supporter of the technology as was Microsoft. It took a few years for Microsoft to release the software iSCSI initiator which then validated iSCSI as a technology. However, storage companies did not like the idea of losing control of their customers (and the revenue) and saw the creation of a new technology as a way to control their customers and make more revenue by selling additional products.

FC has some advantages of course, and at that time, Ethernet/IP networks were not quite ready to offer lossless & low latency networking. As we now know, these problems can be solved.

Comparing FCoE with iSCSI and NFS

The claimed advantages of FCoE are many, including a number of ShamWow! moments, however they don’t often explain that these are “marketing” advantages that can easily be mitigated with technology, design or just good old thinking.

FCoE has higher throughput

By encapsulating data in Ethernet frames instead of IP packets you have less overhead. The TCP/IP frame adds 80 odd bytes per packet which is lost bandwidth. This is of course true, and a 10Gb Ethernet connection will deliver 9 or so gigabits per second of storage bandwidth. They claim that this means “bandwidth is wasted”.

Wasted bandwidth has never been a problem. The Internet itself is testimony to this. tweet

Insofar that IP is not an efficient protocol, this is true. But I don’t see people claiming that the Internet is massively inefficient. No, we just get some more bandwidth and get on with it. And this is exactly what is happening. 40Gb and 100Gb Ethernet is under development and will arrive shortly. By the time our servers are able to use 10Gb connections for storage and data, we should have even faster connections so that this so-called limitation doesn’t matter.

So, who cares about throughput ?

FCoE has lower latency

In the same way that encapsulating in Ethernet is faster that encapsulating in Ethernet/IP, then, yes, FCoE will have lower latency. And the network does not need to route IP packets, but forward Ethernet frames which is inherently quicker and less latent.

FCoE proponents would argue that these few milliseconds per transaction are significant. And for very high performance computing platforms, this argument would make sense. And for a few specific enterprise applications, this argument would also make sense.

But how many people, really need that sort of performance ? If you want a super computer, then go and buy Infiniband which has guaranteed latencies of less than two or three milliseconds. Or what about PCI-IOV ?

FCoE uses less CPU than IP encapsulations

This is also true. The segmentation of data and encapsulation of data into Ethernet frames requires less CPU than the creation of an TCP packet AND an Ethernet packet.

But thinking through this more completely, we realise the FCoE will only reach it’s full performance and features through the use of Converged Network Adapters (CNA). A CNA is a lot like a graphics card with dedicated processors performing the FC signalling and transfer functions, frame creation and a raft of other network functions that remove load from the CPU and Bus of the server.

Of course, when too much would be barely enough, the CNA does a lot of other FC functions such as virtualisation, management and monitoring as well.

But there are many hardware acceleration options for iSCSI in the market, and more are being developed. Because iSCSI is currently perceived as a lesser technology, there is less marketing dollars to promote these products. iSCSI Host Bus Adapters (HBAs) are being purchased is vast quantities by small to medium businesses who need to improve the performance of iSCSI and this means that scale of manufacturing is in favour of iSCSI HBAs in the longer term.

Don’t get confused with software iSCSI initators

Many people think that iSCSI software initators are the only implementation. This is because Microsoft and VMware released software products. As someone pointed out the other day, iSCSI is the ONLY storage technology offering a low cost starting point and can scale up.

If you haven’t looked at hardware acceleration for iSCSI, then you should immediately do this. The throughput and latency improvements are huge, and CPU on your servers will reduce enormously. In fact, iSCSI HBA’s are probably the key to its final success – and cost less than equivalent FCoE CNA because they are sold in much larger volumes.

You still need IP for Remote Copy, Synchronisation and Backup

If you need to perform offsite backup or replication, you will most likely use IP. Unless you can create a L2VPN …… yeah, make it easy for yourself. If you need IP anyway for management / control / administration and for backups, why not just use IP for everything.

iSCSI is well supported

If you are of the opinion that iSCSI is not supported at the high end, its time to think again. All of the major storage vendors have implemented iSCSI and many enhanced features.

For sure, there are several pieces of the iSCSI puzzle that need to happen before it can become mainstream. For example, high performance Host Bus Adapters, Converged Enhanced Ethernet standards for the Data Centre (the same ones as used for FCoE). Deeper adoption by organisations.

As someone pointed out to me this week, iSCSI is the ONLY technology that has a cheap option by using the software initiator. This means that many people will use it by default.

The most important aspect of iSCSI, is that it is the cheaper technology. If there is one thing that I have learned in twenty years of networking and IT, the cheap technology always wins in the end. Whether Beta vs VHS, Ethernet vs Token Ring / FDDI. It a big reason why I believe in iSCSI.

FCoE lacks key standards

At time of writing, the only FCoE implementations come from Cisco and you require a full networking implementation from Cisco. Minimum cost = $$$$SHEDLOADS. This is because the key Ethernet standards that support FCoE are still moving through the IEEE. Collectively these are known as Converged Enhanced Ethernet (CEE) and Cisco is already developing their own interim versions called Data Centre Ethernet™ (DCE). Cisco claims that they will support all standard via software upgrade but colour me skeptical. I have been caught out before where a hardware upgrade was needed.

These standards are also not going well. Recently we saw TRILL initiated at the IETF which is directly competing against the 802.1aq standard which appears to have stalled. This may be a sign of major problems with agreements on the standards.

Secondly, FCoE may not be a Cisco-only effort, but the line-up of Emulex, Intel, Qlogic, Vmware, EMC and Netapp is hardly an industry-wide adoption. What about IBM, HP, Sun, Dell? What about HDS? A new protocol with new HBA cards is precisely in the best interests of Emulex, Intel and Qlogic as they tag along on Cisco marketing efforts. Its a free ride for them.

Remember that HP bought Left Hand networks and DELL bought Equal Logic, for multibillions, to get iSCSI technology into their portfolio and their engineers.

FCoE requires special, new and expensive, hardware to do ANYTHING

FCoE is still Fibrechannel, therefore you still need a FC controller or switch to configure all the usual FC functions like ISL, management. For Cisco this means buying a Cisco Nexus 5000, and possibly Nexus 2000 as edge devices.

Compare this to iSCSI where you CAN use your existing Ethernet infrastructure and progressively plan an upgrade to new hardware, possibly around the Cisco Nexus 7000 as a core switch and the upgrade path is much more organic, and fit into existing budgeting and project processes.

FC has historical software support, but now iSCSI is equally important

Early VMware adopters will know that VMotion, VMDR and a raft other features were supported on FC first because that is what their customers expected. But more recently, as their customer base has expanded into the wider marketplace and SMB, these features are also supported on iSCSI, NFS etc. Take the time to research why many people are FC biased, and you will often find that its historical. That is, “that’s how its always been done around here“. But that doesn’t always mean that’s the way it will be in future.

FCoE doesn’t scale

One of the most hidden, least realised and unknown shortcomings of FCoE is the lack of scale. Encapsulating FC in Ethernet is OK, but the requirements for the Ethernet signalling for CEE (or DCE) are not better or any improvement on the current standards. That is, we cannot create data centres using Ethernet VLANs with more than fifty to a hundred switches before we lose stability. ((Note this is also true for Fibrechannel, however, data centres with more than fifty FC switches in a single network is exceptional…and unlikely))

Does this really matter ?

Evidence: Cisco has introduced what they call Fabric Extenders (Nexus 2000 & 2100) which create “distributed switches”. That is, the Nexus 5000 is the core and the Nexus 2000/2100 act as “blades” of the Nexus. There are two possibilities here, either its a low cost way of scaling the Nexus 5000 to meet customer requirements or its an admission that FCoE cannot scale and the need to keep L2 switch domains very small is high priority.

This might be less of a problem if we can use very large switches, but building lossless, low latency, Fibrechannel-capable is a really expensive proposition. Therefore, just like Fibrechannel, Ethernet storage networks will mostly consist of small switches for storage connections only.

Evidence: 802.1aq/TRILL does not scale. Implementations have a limit of about 100 switches, which isn’t a lot for good sized data centre and certainly not enough for Cloud Computing.

Conclusion

The path to FCoE is not certain. Cisco has managed to force competitor Brocade into delivering Storage over Ethernet (FCoE) and making massive marketing push with its Data Centre 3.0 strategy but don’t be blinded by this ostentatious display of wealth. Be smart and make your own choices by considering what will work in your Data Centre.

FCoE is by no means the default or only choice for storage networking, and I believe it will be optional for the vast majority of the computer rooms because of late standards, high cost and significant learning curve. Look at iSCSI (use HBA’s for acceleration) and NFS carefully because of their lower cost.

This may mean that FCoE will only ever be a niche technology for legacy FC platform integration and not as a standalone high performance technology.

About Greg Ferro

Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualization. He has over 20 years in IT, in wide range of employers working as a freelance consultant including Finance, Service Providers and Online Companies. He is CCIE#6920 and has a few ideas about the world, but not enough to really count.

He is a host on the Packet Pushers Podcast, blogger at EtherealMind.com and on Twitter @etherealmind and Google Plus

You can contact Greg via the site contact page.

  • Dmitri

    Greg,

    The cost of building adequately high-performance IP infrastructure to support massively scalable IP network (for iSCSI) will cost more than CEE/DCE. Your IP infrastructure will need something to ride on, and that “something” is, um, inevitably Ethernet, for your iSCSI you will need high-performance Ethernet infrastructure in place AS WELL as IP (which will ride on top of it).

    Really, the *only* reason for iSCSI is the low entry cost. As soon as you start hitting against your CPU limitations caused by HBAs without iSCSI hardware acceleration, you will run into exactly the same trouble as with FCoE – upgrage, upgrade, upgrade – HBAs, switches, routers, etc.

    I was frankly surprised to see you say that FC switches and ports are more expensive than Ethernet – firstly, FC had ports faster than 1G before 10GE came around and prices came down to anything resembling “reasonable”, and secondly, price per port was very similar or cheaper than 1G Ethernet (for 2/4G FC).

    To summarize, in my opinion there is a place for both iSCSI and FCoE. First is for SMB, second is for those running out of juice on their 4/8G FC.

    • mrz

      What application and storage arrays drive more than 1GE? Mozilla has a 30 host ESX enviro and not a single host drives more than 100-200Mbps.

      • Dmitri

        EMC Symmetrix, for one. Don’t forget that FC is not only for Data Centre and is very actively used in high-end workstation environment – for example, a single uncompressed HD video stream takes up 1.5Gbps, which is already way over what 1GE can handle, and it is not unusual in video production environment to have more than one going at the same time.

  • mrz

    “If you havenít looked at hardware acceleration for iSCSI, then you should immediately do this.”

    Surprised to read that – in our tests at Mozilla with modern 4-core servers, QLogic HBAs haven’t made any difference in I/O performance. I’ve generally found them to be a waste of money with no noticeable gain.

    • mgk

      that is exacty my experience… we are using the intel server GE NICs (with the upgraded firmware to enable iSCSI boot) and performance difference compared to iSCSI HBAs is very small, certainly not worth the addional money. (the intel desktop ones are not as good)

      If you use some cheap 5$ NIC without a TCP offload engine the difference will be a lot bigger though…

  • Jeremy Filliben

    Greg,

    I’m glad to see someone presenting the other side of the story. FCoE certainly has its target niches, but iSCSI is clearly going to be the general purpose solution, when traditional NAS won’t work. If in 5 years, FCoE is ubiquitous, we’ll all have spent way too much money on our SANs!

    One small correction (sort of).. Nexus 2k does not currently support FCoE; it is only a 1gb Ethernet solution. I believe FCoE is on the roadmap for the 2k though.

    Thanks for writing this!
    Jeremy

  • http://www.internetworkexpert.org Brad Hedlund

    Why would the Nexus 2148 need to support FCoE when there is no 1GE CNA available? FCoE doesn’t make any sense at 1GE. Does FCoE makes sense with a 10GE fabric extender? You betcha :-)

    • http://etherealmind.com Greg Ferro

      While I agree with you, I think that many people will choose to implement FCoE on 1Gig because it is cheaper than purchasing a FC HBA. At the moment the vendors (Emulex/Qlogic) are focusing on the high value/high profit 10GB CNA market but it’s only a matter of time until they release a 1GbE CNA. Although iSCSI would make more sense, some people will need to connect to their legacy FC arrays because they haven’t upgraded them, or don’t have support for iSCSI. The transition from block storage over FC, to block storage over Ethernet or IP isn’t going to be easy.

      • http://www.internetworkexpert.org Brad Hedlund

        If there is demand for a 1GE CNA as you predict, AFAIK, the hardware on Nexus 2148 is FCoE capable. It would just require the right knobs to be turned on in software.

        • http://etherealmind.com Greg Ferro

          I am thinking that Dell would like to offer FCoE in their mid range servers such as the1950/2950 series that are used by a lot of mid-size companies. People who may not perceive the value of the Cisco UF but want to move away from FC.

  • http://livingonthecloud.blogspot.com JR

    Greg

    Thanks very much for posting this article on FCoE. I’m currently re-evaluating our storage strategy moving forward (we’re currently a Fibre Channel environment) and find the idea of iSCSI and/or NFS an appealing technology for our VMware environment. We won’t need dedicated fibre switches, can utilise our existing Cisco network switches and make the whole thing easier to manage.

    Your thoughts on why FCoE is not a good idea are very thought-provoking and it’s good to see you expand on the occasional comment made on PacketPushers.

    JR

  • GKM

    People who keep saying that FC will die soon are those who never understood why FC is living still today. It is the sheer reliability, simplicity and deterministic performance that gives FC the edge other other technologies like iScsi, or 10Gb eth. Datacenters never compromise reliability for the cost. FCoE is said to be penetrating into the market for the past 5-6 years, but it never happened till now predominantly due to the high cost, and of course Ethernet by default is a lossy protocol (tho people claim that the Enhanced Eth has all the robust pause mechanisms, nothing matches the BB_Credit and EE_Credit mechanism of FC still). At the best case the FCOE can replace the HBAs to CNAs in the edge to help server side consolidation, but nothing replaces the Fc in the core. Analysts has been predicting for the past 5+ years that FC would be dead, and they will keep saying the same story repeatedly for the next N+ years to keep their jobs going ;) But the shipment of FC ports is increasing @ 8-10% every year. For people who hate FC, the FC is like cockroach. It might be the last technology to die. For people who love FC, it is the Angel because it never lets them down. iScsi is said to be upcoming technology for the past 10+ years, and all that it can do it to play a subtul role in the lo-end storage where the reliability and performance is not a constraint.

Subscribe For Weekly Updates by Email

Get a Weekly Summary of Latest Articles and Posts to your Email Inbox Every Sunday

Thanks for signing up. Look for the email from MailChimp & make sure you confirm your email address. You may need to check your spam or gmail settings to be sure of receiving the email.

Note: You can unsubscribe at any time using the link at the bottom of every email.