The Case Against FCoE and Fibrechannel — a Reasonably Complete View

Why did someone bother to cre­ate FCoE?

The require­ment for Fibre Channel over Ethernet)FCoE is primar­ily to cre­ate an “oppor­tun­ity” for exist­ing Fibrechannel (FC) stor­age to con­nect to exist­ing Ethernet and IP net­works without leav­ing the com­fort of FibreChannel. Customers have recog­nised that hav­ing two data net­works, one Ethernet/​IP, and the other Fibrechannel is duplic­a­tion of cost and effort and are look­ing to ration­al­ise to a single network.

The argu­ment goes some­thing like this: There is more than $55 Billion inves­ted in leg­acy FC stor­age net­works today, there­fore we must provide a migra­tion or inter­con­nec­tion strategy that allows for the integ­ra­tion of the exist­ing stor­age platforms.

However, FCoE sup­port­ers now claim that FCoE is a bet­ter solu­tion for block stor­age access over the net­work than altern­at­ive solu­tions such as iSCSI or NFS et al. Oft quoted reas­ons are back­ward com­pat­ib­il­ity to access leg­acy FC stor­age and ease of use, reuse skill­sets and so on.

FCoE sup­port­ers don’t dis­cuss FC/​IP, pos­sibly because Storage people don’t want to see their net­works converge.

A bit of Fibrechannel History

Fibrechannel was designed as a block trans­port pro­tocol to sup­port lossless and low latency trans­fer of data across a data network.

When stor­age over IP was first pro­posed by the vendors, they approached the IEEE and IETF but were unhappy with time frames and integ­ra­tion require­ments because people were insist­ing on com­pat­ib­il­ity with IP or Ethernet which didn’t suit the vendors pro­pri­et­ary view­point. There was much hue and cry over the devel­op­ment of a second stand­ard when exist­ing mech­an­isms already exis­ted aka why not use IP ?

So they took their ideas to ANSI who allowed to them to cre­ate their own stand­ard. Indeed, the phys­ical and data link lay­ers of the pro­tocol are 100% com­pat­ible with Gigabit Ethernet, all that is dif­fer­ent is the format of the frames. And cus­tom­ers got it mar­keted to them as the new Emperors clothes.

Who was it that said “the nice thing about stand­ards is that there are so many to choose from” ?

Why is it a Layer 2 tech­no­logy ? Layer 2 doesn’t scale.

To improve per­form­ance FC uses a Layer 2 type packet format. It takes much less CPU to encap­su­late a pay­load in a Layer 2 format, than to add an abstrac­tion layer, such as IP. However, when you make the choice to use Layer 2, you lose scale. This Fibrechannel net­works are typ­ic­ally not viable bey­ond a fifty or so switch ele­ments. Careful design and oper­a­tional aware­ness might stretch this to a hun­dred or so, but you can’t get much more.

Why is Fibrechannel still important ?

Because it’s there. Fibrechannel isn’t import­ant in the longer term future because iSCSI or some other form of Storage over IP will be dom­in­ant. All parties seem to agree on this, its just a mat­ter of time (three years to ten years ?) until the tech­no­logy builds up to sup­port it.

FC solved the lim­it­a­tions of sil­icon and design in 1995, FCoE addresses the lim­it­a­tions of sil­icon, soft­ware and design in 2008.

But as Cisco will tell you, it’s the leg­acy invest­ment that mat­ters. There is lit­er­ally thou­sands of tons of FC kit mol­der­ing away in todays data centres with more being installed every day in their own pro­pri­et­ary little clusters and isol­ated from shar­ing and integ­ra­tion unless you have that FC HBA.

Is Fibrechannel Dead ?

The short answer is yes. The long answer: FC isn’t going to dis­ap­pear in the next three to five years given it’s mar­ket momentum. But there won’t be many new devel­op­ments or speed increases because the cost of man­u­fac­tur­ing FC is so high. The lim­ited size of the FC deploy­ments, mean that equip­ment is always going to be more expens­ive than Ethernet.

In prac­tical terms FC as a tech­no­logy has prob­ably reached its mature stage, and suc­cessor tech­no­lo­gies such as FCoE /​ iSCSI when joined with Converged Enhanced Ethernet mean that vendors will no longer make sig­ni­fic­ant invest­ments in FC.

Proof: Brocade, the biggest booster for FC, has bought Foundry to get access to the Ethernet switches. If FC has a future as a stan­dalone entity, then why buy an Ethernet company ?

So What DOES FCoE do ?

The cur­rent mar­ket­ing pitch looks some­thing like this:

• Servers have too many cables
• cables and inter­faces cost money in switch ports, FC ports and labour
• lets use a 10GbE to carry a 4Gb FC con­nec­tion AND an IP Data connection
• then my server will only need two 10GbE ports

However:

• I need new 10GbE switches
• I need Ethernet switches that have FC soft­ware and integration
• I might need switches that are lossless, low latency and a LOT OF BANDWIDTH
• I need a Quality of Service for Ethernet to guar­an­tee ser­vice for shared ser­vice flows

So the big idea is that a single 10 Gigabit Ethernet cable will be able to sup­port all your data AND stor­age needs. A magic trick indeed.

Storage versus Networks

It seems to me, that the motiv­a­tion of the stor­age vendors to cre­ate and use FC was to cre­ate a pro­pri­et­ary pro­tocol that only they sup­por­ted. Say hello to Brocade — tech­no­logy lock-​​in can be a viable busi­ness for a while.

OK, so some people will say FC switches are unique, and have spe­cial fea­tures for low loss blah blah blah. The real­ity is that this could equally have been done with Ethernet or IP. FC suited Brocade /​ MDS /​ McData so that they could stop Cisco /​ Nortel from own­ing this mar­ket at the time. Of course, Cisco even­tu­ally bought MDS and it vis­it­ing its revenge in Brocade. Laugh? oh yeah.

CIO’s have gradu­ally real­ised that they have two net­work­ing teams. One sup­port­ing the IP/​Ethernet net­work, and one sup­port­ing the FC net­work. Why are we build­ing two net­works, with two skill-​​sets and a sig­ni­fic­ant amount of overheads ?

The join­ing of these two teams is one of the key stum­bling blocks to FCoE adop­tion. Of course, iSCSI doesn’t have this prob­lem¹

Why not choose Ethernet or IP at that time ?

Why indeed ? Cisco did choose iSCSI back in 2001 with their SN5420 iSCSI to FC router, and were a massive sup­porter of the tech­no­logy as was Microsoft. It took a few years for Microsoft to release the soft­ware iSCSI ini­ti­ator which then val­id­ated iSCSI as a tech­no­logy. However, stor­age com­pan­ies did not like the idea of los­ing con­trol of their cus­tom­ers (and the rev­enue) and saw the cre­ation of a new tech­no­logy as a way to con­trol their cus­tom­ers and make more rev­enue by selling addi­tional products.

FC has some advant­ages of course, and at that time, Ethernet/​IP net­works were not quite ready to offer lossless & low latency net­work­ing. As we now know, these prob­lems can be solved.

Comparing FCoE with iSCSI and NFS

The claimed advant­ages of FCoE are many, includ­ing a num­ber of ShamWow! moments, how­ever they don’t often explain that these are “mar­ket­ing” advant­ages that can eas­ily be mit­ig­ated with tech­no­logy, design or just good old thinking.

FCoE has higher throughput

By encap­su­lat­ing data in Ethernet frames instead of IP pack­ets you have less over­head. The TCP/​IP frame adds 80 odd bytes per packet which is lost band­width. This is of course true, and a 10Gb Ethernet con­nec­tion will deliver 9 or so gig­abits per second of stor­age band­width. They claim that this means “band­width is wasted”.

Wasted band­width has never been a prob­lem. The Internet itself is testi­mony to this.

Insofar that IP is not an effi­cient pro­tocol, this is true. But I don’t see people claim­ing that the Internet is massively inef­fi­cient. No, we just get some more band­width and get on with it. And this is exactly what is hap­pen­ing. 40Gb and 100Gb Ethernet is under devel­op­ment and will arrive shortly. By the time our serv­ers are able to use 10Gb con­nec­tions for stor­age and data, we should have even faster con­nec­tions so that this so-​​called lim­it­a­tion doesn’t matter.

So, who cares about throughput ?

FCoE has lower latency

In the same way that encap­su­lat­ing in Ethernet is faster that encap­su­lat­ing in Ethernet/​IP, then, yes, FCoE will have lower latency. And the net­work does not need to route IP pack­ets, but for­ward Ethernet frames which is inher­ently quicker and less latent.

FCoE pro­ponents would argue that these few mil­li­seconds per trans­ac­tion are sig­ni­fic­ant. And for very high per­form­ance com­put­ing plat­forms, this argu­ment would make sense. And for a few spe­cific enter­prise applic­a­tions, this argu­ment would also make sense.

But how many people, really need that sort of per­form­ance ? If you want a super com­puter, then go and buy Infiniband which has guar­an­teed laten­cies of less than two or three mil­li­seconds. Or what about PCI-​​IOV ?

FCoE uses less CPU than IP encapsulations

This is also true. The seg­ment­a­tion of data and encap­su­la­tion of data into Ethernet frames requires less CPU than the cre­ation of an TCP packet AND an Ethernet packet.

But think­ing through this more com­pletely, we real­ise the FCoE will only reach it’s full per­form­ance and fea­tures through the use of Converged Network Adapters (CNA). A CNA is a lot like a graph­ics card with ded­ic­ated pro­cessors per­form­ing the FC sig­nalling and trans­fer func­tions, frame cre­ation and a raft of other net­work func­tions that remove load from the CPU and Bus of the server.

Of course, when too much would be barely enough, the CNA does a lot of other FC func­tions such as vir­tu­al­isa­tion, man­age­ment and mon­it­or­ing as well.

But there are many hard­ware accel­er­a­tion options for iSCSI in the mar­ket, and more are being developed. Because iSCSI is cur­rently per­ceived as a lesser tech­no­logy, there is less mar­ket­ing dol­lars to pro­mote these products. iSCSI Host Bus Adapters (HBAs) are being pur­chased is vast quant­it­ies by small to medium busi­nesses who need to improve the per­form­ance of iSCSI and this means that scale of man­u­fac­tur­ing is in favour of iSCSI HBAs in the longer term.

Don’t get con­fused with soft­ware iSCSI initators

Many people think that iSCSI soft­ware init­at­ors are the only imple­ment­a­tion. This is because Microsoft and VMware released soft­ware products. As someone poin­ted out the other day, iSCSI is the ONLY stor­age tech­no­logy offer­ing a low cost start­ing point and can scale up.

If you haven’t looked at hard­ware accel­er­a­tion for iSCSI, then you should imme­di­ately do this. The through­put and latency improve­ments are huge, and CPU on your serv­ers will reduce enorm­ously. In fact, iSCSI HBA’s are prob­ably the key to its final suc­cess — and cost less than equi­val­ent FCoE CNA because they are sold in much lar­ger volumes.

You still need IP for Remote Copy, Synchronisation and Backup

If you need to per­form off­s­ite backup or rep­lic­a­tion, you will most likely use IP. Unless you can cre­ate a L2VPN .….. yeah, make it easy for your­self. If you need IP any­way for man­age­ment /​ con­trol /​ admin­is­tra­tion and for backups, why not just use IP for everything.

iSCSI is well supported

If you are of the opin­ion that iSCSI is not sup­por­ted at the high end, its time to think again. All of the major stor­age vendors have imple­men­ted iSCSI and many enhanced features.

For sure, there are sev­eral pieces of the iSCSI puzzle that need to hap­pen before it can become main­stream. For example, high per­form­ance Host Bus Adapters, Converged Enhanced Ethernet stand­ards for the Data Centre (the same ones as used for FCoE). Deeper adop­tion by organisations.

As someone poin­ted out to me this week, iSCSI is the ONLY tech­no­logy that has a cheap option by using the soft­ware ini­ti­ator. This means that many people will use it by default.

The most import­ant aspect of iSCSI, is that it is the cheaper tech­no­logy. If there is one thing that I have learned in twenty years of net­work­ing and IT, the cheap tech­no­logy always wins in the end. Whether Beta vs VHS, Ethernet vs Token Ring /​ FDDI. It a big reason why I believe in iSCSI.

FCoE lacks key standards

At time of writ­ing, the only FCoE imple­ment­a­tions come from Cisco and you require a full net­work­ing imple­ment­a­tion from Cisco. Minimum cost = $$$$SHEDLOADS. This is because the key Ethernet stand­ards that sup­port FCoE are still mov­ing through the IEEE. Collectively these are known as Converged Enhanced Ethernet (CEE) and Cisco is already devel­op­ing their own interim ver­sions called Data Centre Ethernet™ (DCE). Cisco claims that they will sup­port all stand­ard via soft­ware upgrade but col­our me skep­tical. I have been caught out before where a hard­ware upgrade was needed.

These stand­ards are also not going well. Recently we saw TRILL ini­ti­ated at the IETF which is dir­ectly com­pet­ing against the 802.1aq stand­ard which appears to have stalled. This may be a sign of major prob­lems with agree­ments on the standards.

Secondly, FCoE may not be a Cisco-​​only effort, but the line-​​up of Emulex, Intel, Qlogic, Vmware, EMC and Netapp is hardly an industry-​​wide adop­tion. What about IBM, HP, Sun, Dell? What about HDS? A new pro­tocol with new HBA cards is pre­cisely in the best interests of Emulex, Intel and Qlogic as they tag along on Cisco mar­ket­ing efforts. Its a free ride for them.

Remember that HP bought Left Hand net­works and DELL bought Equal Logic, for mult­i­bil­lions, to get iSCSI tech­no­logy into their port­fo­lio and their engineers.

FCoE requires spe­cial, new and expens­ive, hard­ware to do ANYTHING

FCoE is still Fibrechannel, there­fore you still need a FC con­trol­ler or switch to con­fig­ure all the usual FC func­tions like ISL, man­age­ment. For Cisco this means buy­ing a Cisco Nexus 5000, and pos­sibly Nexus 2000 as edge devices.

Compare this to iSCSI where you CAN use your exist­ing Ethernet infra­struc­ture and pro­gress­ively plan an upgrade to new hard­ware, pos­sibly around the Cisco Nexus 7000 as a core switch and the upgrade path is much more organic, and fit into exist­ing budget­ing and pro­ject processes.

FC has his­tor­ical soft­ware sup­port, but now iSCSI is equally important

Early VMware adop­ters will know that VMotion, VMDR and a raft other fea­tures were sup­por­ted on FC first because that is what their cus­tom­ers expec­ted. But more recently, as their cus­tomer base has expan­ded into the wider mar­ket­place and SMB, these fea­tures are also sup­por­ted on iSCSI, NFS etc. Take the time to research why many people are FC biased, and you will often find that its his­tor­ical. That is, “that’s how its always been done around here”. But that doesn’t always mean that’s the way it will be in future.

FCoE doesn’t scale

One of the most hid­den, least real­ised and unknown short­com­ings of FCoE is the lack of scale. Encapsulating FC in Ethernet is OK, but the require­ments for the Ethernet sig­nalling for CEE (or DCE) are not bet­ter or any improve­ment on the cur­rent stand­ards. That is, we can­not cre­ate data centres using Ethernet VLANs with more than fifty to a hun­dred switches before we lose sta­bil­ity.²

Does this really matter ?

Evidence: Cisco has intro­duced what they call Fabric Extenders (Nexus 2000 & 2100) which cre­ate “dis­trib­uted switches”. That is, the Nexus 5000 is the core and the Nexus ²⁰⁰⁰⁄₂₁₀₀ act as “blades” of the Nexus. There are two pos­sib­il­it­ies here, either its a low cost way of scal­ing the Nexus 5000 to meet cus­tomer require­ments or its an admis­sion that FCoE can­not scale and the need to keep L2 switch domains very small is high priority.

This might be less of a prob­lem if we can use very large switches, but build­ing lossless, low latency, Fibrechannel-​​capable is a really expens­ive pro­pos­i­tion. Therefore, just like Fibrechannel, Ethernet stor­age net­works will mostly con­sist of small switches for stor­age con­nec­tions only.

Evidence: 802.1aq/TRILL does not scale. Implementations have a limit of about 100 switches, which isn’t a lot for good sized data centre and cer­tainly not enough for Cloud Computing.

Conclusion

The path to FCoE is not cer­tain. Cisco has man­aged to force com­pet­itor Brocade into deliv­er­ing Storage over Ethernet (FCoE) and mak­ing massive mar­ket­ing push with its Data Centre 3.0 strategy but don’t be blinded by this osten­ta­tious dis­play of wealth. Be smart and make your own choices by con­sid­er­ing what will work in your Data Centre.

FCoE is by no means the default or only choice for stor­age net­work­ing, and I believe it will be optional for the vast major­ity of the com­puter rooms because of late stand­ards, high cost and sig­ni­fic­ant learn­ing curve. Look at iSCSI (use HBA’s for accel­er­a­tion) and NFS care­fully because of their lower cost.

This may mean that FCoE will only ever be a niche tech­no­logy for leg­acy FC plat­form integ­ra­tion and not as a stan­dalone high per­form­ance technology.

Footnotes

1. note that iSCSI does cre­ate a prob­lem with build­ing lossless, low latency IP net­works that have very high up times. This will require a change in data net­work­ing [back]
2. Note this is also true for Fibrechannel, how­ever, data centres with more than fifty FC switches in a single net­work is exceptional…and unlikely [back]

Please rate this post:

(2 votes, average: 6.00 out of 10)

 Loading ...