The Myths and Magic of Fibrechannel over Token Ring

It’s not a secret the that FibreChannel over Ethernet has caused a lot of change in the Storage industry. In short, the tedious world of slow moving and unexciting storage technologies has long been in need of change. The Cisco-led development of FibreChannel over Ethernet challenges a number of fundamental, and indeed religious tenets of FibreChannel that no loss in the network layer can ever be permitted.

The Challenge of Lossless Networking

Ordinary people have failed to comprehend the magnitude of money and technology wasted to deliver this promise. In the real world, there is no way to build a lossless network, the physical and logical criteria makes this impossible.

Of course, you can OVERENGINEER a network so that it looks lossless. And that’s what the Storage Industry did. In spades.

The elements of Loss

In a storage network loss can occur, in brief, at the following points:

  • the software driver
  • the network adapter
  • the ingress buffer on the network switch port
  • the egress buffer on the network switch port
  • the module or silicon handling the switch port
  • the backplane or module interconnect of the fibrechannel switch
  • the module or silicon handling the switch port
  • the egress buffer on the network switch port
  • the ingress buffer on the network switch port
  • the available bandwidth of the interconnect between switch
  • and so until it arrives at the storage array
  • the egress buffer on the network switch port
  • the ingress buffer on the storage switch

and so on. At each of these points congestion can occur, for microseconds or, in the worst case, indefinitely. Addressing challenges such as Head of Line blocking, Ingress port drops for microburst performance are, at best, a work around.


The FibreChannel industry managed to convince customers that lossless networking was a good idea. In the same way that the American Automotive Industry hoodwinked their customers into believing that American Cars were the best in the world. Which they weren’t and no one outside of America was buying them (which, naturally, those idiot customers continued to buy and are now congratulating themselves for their discovery on non-American cars. Golly, aren’t they clever). The same has been happening in the Ethernet Industry which has looked and laughed at the Storage industry. It’s so easy to declare that the network shall be lossless and hide the dirty little secret of making that happen.

The answer to the over-load problems is to build so much capacity, so much performance, so much buffers and spend so much money that the loss is highly unlikely to occur.

Thus the FibreChannel CNA was born with onboard CPUs and massive buffer memory. The SAN switching fabric that achieves an average of 1% utilisation in real life. Line cards with 48 ports and 48 silicon chips to cope with the rare instance that data actually occurs.

As SAN switches went from 12 ports to 100 ports the scaling became an enormous headache. And required even more massive overengineering, which was promptly dubbed “Director Class” (a classic case of lipstick on a pig). The “Director Class” FC switching had so much bandwidth, buffers and acceleration tricks that it was effectively impossible (except for the very largest of SAN networks) to use even a fraction of its capacity.

Ten Years Later, It’s not really LOSSLESS

Ten years later ad thousands of man-years of self interested brainwashing on the importance of lossless networking, lets make the statement. FC can withstand loss provided that it’s less than one packet in 100 million. That’s right, the FC standard states that loss IS ACCEPTABLE.

Oh dear. Oh FECKING dear.

Anyway, the vast bulk of the Storage Industry has no idea that the lossless networking is a sham to overcharge for overspecified equipment. They don’t appear to think about these things.

Enter: Lossless Ethernet

Of course, Ethernet is not lossless. But we can over-engineer that to buggery, and it’s real easy. You see, the FibreChannel PHY layer is a direct copy of the Gigabit Ethernet physical transport. The FC people couldn’t be bothered making their own so they nicked the IEEE proposals. So, the Ethernet folks lets take the over-engineered FC switch and make forward ethernet frames. Too fecking easy. Cisco knocked it up in a year or so.

But Ethernet isn’t lossless, I hear you cry. And cry you might, because Ethernet protocols are TOLERANT to loss. In fact, they are inherently better designed to accept loss and cope with it. That’s why mainframe used Ethernet for communications back in the day. But I digress.

The problem isn’t that Ethernet is lossless, it’s that it is shared with other data.

The importance of sharing

So now that we have our overpriced, overspecified uber-fancy Ethernet network that can transport FC, it’s the realisation that the networking people want to use that network for everything. TCP/IP for data, and Internet access, and desktop access.

May Crom defend us. HERESY.

We all know that Storage networks have special magical powers that make them different from other networks. I couldn’t possibly share my storage network with “common network traffic”. Well, guess what dipstick. You are screwed. The secret is out about your overpriced, overperforming and underused Storage network kit, and you lost the war.

Deal with it.

The new standards for Data Centre Bridging provide for lossless ethernet that exist in the real world.

Whither Fibre Channel over Token Ring

So for all those Storage Nut Jobs who can’t imagine their precious FibreChannel frames crossing an ethernet network, we are proposing the development of FibreChannel over Token Ring. That’s right, the second best networking protocol ever invented (after FDDI), offers everything you sad, attention deficit ridden, storage losers ever wanted in shared network. Deterministic delivery, over engineered cabling, layer 2 troubleshooting. We can even improve the FC protocol by isochronous transmission for serial clocking performance and guaranteed delivery.

The networking industry obviously wasn’t able to recognise the true value of this amazing network technology and now, it’s time has truly come. It’s for the ANSI standards organisation (yes, the same lunatics who lord it over the FibreChannel standard) to start discussion with IBM and demand the patents for Token Ring and get to work.

Last known Token Ring standards were developed to Gigabit performance, and it shouldn’t be too hard to dust them off and ramp them to 10Gigabit and more. The Madge technology exists even today Token Ring networks are still used (Crom knows I have them where I’m working). Surely, a vital testimony to this persistent and most amazing technology.

The EtherealMind View

It’s time for the Storage Industry people to open their minds, reject their fears of Ethernet networking and demand Token Ring networks. This will guarantee their miserly little jobs and position in the data centre. Instead of spending their days slotting more spinning rust into their oversized array, they can have a new technology to play with that will be all their own.

Because I don’t want you on MY network, I urge you to join me at the FibreChannel over Token Ring blog, and add your voice to chorus of storage people who want to defend their petty little existence in their ivory towers. Join in and the pretend that the world is not changing around you. Put down your backups, stop fiddling with your LUNs, and join with the movement.

FibreChannel over Token Ring. You know you want it.

  • Micah Smith

    LOL, now that’s priceless

  • Stephen Foskett

    It’s time we storage people start seeing just how little lipstick there is left on that notorious pig, Ethernet. We should abandon our efforts to patch duct tape with cellotape and switch to a real reliable data link. That’s why I’m 100% behind the FCoTR effort.

  • Bob Plankers

    The Fibre Channel over Token Ring Alliance ( thanks you for writing so coherently on this topic and showing your support. MAUs forever!

  • PatG

    ROFL. I can’t wait to troubleshoot beacons again!

  • Pingback: The FCoTR Phenomenon Exposes the Weaknesses in Ethernet – Stephen Foskett, Pack Rat()

  • http://n/a Mark

    … and I thought there was a trend to converged networks, silly me! :().


    • Greg Ferro

      That would be a marketing trend, not so much a physical actually-exists-today sort of trend. But they want to think it is there.

  • Ethan Banks

    I read a lot of tech articles. Sometimes I write them. Mostly I stay away from things that smack of politics or opinion. But sometimes I read an article that is so RIGHT, so ON, that I just nod my head in vicious agreement the whole time. Thus was the last x minutes it took me to (finally) read this.

    The FCoTR joke isn’t funny just because token ring has been irrelevant for over a decade. Ahem. Is everyone in the room paying attention? You need to read this article AGAIN if you think the joke stops and ends with the words “token ring”.

    I think the real revolution in convergence arrives when storage, networking, and virtualization skills start converging in “people”. It’s about time for the ivory silos to come down.

  • Matt Simmons

    So much promise…. Once again, “the token fell out of the ring. I’ve rung the vendor, but they’re back ordered” is a valid excuse. I can’t wait!

  • Barrett Lyon

    Woah… using a dead technology to transport a dead technology. Come on, seriously?! Fiber Channel is worthless in a world of cheap servers and distributed file systems. There’s no reason anyone needs to use a SAN or attached storage when all you need to do is let a software layer do all the work.

    Oh well, we had a good laugh in the office about this

    • Greg Ferro

      But it almost sounds like something that Storage people would believe. The Storage industry was stupid enough to buy FibreChannel and invest in it, and they want to believe that their frames and packets are somehow different to data.

      It doesn’t take much to make a joke out of that.

      Glad you had a laugh and make sure you check the for more information on upcoming product information.

  • Pingback: PFC/ETS and storage traffic: the real story – Gestalt IT()

  • Arms

    First, when i checket the public documents over at grober.IEEE.Org on 802.3z, Gigabit Ethernet made use of the Airways developed PHYs of FC. Only upgraded from 1.0625 gbaud to 1.250 gbaud.

    FC only used the ETH PHY for 10gigabit in a similar copy/Paste. (but Most Interfaces Run in the 1/2/4/8[/16] 8b10b stepping, very few make use of 10g FC.

    Second, you are right, overengineering the Network was the answer – and still is.

    The One elephant in the Room you seem to ignore is the congestion Control which is intrinsically different between all These protocols.

    Prior to dce with 802.1au (qcn) Ethernet has None (that actually Works. Spare me 802.3x). dito with TR.

    And tcp always Takes loss As Indication of congestion, trottling back the Sending rate. And tcp loss Recovery, White being very robust, can Break down for really real-Time delivery.

    But you may want to find out what was necessary to make ethernet reliable and good enough to Be installed in the A380 for secondary Systems. Only then you may want to continue your Rant.

    But don’t get me wrong, i’m a Big Fan of ethernet. But your simplistic View presented here is below you usual professionalsm. I will assume that you – As a Primars Networks guy, and Not a Transport Guy – haven’t Set researched enough into the perils of congestion Control.

    Excuse the typos and Mixed Case, Typing this on a over-nursing Apple device.


  • Pingback: vSphere 5 To Include vStorage API for Token Ring Integration – Stephen Foskett, Pack Rat()

  • KLC

    1997 I help migrate a call center off token ring.  its dead leave it buried, will not help the storage guys a bit,