Cisco Nexus 5000 / 2000 Pricing Bundles and Fabric Extension Transceivers (FETs) vs 10GbaseSR SFPs.

Recently I noticed that Cisco is selling “Fabric Ethernet Transceivers” for the Nexus switch family. Some research shows that these are replacements for 10GBaseSX SFP modules. they come in bundles of product, cannot be purchased separately and I can’t find any detail on how FET works at a technical level. I’m guessing that it uses a proprietary PHY layer that strips out unused or pointless Ethernet requirements 1.

Rules for use

Cisco SFP Optic

Some sketchy details here say the following:

You can’t use FET’s for anything but connections to Cisco NX7K, NX5K, NX2K

  • Supported for fabric links only (Cisco Nexus 2000 Series to Cisco parent switch)
  • Cisco Fabric Extender Transceiver must be connected to another Cisco Fabric Extender Transceiver
  • Supported on Cisco Nexus 2200 platform uplinks
  • Supported on Cisco Nexus 5000 and Nexus 7000 Series Switch fabric links

You can find a description on the Cisco FET-10G. Here is the relevant bit.

The Cisco FET-10G Fabric Extender Transceiver support link lengths up to 100m on laser-optimized OM3 or OM4 multimode fiber. It is supported on fabric links only from a Nexus 2000 to a Cisco parent switch. __Note this product is not orderable individually.__

From the Nexus 2000 Hardware Guide

SFP+ Transceiver General Specifications
Transceiver Cable Type Connector Type Wavelength (nm) Core Size (microns) Modal Bandwidth (MHz-km) Maximum Cable Distance
N2200 FET MMF Dual LC/PC 850 50 500 82 feet (25 meters)
50 2000 328 feet (100 meters)
SFP-10G-SR MMF Dual LC/PC 850 62.5 160 85 feet (26 meters)
62.5 200 108 feet (33 meters)
50.0 400 216 feet(66 meters)
50.0 500 269 feet (82 meters)
50.0 2000 984 feet (984 meters)
SFP-10G-LR SMF Dual LC/PC 1310 G.652 fiber 6.2 miles (10 km)

As you can see, the FET doesn’t support the older fibre optic cabling types such as 62.5 micron fibre, or the low modal bandwidth cabling types. In effect, this means you need OM2 or OM3 type multimode fibre cabling and you might to upgrade your fibre to use them. In my view, it’s cheaper to install new cable than to buy the 10GBaseSR SFPs (more on the pricing comparison below).

Pricing – They are cheap

So, here is the good news. They are cheap. The bad news ? You can’t buy them as a standalone product.

How cheap ? Consider the following US List Price (for September 2011)

Cisco Nexus 5000/2000 Bundle Pricing
Part Number Description US$ List Price
N5548PM-6N2248TF Nexus 5548P/Expansion Module/6xN2248TP/48xFET $82,000.00
N5548PM-6N2248TP Nexus 5548P/Expansion Module/6xN2248TP $74,500.00
N5548PM-6N2248TR Nexus 5548P/Expansion Module/6xN2248TP/30xSFP-10G-SR $108,000.00
N5020P-N2248TP-BE Nexus 5020/6x2248TP/30xSR Bundle $105,000.00
N5010P-4N2248TF-B Nexus 5010P/4x2248TP/32xFET Bundle $53,000.00
N5020P-4N2232PF-B Nexus 5020P/4x2232PP/64xFET Bundle $85,500.00
N5020P-6N2248TF-B Nexus 5020P/6x2248TP/48xFET Bundle $79,500.00
N5K-C5548UP-FA Nexus 5548 UP Chassis, 32 10GbE Ports, 2 PS, 2 Fans $25,600.00

So if you were using standards-based 10GbaseSR with the ‘N5020P-N2248TP-BE Nexus 5020/6x2248TP/30xSR Bundle’ with 30 SFPs included, it would cost $108000. Compare that with the ‘N5548PM-6N2248TF Nexus 5548P/Expansion Module/6xN2248TP/48xFET’ which has 48 FETs and costs USD$82000. That’s a USD$26000 saving.

If you think having proprietary FETs is concern, then consider that you are getting 48 FETS, using as two lots of 24 for each side of the north/south connection ie. 24 FETs on the NX5K and 24 FETs for NX2K which means 4 FETs per NX2K. Thats the maximum number of FEX uplinks ready to go on day one.

Nexus 2000 FET Bundles

Lets say that you needs some more Nexus 2000 units with FET, there an answer for that.

Cisco Bundle Pricing
Part Number Description US$ List Price
N2K-C2248TF-1GE Nexus 2248TP with 8 FET $11,000.00
N2K-C2232PF-10GE Nexus 2232PP with 16 FET $14,000.00
N2K-C2224TF-1GE Nexus 2224TP with 4 FET $8,000.00

Compare that with the pricing for bare NX2Ks

Cisco Bundle Pricing
Part Number Description US$ List Price
N2K-C2224TP-1GE N2K GE, 2PS, 1 Fan Module, 24×100/1000-T+2x10GE (req SFP+) $7,000.00
N2K-C2248TP-1GE N2K GE, 2PS, 1 Fan Module, 48×100/1000-T+4x10GE (req SFP+) $9,000.00
N2K-C2232PP-10GE N2K 10GE, 2PS, 1 Fan Module, 32×1/10GE+8x10GE (req SFP+) $10,000.00
SFP-10G-SR= 10GBASE-SR SFP Module $1,495.00

Cheaper to buy new cables

By extension, the FETs cost about USD$250 each (at list price, apply your discount). All of a sudden, 10 gigabit Ethernet is actually possible instead of bloody expensive. With a USD$1000 difference _PER SFP, that’s USD$2000 per cable run just to use 10GBaseSR SFPs.

Lets say you get a 40% discount on your Cisco kit, that’s still USD$1200 for every cable run between switches to use 10GbaseSR SFP modules.


Power Consumption

Power : Approximately 1W per transceiver. Comparable or less that 10GbaseSR.

The EtherealMind View

I’m not comfortable with the lack of technical detail to tell or advise me what I’m not getting but if the transceivers are simply optimised to work with 50 micron fibre thus reducing the manufacturing cost of the launch laser then that concept works for me. Some feedback from Cisco would be appreciated.

The price of 10GbaseSR SFPs is hideously expensive, and for any good sized 10Gb network makes up large dollar value of the network kit and thus makes this a good option for consideration. The IEEE 802.3 committee does try too hard to be backwards compatible and this is probably an example of that. The cost of the those 10GBaseSR SFPs is way overpriced. I’ve proven that for Cisco products, it’s currently cheaper to replace your fibre and use the FETs than to buy 10GBaseSR SFPs. On this basis, it’s probably time for the IEEE to reconsider their ‘Microsoft method of support all twenty year old standards’ way of working and look to cut the backwards compatibility if it’s going to cost this much to implement.

On balance, I’m cautiously positive about FETs. 10Gb networking is important for the networking industry to move forward, and the current SFP pricing is holding up customer adoption. Cisco has big enough volumes to make custom hardware possible, and should act as a kick in the pants to IEEE to do the smart thing more often. When the price of recabling is cheaper than the SFPs, the backward compatibility should be discarded.

PS. You should not be using copper for 10GBaseT in your data centre. That’s just silly. Always use 10GbaseSR. Always. Copper has no future in your data centre.


Update 20111122: John points out in the comments that these are actually Ultra Short Range SFP+ optics. So I did some research and summarised it
What Are 10 Gigabit Ultra Short Reach (USR) Optics ?

  1. The Ethernet standard has built up a lot of cruft over the years from what I can understand. I’m assuming that some part of the interoperability requirements
  • DanielG

    What are your feelings on twinax for short runs?

    • Etherealmind

      Works for me – they can be even cheaper. Although, I’d choose the FETs so that all my cabling is the same everywhere and not have to worry about the different cabling types – especially in larger data centres.

      Plus, those bundles are 10-20% cheaper than single items. Makes that a good deal.

  • Ron Fuller – (CIsco)

    It is important to note that the Nexus 7000 also supports FETs for connectivity to the Fabric Extenders as well.  A great low-cost solution to expand the breadth of FEX connectivity with up to 32 FEX connectioned to a single Nexus 7000.

    • Etherealmind

      But you can’t buy a bundle of NX7Ks with FETs :)

      • Ron Fuller – (CIsco)

        Agreed, but you can buy N2Ks with FETs to connect to your N7K.  The bundles come with enough FETs for both the FEX and the parent device (N5K or N7K) so you’ll get enough FETs either way.

        Also good to note, FETs look exactly like any other SFP+ optic with the exception of the part number and a brown stripe on the latch.  Unless you are color blind, the colors can make it easier to identify in a jumble of SFP+s.  :)

      • Santino Rizzo

        There’s a bundle for the M132XP-12L line card that includes 8x 2248TPs and 32x FETs. 

  • Mike Kantowski

    Good stuff.  For N2K the FETs make perfect sense.  It can also be noted that some vendors don’t wildly mark up their SFPs.  My favorite vendor sells SR’s for about 250 each.

  • Kevin Breit

    Actually you can purchase them separate.  This wasn’t true before but the past few months I’ve seen them available spare.

  • Ryan Malayter

    “You should not be using copper for 10GBaseT in your data centre. That’s just silly. Always use 10GbaseSR. Always. Copper has no future in your data centre.”

    Excuse me? You complain about extortionate 10GBASE-SR transceiver costs, then say that is the only way to go? 10GBASE-T can save US$1000-2500 per link in transciever costs, and much more when you factor in the ability to use existing cabling plants. I understand fiber has a number of advantages in thickness and weight, but it’s simply not economical for in-rack/in-row connectivity. 

    Even twinax-copper with SFP+ connectors typically adds $150-500 per link versus 10GBASE-T, especially if you’re talking about the cost differentials in server NICs. When 10G-on-motherboard finally becomes popular, you can bet it will be 10GBASE-T.

    • Etherealmind

      Yes. The price for Cat-6A cable is very high when properly installed, test and certified in structured cabling system – much more expensive and prone to high BER compared to OM3 which can cause poor Ethernet performance in high performance networks.
      Twinax is only good for a maximum of 10M, but it’s complicated for large data centres to manage multiple cabling types. Copper is over, time to move on. 

      • Ryan Malayter

        Even here in Chicago, where the costs of union electrical work are famously high, the additional testing costs of Cat 6A – and even potentially replacement in less than 5 years –  will not approach $1000-2000 per link. Copper is cheap enough that extra runs can be done as part of install for standby purposes, eliminating the need for re-work down the road. Also, most structured cabling contractors will warrant their work and all the pieces for 10+ years.

        Doesn’t fiber have bend radius problems of its own, particularly near the transceiver?
        says cat 6A worst-case bend radius is 38 mm, which isn’t really a problem with any sensible cable management system.m In contrast, this link says that OM3 bend radius is 25-50mm depending on mechanical load.

        I agree it would be nice to have standardized fiber cabling throughout a DC, but “nice to haves” which increase costs by hundreds of thousands/millions in even a small DC just don’t fly. 
        The total system cost differential for fiber only gets worse as more long links are added. As DCs move from hierarchical oversubscribed networks to flatter Clos-style multipath networks, the number of inter-rack links increases dramatically.

  • Anonymous

    We are currently procuring a pair of 7ks and some 5ks with a smattering of 2ks as a part of a bigger facility upgrade. To connect the older Top of Rack switches to the 5ks we are going 3rd party for 10GBase-SR and half of the required LRs (to be on the safe side in case Cisco changes things to make 3rd party unusable).

    The cost savings are somewhere between incredible and insane – even with our hefty Cisco discount 3rd party weigh in at 1/5 to 1/10 of branded. For that kind of money I can afford a to simply discard a faulty module every now and then. If we’d bought branded optics throughout the SFPs alone would have accounted for 1/3 to 1/4 of the _total_ budget for this project.

    Put it this way – the handful of branded LR optics cost us as much as roughly 100 (yep, one hundred) 3rd party SR modules.

    Yes, the bundles provides affordable equipment, but Cisco just isn’t able to compete with the 3rd party vendors on optics  (for price anyways).

    (If you’re concerned about the reliability of 3rd party versus branded my anecdotal evidence is that in our installation with several hundred 3rd party modules and a handful of branded we have yet to experience failure or service degradation)

    Something to consider.

  • Colby

    Don’t forget about the new LRM stuff. It supports greater distances as well as SMF or MMF. We’re likely going to be using this exclusively for switch-switch links (where FETs cannot be used).

  • Bob

    We have found it too hard to plan on how long the cable runs are from our 7K Nexii to TOR 2248s or 2232s..  A good part of this is due to our 6+ month procurement cycle and others due bad (legacy) data center practices.  We just thew in the towel and went with SR SFPs. 

  • John G.

    They are actually USR optics, not SR.  This should explain the distance question you have.

  • Pingback: What are 10 Gigabit Ultra Short Reach (USR) Optics ? — My Etherealmind()