11th February 2012

You are here: Home / Blog / Why buy Dual Supervisors on Cisco Nexus 7000 ?

Why Buy Dual Supervisors on Cisco Nexus 7000 ?

9th July 2010 By 42 Comments

I was doing some research today and looking closely at Cisco Nexus 7000. Cisco strongly recommends purchasing two supervisors. Of course, I thought, two Sup’s means more sales and it’s just FUD to sell more than I really need. I already deploy two separate chassis, and they have full redundancy.

Turns out there is a good technical reason. The Nexus 7000 takes more fifteen minutes to boot, and often up to twenty minutes. Therefore a code upgrade means that your network is going to be down for at least thirty minutes. Having a second supervisor means that you can upgrade the backup Supervisor, and then failover to it within a few seconds. ( When I think about it, the same logic applies to Catalyst 6500 Supervisor engines. )

I’m not a fan of buying an second Supervisors (USD$25000 list price) just to be able to perform a code upgrade and Nexus 7K will require frequent updates of about twice a year for most people until features stabilise.

The EtherealMind View

Fifteen minutes to boot is too long. What sort of architecture does Cisco have in there that needs that much code loading and testing to achieve such bad performance ? Is this a symptom of bad engineering and design ? I sure don’t know, but being penalised by the very expensive purchase of redundant supervisor modules (for Cisco’s benefit) doesn’t seem like a good answer.

Bad Cisco. Bad.

This post is copyright of Thropos Ltd ©2008-2011 at Etherealmind.com - contact | email: greg.ferro@packetpushers.net - twitter: @etherealmind | All rights reserved
Filed Under: Blog, Cisco Tagged With: , ,
About Greg Ferro

Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualization. He has over 20 years in IT, in wide range of employers working as a freelance consultant including Finance, Service Providers and Online Companies. He is CCIE#6920 and has a few ideas about the world, but not enough to really count.

He is a host on the Packet Pushers Podcast, blogger at EtherealMind.com and on Twitter @etherealmind and Google Plus

  • http://rickmur.com Rick Mur CCIE2 #21946

    Have you ever booted a Juniper MX960, Juniper E-series, Cisco XR12k or Cisco CRS-1? All those boxes take at least 15 to 25 minutes to boot. Remember all the linecards run a small version of NX-OS so all linecards need to boot separately when your system is fired up. Same counts for IOS-XR and other hardware forwarding routers like Junipers.

    I still think a second supervisor wouldn’t be a large extra expense on top of an order worth of several times $100k when buying Nexus 7k’s. Of course it’s not necessary as you always have your second 7k ready to take over in a few seconds (if all protocols are timed/configured correctly), but the extra 25k per box shouldn’t be that much for being able to upgrade totally in-service and never having to reboot the box (sounds like carrier grade :-)

    • http://etherealmind.com Greg Ferro

      For core switches, I agree that dual supervisors are necessary. But for some use cases, it’s just not a requirement and being stitched for the extra cost is a rum deal.

      And if Juniper does the same thing, then that’s rubbish as well. I’m thinking that’s lazy engineering – as in “the customer won’t notice that so we won’t fix it”.

  • Craig Askings

    The Foundry MLX series is similar in that it has 2 supervisor cards and each line card has it’s own firmware/os that needs to be booted. That being said power on to interfaces up with basic ip was on the measure of 5 minutes. You had probably another 60-90 seconds after that for all your routing protocols and label switch paths to fully converge.

    15 minutes just seems bizarre. Maybe it’s all that FCoE stuff gumming up the works :P

    • http://etherealmind.com Greg Ferro

      The FCoE stuff isn’t released yet, so I guess it’s going to get worse not better.

      I think customers might be getting ripped off. Not many people would ask about this, and it is in Cisco’s interest to recommend a second module so why bother to fix it ?

      Nice to know that someone can make a system like this boot faster, leads me to think it really sounds like lazy design.

      • joe smith

        I’ve not used the Nexus series myself but is there some sort of warm upgrade option like on the ISR’s to speed up code upgrades?

  • Jason Myers

    It runs a more Unix like OS, and supports virtualization of the device into up to four contexts. I’ve worked with several Nexus 7Ks to this point, and yes the boot is long. I find the return for Medical grade and Campus networks is huge when you cost is a major factor and you want a seperate Core and Distribution architecture. In many cases there is not space or budget for four 6500s, and you can accomplish the same time with dual Nexus 7000s and their contexts. In additional to In Service Software Upgrade, the second supervisor provides redundancy in the most critical part of the datacenter or network.

    • Jason Myers

      accomplish the same thing not time

  • http://wikibon.org/blog Stu Miniman

    Face it Greg – next generation Ethernet environments are starting to look a lot like Fibre Channel SANs – highly available, online code loads, redundant components. Ethernet wins, but it doesn’t look like it has in the past…

    • http://etherealmind.com Greg Ferro

      Stu

      One day you will understand…. :)

      You describe the very heart of the problem with Fibrechannel. Overspecified, overperforming and overpriced. The enforced subscription to a model of overcharging and excess features means that it’s been highly profitable for vendors and mostly unnecessary.

      At least you have choice with Ethernet, you can choose a simple and cost-controlled path, or select the ultimate path. There are valid use cases for both.

      • John

        Greg,

        I don’t understand — you just finished complaining about a several-hundred-thousand-dollar Nexus 7000 price tag, and then turn around and give Stu a hard time about overpriced and such. Please don’t misunderstand, I’m rather new to this field and would appreciate a bit more context to understand better. Could you spare a moment to help me see the bigger picture between these two factions (FC / Ethernet)?

  • http://bradhedlund.com Brad Hedlund

    The second supervisor allows for a *hitless* code upgrade, not “within a few seconds” as you say here. The upgrade is *hitless* for all L2 and L3 protocols, something no other platform in the same class is capable of.

    Some network engineers like the fact that they can upgrade code without the headache of warning the Apps and Storage teams and dealing with all of that political drama.

    If that doesn’t matter much to you, just buy (1) supervisor and upgrade your 7000′s one box at a time. If you have designed your network properly all of your traffic should switch over to the remaining 7000 in one or two seconds, or sub-second if you’re using vPC.

    Cheers,
    Brad

    • http://etherealmind.com Greg Ferro

      I know that, but explain that to current change management hegemony. No, it’s a case of pay US$25000 to get the hardware that supports ISSU. That’s not so nice.

      • Kaj

        you’re doing it wrong if you’re thinking the sup costs 25k.

        • http://etherealmind.com Greg Ferro

          USD$25000 list price is what I actually said. And I looked it up when I wrote it.

          • Kaj

            a new purchasing department is needed if you’re regularly buying things at list. ;-)

    • http://blog.ioshints.info Ivan Pepelnjak

      Could we get back to the original question: why does it take 15 minutes to boot a switch?

      • PatG

        I thought the original question was “Why buy Dual Supervisors on Cisco Nexus 7000?” :) Hitless upgrades is the main answer and it’s not a marketing gimmick. It works beautifully, the only traffic that is dropped is your SSH session to the mgmt port on the supervisor. Do you not think this is important for a switch that will carry FCOE or iSCSI traffic? I know iSCSI sessions can survive some dropped packets but it’s better to drop none right?

        I’ve done a number of hitless upgrades on N7k and the last one I did I had so much confidence I upgraded the N7k at 2 different data centres at the same time, remotely, without worrying about whether it would break so badly that I couldn’t manage the device.

        Where you have redundant chassis’ in the same data centre there is not as much value in having hitless upgrades. However, the multi context capabilities allow smaller enterprises to consolidate into a single switch with Virtual Device Contexts giving you multiple logical switches (eg, DMZ/Internet switching/routing in one VDC and prod/servers in another). In this case dual sups are essential.

      • MikeInSeoul

        My guess is some of the time is spent waiting for timeouts on all the “Call Home” timers back to cisco.com.

        Well, that, and diagnostic checks on each and every port in the switch. Plus bootup times for my IDSM-2, FWSM, and NAM blades. But given the diagnostics, I would think an old 48 port switch blade would take longer than any of the service modules.

  • Roland

    Have anyone here really timed how much time it takes to boot? I have one in lab, first thing I’ll do on Monday ;-)

  • Check the Costs First

    The Sup lists for only 24k, cheap in comparison to everything else. Others have posted about juniper, and other cisco products taking a long time to boot.

    I truly think you are missing the point. With 6500s or even 4500s the dual sup thing was a total pain in the tush but with the Nexus7k or the MDS for that fact, the dual sups is a beautiful thing. Have you ever done an upgrade of the code on either of these devices? Completely hitless to the end user. That is the reasoning, not boot time. For storage, or a true DC today, zero downtime is the goal. A nexus 7k, ASR 1l/9k, Juniper EX8200/MX480/SRX5k yadda yadda should be able to have code upgrades and hardware failures without impacting anything. And they can, that is what you pay for.

    • PatG

      “Have you ever done an upgrade of the code on either of these devices? Completely hitless to the end user. That is the reasoning, not boot time. For storage, or a true DC today, zero downtime is the goal.”
      Indeed! These switches are designed to run constantly and never need a real reboot. Why cripple a DC switch by requiring downtime to do upgrades? If it’s hitless, who cares how long the sup takes to boot?

    • Kaj

      MX is sweet with ISSU. Works quite well cross versions, too. If everything you’re running is supported for NSR you’re golden. Also two fabrics are very handy, I’ve had one fabric fail and the other (on the other SCB1) and people didn’t notice with the exception of a bunch of NMS alarms. If MX is in PE role you should certainly be having both dual fabrics and REs. If you’re having it in a P role you can in theory skip all that if you’re on low budget. ;-)

  • mano7042

    if you are putting the N7K in a datacentre, no question you need a 2nd sup. that fact will be reinforced when we get multihop FCoE later in the year and start to move shared storage/SAN as a layer off the core, in a similar model to our Ethernet aggregation layer. 24K * 2 is a minor %age of the full DC cost, and ISSU rocks.
    FCoE was on the road map from day one with the N7K, that why Cisco are recommending. if you are deploying an N7K in enterprise or SP and only going to be using ethernet, and have followed the classic design (dual homed aggregation), then i guess you can get away with single sup, as the other core switch will take over forwarding

  • Pim

    Well what about 2 6500′s with VSS enabled,
    Would you need 2 SUP’s in every chassis?

  • skm

    Just stumbled upon this topic. I am still not clear about the ISSU upgrade. I understand the Supervisor image upgrade can be in service with dual supervisor configuration, but how do the line cards get upgraded. Do the line cards have a CPU that run the NX-OS. Are the L2/L3 protocols centralized on the supervisor CPU or distributed. Do the line card CPUs get rebooted when the code upgrade happens on the supervisors? Does the line card CPU upgrade cause the data plane of that line card to go down? This is not clear from any of the Nexus documentation.

    • Carlos

      Wish I knew, but the whole control plane decoupling tale goes along the line that the data plane can go on unimpeded while the con≠trol plane upgrades * and the world does not change * :)

      • Carlos

        BTW, anyone with real experience can comment on EPLD upgrades ? Those actually need the line card to go offline while upgrading, so kind of ruin the ISSU beauty for the clients of the upgrading line card ports. May be we need dual controller line cards ? :)

        • AnonNOCMAN

          EPLD upgrades require the linecard to be powered off and on.  So unless your servers are dual homed you’re out of luck, and if you’re a workstation you’re down.

          FYI EPLD upgrades take around 25 minutes per card.

  • robert r

    My core 6500s have been up close to 5 years, CatOS -> IOS upgrade time.
    I have 2 5020s that had been up 436 days. I rebooted them to get FET & 2248 support.

    I suspect my 7Ks will be the same.

    I could not care less if the boot time is 15 minutes or 15 hours if I never, ever have to do it again.

    Greg,
    I like your writing, I really do. But you are not looking at this from a end user customer viewpoint – they want less than zero down time (I am in healthcare with 6000 employees and 30+ sites across multiple states) and if a 15 minute boot time is required … I’ll plan for that … in 2015. Until then I don’t expect to need to power cycle my 7ks. These are not AGS3000s and I traded the 6 9s reliability and a totally ignorable amount of boot time for a pittance of extra cost – my extra sups cost less than the fiber runs connecting them to my core and to each other – thats a bargain.

    If I could put in a 3rd sup in my 7Ks – I’d do it in a hearbeat. I even want that 3rd power supply.
    It is not that zero down time saves me money; everybody says that – it is that not having zero down time costs my organization calculable money.

    I used to write forth code back in the Z80 days and while I am “curious” on what the sups and line cards are doing when they hook up … I stopped really caring years ago.

    • http://etherealmind.com Greg Ferro

      Well, I care. It’s not that every customer needs to have dual supervisors, there are circumstances when outage times that are short enough a perfectly acceptable and pouring money into a bucket for no good purpose is not an answer.

      In this case taking 15 min to boot, is probably equivalent to your desktop taking 10 min to start. What possible advantage does this bring what testing is going on what magic is happening on that board that take so long for it to get started. I think that the CPUs might be way underpowered and not able to start rapidly.

      And because you blindly accept such poor quality. Cisco does not feel the need to do anything about improving the boot time. Having dual supervisors to address Cisco’s own limitations is bad practice and you are paying the price. At least that’s the view that I take and I shouldn’t have to cover up for poor quality product and spend twice as much money.

      Now I accept that there are times when dual supervisors are needed for high availability or high uptime requirements but what about those customers who don’t need this ?

  • robert r

    You don’t manage 7Ks do you ?

    Skip dual sups and go with L2/L3 convergence. Maybe 7Ks are not the best solution for all. I can’t afford the VW diesel Phaeton that you can buy in the UK, sure would like an all wheel drive diesel supercar … a plug-in electric Nissan Leaf is more what I need at 1/3 the price.

    I see a business opportunity for someone to rent a secondary Sup to allow customers to add it to a 7K, do an ISSU then move the Sup to their 2nd 7K … I’d pay a grand to rent it for a few days if it meant ’0′ packet loss.

    If my desktop took 10 minutes to start but it only did it once every 5 years in prod – I would probably move on to something else that bothered my network.

    What customer do you know who is needing to cold boot their 7Ks often enough that a 15 minute boot time is a drag ? Even in a pre-prod test mode you don’t do this … a lot ….

    Have you watched a 7K boot via the console ? There are like 100 temperature sensors alone and I want every one of them to work – they HAVE alerted me to insufficient cooling in their previous locations.

    I do not blindly accept poor quality – nor do I think that this product is. I paid my own way to Networkers this past summer and in all of the data center sessions heard not a single person complain about their boot times.

    I think this is a non-issue to us who actually use this stuff.

    And for being overpriced – It was less costly to do 7Ks and 5Ks with twinax, FEX & FETs than it would have been to even buy the fiber gbics and 6704 or 6708s for 300+ servers in 2 data centers undergoing facelifts and I still would not have had mcec or vPC / vDC (we use “spare” vDCs to emulate prod network in test … saved buying core test gear ). Yes we could have gone to VSS to get mcec but there is currently no path towards OTV with current Sups and we own our own metro dark fiber strands so I want to not lock us out of that.

    Hard to write this with a straight face but Cisco saved us money on the Nexus line. Probably on the UCS chassis stuff as well.

    I do have a complaint on Nexus gear however – I do wish that FEXs could be removed from the rear. It would make upgrading or replacing FEXs easier in racks with minimal free cable length.
    The mounting hardware can be attached this way but the RJ45 front protrude too far forward in our racks.

  • Gordon

    For all of you Nexus 7K fans out there, this equipment is just now becoming available on the refub market. A lot of companies purchased these and their projects got delayed for so long that they’ve started re-selling them until they need them. We had 2 complete N7K bundles last week but sold them. You could pick up a cheap redundnat SUP that way. We do have a lot of factory sealed N7K-M148GT-11 (48 port 10/100/1000) blades if anybody needs any of them at a low price.

    • chris stand

      Gordon,

      how do we contact you ? ( if moderators don’t mind )

  • Gmarsh78

    There are some other things to consider here. First it doesn’t take 15 minutes to boot. It takes about the same amount of time a 6500 does. Now, the more line cards you have in it and how detailed the diagnostics you run during boot, will surely increase your boot time. Second, the dual supervisor is yes recommended for ISSU, but if you have to ask yourself how many supervisors are in an MDS switch? There are 2, so with the converged network strategy and design, you will see having a 7k with redundant supervisors will not only provide maximum up time for ethernet, but also FCoE environments. 

    • http://etherealmind.com Etherealmind

      Yes but my point is that you don’t HAVE to have to dual supervisors etc in low performance / moderately critical systems. You can have outages in a campus network for example on the weekends, for hours at a time.

  • Fan

    Or else you buy the Avaya VSP 9000, takes between 3 & 4 minutes to boot.

  • SoN][c

    I have a quick question about the linecards listed on Cisco’s website for the N7K.  Why do some of them list the maximum number of IPv4/6 routes, ACL’s, MAC Addresses, etc? Wouldn’t this be a limitation of the suprvisor as opposed to a line card? Do line cards have their own routing tables independent of other line cards? I’m just starting to research N7K (I’m a Cat 6500 guy) so I’m a little confused…

    • SoN][c
    • http://etherealmind.com Etherealmind

      The Nexus 7000 architecture is completely different to the C6500. In the C65K supervisor, the silicon switch is on the supervisor and some forwarding is done on the line card itself (that’s what the DFC does).

      On the NX7K, the silicon switching is done in the Fabric modules, and the forwarding is done on the line cards which each line having a copy of the CEF table. The Supervisor performs only control plane tasks such as software downloads, routing protocols such as OSPF and BGP, CEF table calculation and distribution, and system status.

      There are several papers on Cisco’s website that talk about the architecture. You should review them to understand the system in more detail. Or look at the Cisco Press book on the Nexus switches.

  • Justin Other

    How about having dual SUPs in just one of the chassis?  could you hot swap the one where needed during OS upgrades?

    • korman

      The single chassis with Dual SUPs offers 5 x 9 availability no?  Are 2 chassis recommended for port density or is it simply to protect ia chassis / backplane issue?  I would guess most business have similar risk in Blade chassis / and hardware such as San arrays with clustered controllers in 1 physical enclosure such as a vmax or  a Netapp clustered 3160?