Notes on Cisco Catalyst 6500 architecture. (Or what does 720 in Supervisor 720 mean ?)

Switching Fabric

The Supervisor 720 contains a 720 gigabit per second crossbar switching fabric that providing for multiple conflict-free (or non blocking) paths between switching modules. At the back of the module you can actually see the chip that is the switching fabric:

c6500-architecture-3.jpg

Using the “supa klever” form of marketing math, a switching fabric is actually 360 GB/s but because it goes in at 360Gb/s and out at 360Gb/s and therefore you could count it twice. So that is seven hundred and twenty gigabits. That’s the number.

In fact, the switching fabric is eighteen inputs of 20Gb/s per input. Of these, the backplane in your Catalyst 6500 will decide how many are presented to your line cards. The C6503, C6506 and C6509 will present dual fabric connections while the C6513 is dependent on slot.

Per Slot bandwidth

Each slot in the backplane of the C6509-E chassis has two 20Gb/s backplane connections. There are nine slots. Nine slots at 40Gb/s is 360GB/s.

c6500-architecture-1.jpg

and the Switch Fabric connections look like this:

c6500-architecture-4.jpg

Catalyst 6513

Note that backplane slots on a 6513 do not support dual 20Gb/s on all slots. Because the switching fabric on the Supervisor has eighteen 20Gb/s inputs, you must layout the inputs differently.

c6500-architecture-2.jpg

Therefore the slots at the top of 6513 are only 20GB/s and not 40Gb/s. This is what makes the C6513 signficantly different from the C6509 chassis. The 6513 was a popular choice when low performance line cards were used, but today, high density gigabit ethernet is common and much increased bandwidth of servers and desktops †means the C6513 is not commonly used, especially for data centres.

The Switch Fabric connections look like this:


Caption Text.

Fabric Connections for C6515 chassis.(Click for a full size image)

Not All Modules are created equal

Consider the following modules that are commonly used today (2009):

? WS-X6724-SFP: a 24 port Gigabit Ethernet SFP based line card supporting a single 20 Gbps fabric channel to the Supervisor Engine 720 crossbar switch fabric. Also supports an optional Distributed Forwarding Card 3a (DFC3a, DFC3B or DFC3BXL)

? WS-X6748-SFP: a 48 port 1000Mb SFP based line card supporting 2 x 20-Gbps fabric channels to the Supervisor Engine 720 crossbar switch fabric. Also supports an optional Distributed Forwarding Card 3 (DFC3a, DFC3B or DFC3BXL)

? WS-X6704-10GE: a 4 port 10 Gigabit Ethernet Xenpak based line card supporting 2 x 20 Gbps fabric channels to the Supervisor Engine 720 crossbar switch fabric. Also supports an optional Distributed Forwarding Card 3a (DFC3a, DFC3B or DFC3BXL)

? WS-X6708-10GE: an 8 port 10 Gigabit Ethernet X2 optic based line card supporting 2 x 20 Gbps fabric channels to the Supervisor Engine 720 crossbar switch fabric with integrated Distributed Forwarding Card 3CXL.Not this card has 80 Gb/s of input (in one direction, 160 Gb/s in two directions) and could easily oversubscribe the backplane connection and switch fabric connection on those inputs.

An oversubscribed card means that packets may be delayed or dropped under heavy loads.

Cross Bar Connector on the module

You can easily tell if the blade has some form of connection to the Cross Bar fabric by looking at the connectors on the back of the cards. If the Cross Bar Connector is present, then the module has some form of the connection to the fabric.

c6500-architecture-6.jpg

Shared Bus Connector

There is also a Shared Bus on the C6500 that operates at 32Gb/s (yes, 16Gb/s in both directions).


Caption Text.

Shared Bus Conenctor.(Click for a full size image)

This is what was used in the early days of the chassis with the first generation C61xx and C63xx modules which could easily oversubscribe the backplane connection. Most new installation are completely fabric switched (excepting for IP Telephony installations who use the oversubscribed model and like the QoS features of the C6500 and the high power density of up to 8500W in a single chassis).

Not Comprehensive

This post is not a comprehensive look at the modules and supervisor architecture. This is covered a number of documents but check the White paper link just below this which has a lot more information and is a complete reference to the architecture of the C6500.

Observations and Comments

The reason for this article is to outline some of the issues that a Data Centre designer needs to understand. The architecture of the switch, and same logic applies to storage switches, can impact the design of the network. The decision on whether to purchase a WS-X6704-10GE or WS-X6708-10GE can be impacted by the predicted traffic load.

There are other factors that might affect a design, such as the Multicast performance or the QoS features, but this article points out some of the first factors for thinking about whether C6500 is the right switch. I hope you realise that knowing something about the guts of your switch, and how the insides look, is important to getting your network right.

Reference

1) Some images and concepts are drawn from the CISCO CATALYST 6500 SWITCH ARCHITECTURE SESSION RST-4501 from Cisco Networkers in 2004.

2) Cisco Catalyst 6500 Architecture White Paper https://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html

About Greg Ferro

Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualization. He has over 20 years in IT, in wide range of employers working as a freelance consultant including Finance, Service Providers and Online Companies. He is CCIE#6920 and has a few ideas about the world, but not enough to really count.

He is a host on the Packet Pushers Podcast, blogger at EtherealMind.com and on Twitter @etherealmind and Google Plus

You can contact Greg via the site contact page.

  • http://www.booches.nl RenÈ Jorissen

    Clear and to the point. Nice article!!

  • mrz

    Cisco was less forthcoming about this or at least on this level of detail when we built out our Phoenix location.

    Like you said in your observations, this does “outline some of the issues that a Data Centre designer needs to understand” and is what drove us over to Juniper for Phoenix.

  • http://blog.ioshints.info Ivan Pepelnjak

    Thank you! Very clear and illustrative ;) I’m assuming when you talk about 20 Gbps connections, they are really 10 Gbps (in+out)?

    • http://cisco.markom.info Marko Milivojevic

      Since they sell 4x10GE line card as not oversubscribed, I can only assume it’s 20+20.

  • http://aconaway.com Aaron

    Good stuff, Greg. I’ve never really dealt with 6513s that much, but, when I have seen them, they’ve always been populated below the Sup. This explains why. :)

    I always had a dislike for the stating of bandwidth as the sum of input and output. Yes, the port/interface/module/yourmama can handle that much traffic at once, but, in a lot of situations, I’m only interested in sending traffic from A to B, so I only care about one direction for the most part.

    Thanks for the info, Greg.

  • Robin

    Any chance you could do the same analysis on their 7600 chassis and it’s SIPs, SPAs and hot tubs ;)

  • Pingback: News – February 8, 2010 | xcke's blog

  • Luis

    Google “Cisco Catalyst 6500 Switch Architecture.ppt”

    These images are from a Power Point that Cisco distributes

  • Luis

    Sorry forgot to mention , you can also search 2005 networkers

  • 6513 – worst product possible

    every time i have had conversations with people about 6513s, I always tell them to avoid it like the plague. Of the dozens I have encountered it has always been to remove them in place of 6509s or 6506s so that any card could go anywhere. So many customers are sold on the density but don’t realize the limitations. Cisco shouldn’t even sell this atrocity, they have been talking about fixing it for years but with the nexus 7ks it will never happen. At least the 7010 and 7018 have the same capacity in every slot.

    • http://etherealmind.com Greg Ferro

      As I remember, The C6513 was a popular choice in the early days of IP Telephony (now Unified Communications) because you needed slots for DSP banks for Transcoding and Gateways function as well as a lot of WS-X6148 modules for high density floors. At that time, the 256Gbps Fabric was new and exciting and the WS-C65xx modules were very expensive. Since the WS-65xx could only use one backplane channel at 8Gbps, the C6513 backplane looked like a good idea at the time.

      That’s the way the marketing moves sometimes. If you give the customers what they ask for, you might regret it later.

  • http://www.opticfit.com/ Gigabit Converter

    It’s a great to know about how cisco catalyst works, but it would have been better if you have explained for what and why it is used.

  • Pingback: Catalyst 6500 Information | AlwaysTheNetwork

  • Justin

    You forgot to mention that 40Gbps per slot is the Half-Duplex speed. Its Full-Duplex speed is 80Gbps 40Gbps in + 40Gbps out of the slot. That means the sup720 is 360Gbps + 360Gbps or 720Gbps.

    Also the Sup2T supports 80Gbps per slot and supports dual channel on every slot in the 6513. That Half-Duplex on the slot again for a total of 160Gbps per slot or 2080Gbps where they get the 2T.

    Why the marketing folks list the speed of the slot in Half-Duplex is odd given their willingness to jump at any reason to publish a bigger number but I suspect changing their ways might result in some serious backlash as “its how they have done it for years”.

  • Pingback: Post-LISA Pre-Holiday Filler | Standalone Sysadmin

Subscribe For Weekly Updates by Email

Get a Weekly Summary of Latest Articles and Posts to your Email Inbox Every Sunday

Thanks for signing up. Look for the email from MailChimp & make sure you confirm your email address. You may need to check your spam or gmail settings to be sure of receiving the email.

Note: You can unsubscribe at any time using the link at the bottom of every email.