What is the Definition of a Switch Fabric ?

The marketing people in IT tend to be overwhelmed by complexity and deep technology. For many liberal arts graduates, they take the drowning option and latch onto certain terms and then grossly abuse it. The most egregious abuse today is “cloud” but “fabric” comes a close second. In this series of posts I want to look at what is a FABRIC and provide a canonical look at what it does and how it works for us.

Some History

While I sometimes indulge in nerd history for fun, we need to look carefully at the way Ethernet developed before moving into more complex purposes. The first versions of Ethernet used a shared bus technology

What switch fabric 1

where all connected systems would simply transmit onto the wire:

What switch fabric 2

Of course, if two systems attempted to transmit at the same time, then frame loss occurred

What switch fabric 3

This was solved by defining CSMA/CD (Collision Sense Multiple Access / Carrier Detect) and the Ethernet card would check that the bus was clear before transmission and transmit if there was no carrier. In the event that another system transmitted while the Ethernet frame was spooling onto the wire, the collision sense would detect this and it would back off for a pseudo-random interval and retry. In practice, for low levels of utilisation (less than 60%) this worked OK, but as peak utilisation increases this solution becomes a deadlock and few frames are successful.

Enter Switching

In order to reduce the size of collision domains and thus reduce the number of collisions, Ethernet bridges reduced the number of systems per segment. This later developed into switches ( also more correctly known as multi-port bridges ) and each connection is a single “bus”, like so:

What switch fabric 4

This lead to the development of full-duplex Ethernet – because only a host and switch are connected, there is no change of sharing the “bus”. So full-duplex Ethernet meant that the switch becomes an input and output device.

What switch fabric 5

Any to Any Switching

The challenge now becomes that any input needs to forward frames to any output and there is no collision detection. This “any to any” function is a complex function to achieve:

What switch fabric 6

For example, if port 1 and 4 attempted to transmit to Port 2 at the same time ( because a server is connected there) a collision would occur.

What switch fabric 7

If you realise that this looks like Time Division Multiplex circuit switching – you would be correct. THe crossbar theory is derived directly from telephony, and reflects the nature of the flow. That is, a frame has a source and a destination

Enter the CrossBar Switch

The Networking industry then looked to telephone exchanges of the era which had exactly the same challenge of switching TDM circuits and copied the Crossbar switch theory but developed it in silicon form. The basic purpose is to switch any input to any output.

What switch fabric 8

Lets consider Input 1 sending a frame to Output 3:

What switch fabric 9

Lets consider multiple inputs forwarding to multiple outputs:

What switch fabric 10

It’s worth remembering that these forwarding events occur at the same time. At a given time interval, this diagram shows that three frames are concurrently being forwarded in the same time slice.

Where does the term Fabric come from ?

The diagram for a crossbar switch is correctly shown as this: What switch fabric 8

But most commonly, the marketing people find the lack of symmetry unpleasing and aesthetically insensitive. As a result most PowerPoint waterboarding of the 1999/2000 era showed the crossbar as this:

What switch fabric 11

And this looks like a woven piece of fabric.

What switch fabric 12

Hence the “crossbar silicon switch chip” became known as the “silicon fabric” and later was further shortened to just “fabric” out of laziness and incomprehension.

The EtherealMind View

Now that we have established what a Silicon Fabric is, I’ll extend into a series of posts that look into some of the more advanced features of switch fabrics, and other more technical aspects that affect your day to day working.

Other Posts in A Series On The Same Topic

  1. ◎ What's Happening Inside an Ethernet Switch ? ( Or Network Switches for Virtualization People ) (11th January 2013)
  2. Tech Notes: Juniper QFabric - A Perspective on Scaling Up (14th February 2012)
  3. Switch Fabrics: Input and Output Queues and Buffers for a Switch Fabric (6th September 2011)
  4. Switch Fabrics: Fabric Arbitration and Buffers (22nd August 2011)
  5. What is an Ethernet Fabric ? (21st July 2011)
  6. What is the Definition of a Switch Fabric ? (30th June 2011)
  7. Juniper QFabric - My Speculations (1st June 2011)
  • http://twitter.com/sfouant Stefan Fouant

    Excellent article. It’s amazing how few people understand the origins of
    switching fabrics in this day and age.  Definitely looking forward to the rest of the posts in this series.

  • Juan Lage

    Good post Greg. History always gets forgotten … :)

  • Amar

    Good one Greg.

    Also what happens when in cross bar switch when ( for ex) 1 and 3  simultaneously send data to 4 which is a server?

    Regards

    • http://etherealmind.com Etherealmind

      I write some more posts in the next few weeks so follow on RSS or check back.

    • http://www.myteneo.net Aaron Paxson

      I would imagine that is where your queuing and buffers come in.

  • Rob

    Nice article Greg, and a very useful one for newbies to get up to speed quickly on how we’ve got to where we are today in the layer 2 world.  Just a small typo: in your expansion of CSMA/CD, you need to swap the words ‘Collision’ and ‘Carrier’ around (Carrier Sense Multiple Access with Collision Detection).

  • http://blogstu.wordpress.com stu

    Excellent background. While “cloud” is on the to-do list for lots of C-levels (even if it’s due to management by magazine or what they heard at a conference), I haven’t seen “fabric” move beyond the vendor dialogue and those of us that cover the topic. The top line message for cloud is to change the way that IT is managed through pooling, orchestration and trying to shift the $ of IT from upkeep to “innovation”. Fabric on the other hand is mostly about reactions to virtualization, cloud and related environments that need massive scalability and simpler management; while there are real problems to solve, not sure that fabric terminology is resonating with those that buy…

    • Anonymous

      “Management by Magazine”  Haha I like it.

    • http://twitter.com/gcrossEXTR gcrossEXTR

      Stu: I think fabric is really about switching and what a handful of vendors now promotes to scale 10/40/100GbE. 

  • http://twitter.com/BenoitGONCALVES Benoit GONCALVES

    Nice article Greg ! Thanks !

  • Rob Horrigan

    Cool…looking forward

  • http://twitter.com/hquackenboss Harry Quackenboss

    Greg, Good article.  You didn’t mention rings, which, although not allowed in traditional Ethernet, have been used to implement switch fabrics inside Ethernet switch boxes, and of course were relevent to Token Ring and FDDI, not to mention in telecom networks, both TDM circuit and packet-based. From one perspecive a ring is simpley a bus where the ends of the bus are connected together and has the advantage of reducing the number of hops.

    • http://etherealmind.com Etherealmind

      Yeah, that more of a university level topic and I was trying to skip it here.

  • http://twitter.com/silverbenz Ben Johnson

    Thanks for this, Greg.  Fabric does seem to follow closely on the heels of ‘cloud’ as the buzzword-du-jour.  I’m fairly certain I heard it used in the context of Layer 3 in a recent Packet Pushers episode as well.  Extrapolating what you’ve said above, this would be something like imagining each IP address as if it were a switch port (extending the crossbar analogy, essentially a point of ingress or egress) for packets, right?  Still getting my head around network esoterica.

    • http://etherealmind.com Etherealmind

      Umm, I dont think so. Fabric should be used when referring to L2 switching – but because frames contain packets it’s easy to ‘brain fart’ and use the wrong term. Today, IP is nearly always encapsulated in Ethernet at the link layer and thats how it gets switched – the fabric doesn’t care about IP addresses only MAC addresses.

  • Paul Leroux

    Hi Greg.  @paulaleroux here.
     
    It is often said that one of the most difficult things in the world to accomplish is to take something extremely complex and explain it in a simple and concise manner.  well done on this write up.

    As for Switch Fabrics. One of the marketing terms I have difficulties explaining or understanding myself is when companies like Cisco say that the Nexus7000 has a 1.2 Tb/s system bandwidth. or when Avaya says that the ERS8600 has a 1.44tb/s back-plane. etc etc. 

    Backplane, Switch Fabric, Switching Speed. ….. I can only assume the terms are interchangeable and that these companies are making claims that their devices can “push&pull” that much data any given second?  would that be fair?

    • http://etherealmind.com Etherealmind

      It’s a complicated topic – but vendors use the full duplex bandwidth when giving out numbers wch is misleading. And the speed of each component can vary according to it’s capabilities. Thus the backplane, switch fabric, line cards can all have different speeds – and as the platform matures and new hardware ships it will get closer to it’s maximum performance.

  • http://www.myteneo.net Aaron Paxson

    Nice and clean images.  Concise and to the point.  Thanks for sharing!  Looking forward to your future posts on the subject!

  • http://twitter.com/tompetrocelli Tom Petrocelli

    Not all marketing people are technical morons or were liberal arts graduates. Quite a lot of technical marketing people are engineers who designed everything from embedded systems to distributed systems to large scale enterprise systems. Not that that matters. Terms like “cloud” and “fabric” get overused or misused because it grabs the attention of managers with budgets not because of simple ignorance. I’ve seen IT and engineers people get both of these concepts wrong.

  • http://www.str82heaven.de LeoN

    Thanks Greg. Until your post I had no clue what this fabric hype is about

  • Pingback: What is an Ethernet Fabric ? — My Etherealmind()

  • http://andrewspiers.pip.verisignlabs.com/ Andrew

    I appreciate everything in your article from the second paragraph onward, but disagree with your first paragraph. As much as you accuse marketing people of mischaracterizing data fabrics, I think you unfairly impugn liberal arts graduates. Arts and law graduates tend to be very careful about how they use language.

  • Steve

    Hello,
    I’m a long time podcast listener. Thanks for doing this. I do have one question…

    How does the Crossbar solve collisions? i.e. If input one and input four wanted to send to output 3 inthe same time slice. How would the crossbar deal with that?

    Thanks again!

    • Steve

      Apologies.
      I should have read all the comments.

  • Pingback: Switch Fabrics: Farbic Arbitration and Buffers — My Etherealmind()

  • Pingback: A Look at Cisco PONG()

  • Pingback: Juniper QFabric - — My EtherealMind()

  • Joe Smith

    Hello. Interesting article.

    I want to reiterate Steve’s question. How does a Crossbar fabric prevent collisions? It doesnt in and of itself, correct? Fabric arbitration, buffers and other methods to prevent HOL are really what prevent the collisions. No?

    Thanks

  • Pingback: Qfabirc and its role in the enterprise « CCIE 31104, what's next?()

  • Foozlesprite

    I’m a CCNA student who has taken CCNA 1 and 2 (Intro and Routing) but Switching is next semester. I came across the term ‘switch fabric’ while browsing the Internet and needed an explanation; your article and diagrams were clear and comprehensible even to me. Definitely going to follow your site as I learn more about networking. Thanks for being awesome!

  • Pingback: The Fabric ERA « Routing-Bits()

  • http://twitter.com/miguelinnit Miguel

    Sticks out like a sore thumb, the typo in your article:
    Carrier Sense Multiple Access/Collision Detection (CSMA/CD)
    vs. CSMA/CA (Carrier Sense Multiple Access/Collision Avoidance)

    You swapped Carrier and Collision in your post.

    Good article none-the-less.

  • Pingback: Ethernet Fabrics Cards | Network Static()

  • Pingback: Network Static()

  • Pingback: What’s Happening Inside an Ethernet Switch ? ( Or Network Switches for Virtualization People ) — EtherealMind()

  • Ikem Krueger

    > _The_ crossbar theory