Switch Fabrics: Fabric Arbitration and Buffers

In the article, “What Is the Definition of a Switch Fabric ?” on Switch Fabrics I looked at how a Crossbar switching fabric allow for concurrent circuit forwarding and how this is used to build a fabric.

In most cases, frames would be received and forwarded from an input to an output, as show in this diagram since the value of a crossbar fabric is that multiple simultaneous inputs are connected to multiple outputs.

What switch fabric pt2 1

But it’s inevitable that eventually, two frames will need to be forwarded to the same output either at the same time, or during the forwarding of a frame. In these examples of collisions, data is already being forward from input port 5:output port 2 and a new frame for input port 1:output port 2 is not possible.

What switch fabric pt2 2

In reality, this is a common event. For example, output port 2 might be a server port that has hundreds of clients sending data to it. Or an uplink to the network core from an aggregation switch with thousands of downstream desktops on access switches that connect. Or several other possible conditions in a distributed path network.

This collision can not be permitted in a fabric and must be prevented. The silicon circuits within the chip are simply not designed for it to keep them simple. The silicon fabric is controlled by a CPU-like controller. It controls a range of system functions for the chip such as the clocking, buffers and others.

We can represent this like so:
What switch fabric pt2 3

To control the input process to the chip, the current theory is that using buffers that hold a small numbers of frames will optimise the overall performance of the fabric. The buffer is an amount of memory that is synchronised and managed by the Fabric CPU which is known as the Fabric Arbitration.

What switch fabric pt2 4

The Fabric Arbitration is able to detect that a collision would occur, pause the frame in the buffer and wait for the output to clear.
What switch fabric pt2 5

The frame would then be forwarded once the Fabric Arbitration knows that the output is available.

What switch fabric pt2 6

Managing Fabric bandwidth

The importance of this management cannot be overstated. The performance of a Crossbar Switch is determined by it’s ability to forward as many frames, without collision, over the available circuits as possible. The problem with networks is that contention are inherently part of a hierarchical network design. Thus Fabric Arbritration is vitally important silicon design process to.

The path from the buffer to the Arbiter ASIC is separate from the Fabric I/O to ensure that the Arbiter ASIC is always able to control the Crossbar.

Buffer Management

The real challenge now becomes the buffer management process. After all, there is no contention within the fabric as frames do not enter unless the output is available.

Consider what happens if output buffer becomes full, or what happens when the input buffer overloads. What if the input buffer becomes full of frames for a single output ? It would drop frames even though other outputs are free and ready to receive to data. This is the topic of the next post.

Other Posts in A Series On The Same Topic

  1. ◎ What's Happening Inside an Ethernet Switch ? ( Or Network Switches for Virtualization People ) (11th January 2013)
  2. Tech Notes: Juniper QFabric - A Perspective on Scaling Up (14th February 2012)
  3. Switch Fabrics: Input and Output Queues and Buffers for a Switch Fabric (6th September 2011)
  4. Switch Fabrics: Fabric Arbitration and Buffers (22nd August 2011)
  5. What is an Ethernet Fabric ? (21st July 2011)
  6. What is the Definition of a Switch Fabric ? (30th June 2011)
  7. Juniper QFabric - My Speculations (1st June 2011)
  • Will

    Greg,

    Your switch fabric posts are superb.  Thanks so much for sharing this knowledge.  Would like to know if the same holds true within the switching within a router.

    • http://etherealmind.com Etherealmind

      the short answer is no. The long answer is It Depends ™. Lets avoid debates on switches being routers and assume that you mean low end Cisco IOS routers such as the 19xx/39xx. These devices don’t use use silicon forwarding, but are fully CPU/Memory based for software operation.

      IOS routers use different techniques such as distributed forwarding, CEF and other software tricks to try and drive better performance at certain price point.

      There are a number of presentations from Cisco that talk about these techniques, you should try to get hold of those.

      greg

  • Jon Langemak

    Hi Greg,
     
    I’ve been trying to learn more about the inner workings of the 6500 series switch and this was the only post I could find that did any justice whatsoever to the ‘crossbar’ fabric concept of switching.  However, Im still not clear on a couple of items.  
     
    The ‘fabric’ physically exists in the SUP of any modern SUP720 driven 6500.  That being said, all of the other modules (let’s talk about the 6509) that are fabric attached need to connect into this fabric.  Let’s also assume that they are CEF720 cards that have dual 20Gbps connections into the fabric.  This is where I start to get a little confused.  Most cross bar diagrams I have seen show the cross bar as an ‘input/output’ diagram.  While I realize this is likely just to make a point, that’s not really how the dual fabric connected line cards work is it?  Im assuming not since some line cards (and SUPs) only have a single connection to the fabric.  That makes me wonder how those single fabric attached cards fit into a cross bar fabric.  Would that imply that dual attached cards could really only connect to the single attached card from one of their fabric connections?
     
    The bit about the fabric queuing and fabric arbitration helped clear that piece up for me.  I assumed that only one fabric connection could talk to one fabric connection at a time but wasn’t ever 100% sure.
     
    Rereading this I feel like Im missing a crucial part of my question but I can’t seem to clarify it any further.  I guess Im just curious about how the actual fabric connections on a chassis switch (like the 6500) really connect and work.
     
    Thanks for the great article!
     
    -Jon