Nexus 5500 Packet Forwarding – Cut Through or Store & Forward

Was reading through some documentation as noticed that the Nexus 5500 series has some unusual behaviours for Store and Forward. I made myself notes about the functional modes.

Cut-Through vs Store and Forward

In cut-through mode of operation a switch will start transmitting a frame before the frame has been completely received and this is the low latency mode of operation. Store-and-forward switches, on the other hand, cannot achieve low latency since the packet must be completely stored in switch memory or ASIC before being propagated out the port.

Store and Forward was popular in late 1990’s because it was common for Ethernet frames to be corrupted on the wire before reaching the switch due to collisions and broadcasts which would cause runts. Cable quality would also cause bit damage in frame payloads (this is quite rare today). Therefore, only forwarding valid frames was a real advance. Today, it mostly adds latency for little gain.

But not all cut-through switches are created the same when it comes to latency as the underlying switch architecture makes a big difference. For example, in the Nexus 5500 switches it will automatically use the following modes for frame handling

Forwarding Mode Behavior (Cut-Through or Store and Forward)
Source Interface Destination Interface Switching Mode
10 GigabitEthernet 10 GigabitEthernet Cut-Through
10 GigabitEthernet 1 GigabitEthernet Cut-Through
1 GigabitEthernet 1 GigabitEthernet Store-and-Forward
1 GigabitEthernet 10 GigabitEthernet Store-and-Forward
FCoE Fibre Channel Cut-Through
Fibre Channel FCoE Store-and-Forward
Fibre Channel Fibre Channel Store-and-Forward
FCoE FCoE Cut-Through

I’ve highlighted the one line which is surprising. Store and Forward is almost mandatory for FibreChannel standards compliance to maintain low error rates. Equally, you can’t transmit a frame from a low speed interface to a high speed interface until you have received the entire frame.

The reverse is not true however since Cut Through for 10G to 1G is practical as the frame would be received on 10G port before transmission can start on the 1G.

Rules of Thumb.

Faster to Slower is Cut Through – No buffering needed to transmit the frame.

Slower to Faster is Store and Forward because the frame must transmit in a single elements, you can’t start transmitting the frame at 10Gbps and then pause while you wait for the rest to be received at 1Gbps.

FibreChannel to anything is Store And Forward. probably for error checking as part of ANSI requirements[8]

The surprising feature here is the use of store and forward for 1 gigabit connections. I’m not sure why this is true.

References

Cisco Nexus 5548P Switch Q&A

Cisco Nexus 5548P Switch Architecture

  • Krunal

    Each port has 10 Gbps UPC so there is a speed mismatch. 1 Gbps port to 10GbpS UPC to 10GbpS UPC to 1 Gbps port. So 1Gbps to 1Gbps also has to be store and forward.

  • jpresnel

    According to the Live 2012 deck BRKARC-3452 page 35:

    “The X-bar fabric is designed to forward 10G packets in cut-through which requires that 1G to 1G switching also be performed in store and forward mode”

    So, it isn’t the UCP (ASIC), it’s the UCF (Unified Crossbar Fabrix aka XBar) that forces the restriction.

  • http://twitter.com/dcswitchanalyst DC switch analyst

    This gets more complicated with Broadcom’s Trident ASICs. The first version ran in store-and-forward mode up to ~768 bytes and would change to cut-through mode. The newer Trident+ shifts that mode change earlier at 512 bytes. Does that mean you should benchmark in FIFO or LIFO or split benchmarks at the threshold to measure accurately? ;^)

    And don’t forget 10GbE to 40GbE speed mismatch and vice versa. But Nexus 5500 does not have 40Gb yet so that issue is only on Nexus 3000.