I been doing some reading on the 40 Gigabit and 100 Gigabit ethernet standards. There are number of interesting facts that are going to change the way we work in our data centres. Most interesting is that I think that our fibre cabling plant will change dramatically from current practices.

Introduction

The currently approved standard for 40GbE and 100GbE are actually using multiplexing 10GbE channels to make a single “physical” connection over multimode fibre. I believe that this is a problem with the current silicon and lasers that perform the SERDES functions. The current technology is not able to signal at higher clock rates and thus 40GbE & 100GbE are actually “channel bonded” solutions. Since the bonding is performed at the physical layer, this is invisible to the user.

What is SERDES ?

SERDES refers to the Serialiser / Deserialiser chips that perform the physical signal generation. From Wikipedia

A Serializer/Deserializer (SerDes pronounced sir-dees) is a pair of functional blocks commonly used in high speed communications to compensate for limited input/output. These blocks convert data between serial data and parallel interfaces in each direction.

The basic SerDes function is made up of two functional blocks: the Parallel In Serial Out (PISO) block (aka Parallel-to-Serial converter) and the Serial In Parallel Out (SIPO) block (aka Serial-to-Parallel converter). There are 4 different SerDes architectures: (1) Parallel clock SerDes, (2) Embedded clock SerDes, (3) 8b/10b SerDes, (4) Bit interleaved SerDes.

There are future versions being developed that will use 25Gbps channels to reduce the cabling count and improve die sizing.

How many cables ?

• One interesting outcome is that 40GbE on Multimode fibre will need eight fibre cores. For 100GbE on Multimode you will need twenty fibre cores.
• The connector that has been standardised is the QSFP for 40GbE
• The connection that has been standardised for 100GbE is the CXP (mostly)
• There is no UTP copper standard, and according to comments, never will be.
• The 40GBASE-CR4 uses four Twinax cables (I believe)
• There will be a Twinax version for 100GBASE-CR10 that uses 10 x 10Gbps signalling. (Q. Why ? A. For servers edge connections because there will be someone who thinks that 100GbE connected server will go faster)
• The CFP (C form-factor pluggable) transceiver features twelve transmit and twelve receive 10Gb/s lanes to support one 100GbE port, or up to three 40GbE ports. Its larger size is suitable for the needs of single-mode optics and can easily serve multimode optics or copper as well.
• The CXP transceiver form factor also provides twelve lanes in each direction but is much smaller than the CFP and serves the needs of multimode optics and copper.
• The QSFP (quad small-form-factor pluggable) is similar in size to the CXP and provides four transmit and four receive lanes to support 40GbE applications for multimode fiber and copper today and may serve single-mode in the future. Another future role for the QSFP may be to serve 100GE when lane rates increase to 25Gb/s.
• The current generation of modules are large because of heat dissipation issues due to high power consumption. For example, the CFP is rated for up to 24 watts of power dissipation but also needs to have a range of high density electrical connectors to connect to the baseboard. I take this to mean big, hot and heavy.

Cabling Notes

Presenting these fibres to the connector will required specialised cabling. I don’t have too much information here but this image shows the MPO plug for 40GbE multimode.

• In a 10GbE network, OM3 fiber can span up to 300m while OM4 supports even longer channels.
• In a 40GbE or 100GbE environment, OM3 can be used up to 100m and OM4 up to 150m according to the IEEE802.3ba standard.
• For applications approaching 150m, the cable should be terminated with low loss connectors.
• The MPO connector uses 12 fibre core.
• For 40GbE this means that four cores are unused.
• For 100GbE uses 10Gbps SERDES on multimode, you need 2 x12 core MPO or 1 x 24 core MPO with 2 core per each unused.

Reference: 40 GbE: What, Why & Its Market Potential – Ethernet Alliance

Can I use my existing cabling ?

• From the research I’ve done, I don’t think that you will use your existing cabling infrastructure because you won’t have enough of it.
• I think it’s more likely that you will purchase manufactured cables up to 150 metres with the idea of using them as patch leads to run from location to location within the data centre. There are number of what I call “click/clack” cabling systems that deliver modular cabling solution.
• Therefore any investment in your static fibre plant for the data centre is probably wasted. The method that you run fibre to top of rack and manually fusion splice pig tails is probably over. I predict that we will purchase all fibre optic cables as manufactured items.
• This will help to guarantee the performance and the reuse of the cables as the data centres change over time (thus improving the green initiatives).

References

• 40 gigabit and 100 gigabit ethernet are Here! – Brocade
• Ethernet Alliance
• Cisco FabricPath presentation – received under NDA so no reference permitted. I’ve only used images that are otherwise available on manufacturer websites.

Other posts in the series

1. Tech Review: NETCONF and YANG
2. Futures Review on 40 and 100 Gigabit Ethernet (This post)