23rd May 2012

Interesting Switch Designs for Data Centre Racks From Juniper

As a result of my recent post on Stackable vs Chassis Based Switches I was looking Juniper Virtual Chassis Technology and found some really interesting ideas on the architecture.

Intro to Juniper Virtual Chassis Technology

From the VIRTUAL CHASSIS TECHNOLOGY BEST PRACTICES here is a brief description

Juniper Networks® Virtual Chassis technology is a feature of the Juniper Networks EX4200 line of Ethernet switches allowing the interconnection and operation of switches as a unified, single, high bandwidth device. Up to 10 EX4200 switches may be interconnected via dedicated Virtual Chassis ports on each device, or through optional uplink module ports that are configured as Virtual Chassis ports, with a combined backplane bandwidth of up to 128 Gbps.

OK so you get the idea. A stack connection is always a fat bandwidth connection between each switch that attempts to extend the switching backplane to a much larger physical space than just a few centimetres between chips, or a 75 centimetre haul down a backplane. However, the provision of high bandwidth over a cable or a backplane is limited by a number of physical problems such as clocking, signal waveform and importantly, distance. You can’t beat the speed of light, and it’s hard to get a square waveform at high frequencies.

But the Juniper stack connection must have a lot of parallel bandwidth in there, since they are getting 64Gbps in their connection (note that is twice as fast as a Cisco C3750 stack connection). But interestingly, the Juniper Stack connector cable can be up to 5 metres in length. This opens up a number of interesting design possibilities.

For example, the Cisco C3750 has a long stack cable that can be 75cm long, and is used when you stack gets bigger than four units. Without that ‘extra special’ cable, you can’t get the stack loop installed. This means that a Cisco C3700 stack must always be adjacent and you don’t have a lot of choices for cable management.

Design Options

So lets look at the design options proposed by Juniper in the best practices document.

Daisy chained Ring

juniper-virtual-chassis-1.jpg

This is pretty much the standard layout for a stack. Short distance between one leg and a long return cable to complete the redundant path.

Braided Ring

This is different. I’m guessing that the ring arbitration system doesn’t need the stack to be in any particular order. Combine that with an electrical signalling that allows high speed (in this case 64Gbps on that cable) for up to 5 metres and you can actually have those switches with an end to end span of 22.5 metres.

juniper-virtual-chassis-2.jpg

Top of Rack, Daisy Chain

Lets turn that Braided Ring into something practical. What about a top of rack design for low density server zone.

Figure 15 shows an example of extending the data center, top of rack deployment across several racks. This figure illustrates a braided-ring cabling configuration, which allows spacing of up to 5 meters between alternate racks.


Caption Text.

Caption Text.(Click for a full size image)

In this practical scenario, this means a 48-port switch in the top of each rack with up to ten racks in a row. Multiple uplinks can can be allocated according to the bandwidths requirements as needed. For certain low ethernet densities this a very interesting solution. I can imagine using a design that has this setup twice. That is, I have two switches at the top of every rack, but connected as show here to provide dual switch redundancy. And I’m thinking that this would be a cost effective solution.

Not a fan of Stackable Switching

I recently wrote about Chassis vs Stackable switches and that I don’t think Stackable switches are very good. But the design convenience of this option with ease of use and low cost, might outweigh my concerns about reliability. And since this type of design is only going to appeal to low end, low cost customers then it might well be the correct choice.

The Etherealmind View

It’s always interesting to look at other vendors equipment and learn about the different techniques to solve problems. This design shows that Cisco doesn’t have all the answers and other vendors have solutions that can be very different, and better. Note to self: must look at other vendors more often [grits teeth], even if that is a lot of work.

This post is copyright of Thropos Ltd ©2008-2011 at Etherealmind.com - contact | email: greg.ferro@packetpushers.net - twitter: @etherealmind | All rights reserved
About Greg Ferro

Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualization. He has over 20 years in IT, in wide range of employers working as a freelance consultant including Finance, Service Providers and Online Companies. He is CCIE#6920 and has a few ideas about the world, but not enough to really count.

He is a host on the Packet Pushers Podcast, blogger at EtherealMind.com and on Twitter @etherealmind and Google Plus

  • Jack

    Greg,
    I think you’re missing some information on Cisco’s offering. First, Cisco’s E series 3750s have doubled stackwise throughput to “64 Gbps bidirectionally” and now has local switching. Second, Cisco has a 3M stackwise cable that allows for 9 switches abreast.
    http://bit.ly/cn2EGa
    I’m not sure if there are any limitations to the ring diameter (cable feet).

    Both of those points however, don’t bring the E series to the level of the Juniper. For example you can’t extend the stack bus with 10GbE uplinks and the stack cables are 2M shorter. I’m just making the point that Cisco’s offering is not as far behind as you made it out to be. But hey, second is still second right?

    Incidentally, I gather from Juniper’s Virtual Chassis Technology Best Practices document that in Figure 14 (above) they’re extending the stack ring via 10GbE. I don’t have any figures for it but it would seem then that stack throughput is variable. Is that accurate?

  • http://eatyourpets.com/ yeled

    Have you seen how expensive those VC cables are? It’s not worth it until you spend £200-400 on cable management between each rack.

    • t3lurid3

      Couple of other advantages with the Juniper take on stackable. 1Gb SFP and 10Gb ports can be pressed into service to provide additional VC connections at a pinch and provide additional meshing if that is of value in a particular use case and the shortest path calculation means that VC hops are kept to a minimum unlike cisco where the traffic goes around in a ring. Add in graceful RE failover and this approach is streets ahead of the 3750 which as a platform has never been more than just okay.

    • http://etherealmind.com Greg Ferro

      Guess they have unobtainium cores and awesomesauce shielding.

      -grin-

      • t3lurid3

        Woah that did sound a little too close to vendor presales babble – I am listening to my Juniper SE too much. That said these are miles better than 3750s, their performance is more in line with the Cisco 4948s/4928s which are not stackable. I have to say we don’t tend to stack much though be it Cisco or Juniper if we have the choice.

  • http://www.packetwrangler.com/ Jay Moran

    These, they are good. Make sure you’re running later versions of code though. The first 9 months of code was NOT good at all.

  • Pingback: Drowning in Features « The Network Therapy Blog

  • Pavel Lunin

    EX4200 can also for a cluster using ethernet links. Even optical. This allows to have even more interesting distributed VC architectures.

    The arguments of big chassis vs stackable used in the referenced article are more Cisco oriented. When C6500 are really more interesting from both feature and hardware stability point of view, Juniper’s EX3200/4200 are much more featured than EX8200. The only situation when EX8200 is better than VC of EX4200 is when you need high density of 10G (though EX4500 is on its way it is relatively raw and doesn’t support virtual chassis yet) or higher than 128G ring backplane capacity.

    Well, as a Juniper oriented guy I can say Juniper’s stackables switches are much more interesting than big chassis.

  • Pingback: Brocade HyperEdge and “Effortless Network” — My EtherealMind