Thought: What Facebook Wedge, Cumulus Linux Shows About Network Operating Systems

Yesterday, Facebook went public on their FBOSS network operating system for whitebox Ethernet switches. This shows that NOSs like Cisco’s IOS and Juniper’s Junos are under attack and devalues those businesses. The destruction of the software moat means that customers have far more choice for their networks.

Introducing “Wedge” and “FBOSS,” the next steps toward a disaggregated network | Facebook Engineering Blog

There are two aspects of this announcement that interest me on hardware and software.

Hardware Facebook Wedge is yet another whitebox switch implementation based on Broadcom Trident silicon and COTS components for the management plane. Using x86-based server inside the switch enables standard Linux. It is a yet another sign that the “silicon moat” that protected incumbent vendors from competition is rapidly eroding. For the last two decades, a new networking startup needed to invest tens of millions over a 3 to 5 years period to bring a network switch to market. Companies like Juniper, Dell and HP are increasingly turning to Broadcom, Intel or Marvel for the that silicon. At the same time, startups like Pica8, Big Switch, Cumulus Networks and Penguin Computing are able to offer switch hardware to the market.

Network Operating Systems (NOS) FBOSS is the NOS on the device, based on Linux, developed by a very small team (less than 10 people). FBOSS is based on Broadcom OS that is widely available and the team was able to add code for feature that Facebook sees as relevant to it’s unique requirements. What I know about FBOSS from discussions and OCP sessions is that they are driving switches at above 95% utilisation and Facebook has found that most commercial switches drop packets, are unreliable and lack OS performance when implementing path updates.

Building a NOS is ‘easy’ – devalues products like IOS and Junos Facebook has the money and will to create a project team of 10 people to design and test the FBOSS NOS but Its hard to see a large financial or retail company developing their own NOS. At the same time, commercial NOS like Cisco IOS / NX-OS and Juniper Junos have become less valuable. At the very least, the mystique around NOS has changed to become more like a commodity.

The best expression of this process is that startups like BigSwitch Switch Light or Cumulus Networks Linux have developed their own NOS.

The Etherealmind View

Lets accept that commercial NOS products remain valuable through their incumbency and proven function. For some customers, these commercial and expensive NOS products will be their preferred platform in the same way that companies choose Windows Server instead of Linux.

The arrival of independent NOS products has parallels with Windows/Linux. Some customers choose Windows at a high price, some customers choose commercial Linux and others will choose free Linux.

Now that there are choices in the network marketplace, I’m watching to see what choices customers make ? Will free & open source NOS solutions get traction in the marketplace ?

I’m saying yes, it’s going to be big because there is too much money to be saved – both CapEx and OpEx. And that a trend that is seems to be unstoppable through 2014/2015.

  • Samir Savla

    Do you think this will also impact Brocade Communications, since they have a big push towards SDN.

    • Etherealmind

      No more than it affects other incumbent vendors. In its current form, FBOSS & Wedge are tightly focussed on Facebook needs and not suitable for most enterprises.

      • Chalon Duncan

        I am glad you pointed this out. Facebook’s own engineering blog actually states “With “FBOSS,” all our infrastructure software engineers instantly become network engineers”. If a company is not resourced to operate in the same way this is not really realistic for the masses. I am excited to see how this evolves though. #CoolTechDisruption

  • Greg Mueller

    While it will take some time for adoption, I expect FBOSS to be forked, assuming the license allows it, and other open source NOS projects to become established just like thousands of open source projects that exist today. When a NOS hits the critical mass the way the apache web server did, and companies can sell support contacts to enterprises, then an open source NOS can become entrenched in the enterprise. The questions are when will it happen and who will ride is success.

  • Mihir Vyas

    Google seems to be doing a similar thing with their Data centers. Google has been pushing actively on white box switches. Frankly, these are companies whose products are massively dependent on data centers and the huge amount of data they require. An interesting question would be, do you foresee Service providers and enterprises go through this route? Or will they stay with tried and tested vendor solutions? Any thoughts?

    • Etherealmind

      First, lets recognise that Enterprise Data Centres are different and not everything can be adapted.

      But what can be proven at scale can downsized for the enterprise. So SDN is a big part of those networks and knowing that it is used means we can use that as “permission” to implement new technologies. And your data centre can have many networks – why not have an L2 STP network, a L3 ECMP network and an SDN network where each one has a specific purpose. The days of requiring just one network in the data centre are behind us. Get the right network for the right purpose/use case.

      Things that I think are possible/permitted now are whitebox Ethernet, new operating system on devices, containers and ECMP L3 networking with overlays.