◎ SDN Use Case: Firewall Migration in the Enterprise

We all know replacing a firewall is really hard. Even using the same vendor but upgrading to a new platform is risky and time consuming. Switching firewalls that have hundreds or thousands of firewall rules can be an enormous risk and technical challenge where the devil is firmly located in the details. One wrong move and the entire data centre network “gets it in the neck”. If you’ve every owned a CheckPoint firewall, you will know that you want to get replace it as quickly as possible to improve the reliability of your network and reduce costs 1

It’s been while since I climbed onto my “software defined horse” and talked about Use Cases for SDN. Vendors continue to tell me that no one wants OpenFlow and SDN – they say they don’t need it. What vendors lack is imagination, so I’m here to help. This post is describes how SDN could solve the firewall migration challenge.

Of course, it’s change management/ITIL that makes firewall migrations the most miserable project to work on. Endless change meetings and discussions around risk that deliver zero benefit to the business. But, hey, that’s ITIL & ITSM for life defeating experiences.

Firewalls and SDN are about Flows

A firewall doesn’t handle packets, it’s handles flows. That is, the flow between a client & a server from the time the SYN starts until the FIN/ACK tears town the TCP Session. The firewall need to handle all flows in a sessions so that the stateful packet inspection

Lets start the design with a firewall “security locked” between a Router and some Servers in a standard mode for a Internet connection 2.

Sdn firewall migration 1

Lets say that some traffic is coming from the Internet. Each IP session flow would pass through the firewall to the servers and vice versa. And the return session also need to flow through that firewall ? Of course, otherwise the firewall inspection route cannot analyse the flow state of the session and would discard the frame. The yellow arrow shows the traffic flow.

Sdn firewall migration 2

What happens when you want to upgrade that firewall to another model or another vendor. Lets face the fact that you can put up with the CheckPoint FW-1 that causes weekly problems forever. You need to divert traffic flows to your new Juniper or Cisco firewall but the only tools you have is routing. Routing is based on destination addresses only and even using /32 routes can affect multiple firewall rules. Or maybe some policy based routing (which is actually flow based routing).

Sdn firewall migration 3

By far, the most common rule for a firewall is based on SRC IP, DST IP and DST PORT. Although other possibilities exist no one actually uses them much, if at all. Lets say that this “three tuple” is enough to describe 99% of firewall rules.

Firewall Flow Table
SRC IP DEST IP DST Port Action
155.66.77.11 172.32.32.32 80 Drop
155.66.77.12 172.32.32.32 80 Drop
155.66.77.13 172.32.32.32 80 Permit

For this collection of “firewall rules” you can see that destination address flow redirection is the same. If you are using routing, even /32 routes, then you may need to move a complex set of rules for every /32. For very large firewall complexes, this is certainly true.

Is that too granular to match a single IP/Port  ? Consider a “source load split” scenario. What if you move all traffic with a source address of 10.0.0.0/9 through one firewall, and 10.128.0.0/9 through another ? What about 10.0.0.0/12 ? What about a source of 10.0.0.0/12 with a destination of 192.168.0.0/26 ? This approach would allow testing of a new firewall configuration with some smaller fraction of the overall network and reduce the risk to the business.

Here how OpenFlow Helps

Now, you might have heard about OpenFlow. There is a key fact in the name that it’s also identifies traffic flows and also allows a OpenFlow device to manipulate the flows. Here is a sample OpenFlow table showing flow data and a typical set of actions on the data (screenshot from my upcoming book on OpenFlow).

Sdn firewall migration 6

In the context of a secure environment, you can see that OpenFlow has an extended set of criteria for managing network flows. Using OpneFlow on an OpenFlow switch we could selectively identify traffic flows and redirect those flows to a different Ethernet port. Lets add a couple of OpenFlow enabled switches (it could be just one switch, but I’ll show two here for simplicity).

The elements of the solution are:

  1. OpenFlow capable switches (like PICA8 which are low priced and good quality.).
  2. OpenFlow Controller to perform the OpenFlow / Switch integration. (BigSwitch for commercial or Floodlight for testing).
  3. FW Migration App – this is what I’d like someone to deliver.

Sdn firewall migration 7

Here is how I see this working once you have the basic equipment in place

  • The FW Migration App reads the configuration from the old firewall.
  • App parses the configuration for firewall rules and build a flow table based on the rules.
  • App then loads the firewall rules into the new firewall.

The application would need to be able to read the configuration using an API or, more likely, by scraping the CLI. As a one off activity that focusses on the just the firewall rules, this should be less complex than attempting to parse the entire configuration including routing, inspection rules and other random firewall features. Configuration of those “advanced features” could be the second phase  (or never phase) of product development.By keeping the configuration extraction simple the product remains simplifed.

The second phase of configuration is to start the cutover on a rule by rule basis. Here is rough outline of the flow:

  • FW Migration App GUI has selection button of available rules ready to cutover.
  • Select the rules for cutover from the list and select apply.
  • OpenFlow rules are sent to the OpenFlow switches.
  • Flows are now diverted to an alternate firewall.

This diagram summaries the process.

  1. Read the configuration of the legacy CheckPoint firewall, parse the standard rule base. Highlight unrecognised rules or actions for user intervention 3
  2. Push the rules into the new firewall. Engineer performs a manual verification.
  3. Engineer selects a few rules for migration to the new firewall and OpenFlow rules are dispatched to your OpenFlow switches. All IP Flows matching the flow table are diverted into a different Ethernet port where the new firewall is connected.

Sdn firewall migration 4

Now you have forced the flow over to the alternate firewall while other flows continue to traverse the old firewall.

Sdn firewall migration 5

Once you have migrated all the rules, you can shut down the old CheckPoint and rejoice like these fine people did.

Scale

The OpenFlow flow direction is performed in silicon therefore forwarding performance is not consideration. Using policy based routing can be tricky since not all hardware supports line rate forwarding. 4 There are limits in the switch devices on the maximum number of OpenFlow entries in the TCAM that you may have to be careful. Some of todays switching can only hold a few thousand OpenFlow flow entries but is expected to increase in the next year or two as the next merchant silicon chips arrive.

How much is this worth ?

I’ve performed about a half a dozen large scale firewall migrations over the last 10 years. Each of these projects costs millions of $moneys in project time and resources. The real cost is change management, location of service owners and planning meeting which take thousands of man hours. The largest project I’ve worked had a 10 person FTE team for nine months to migrate a firewall cluster (only 2 engineers) plus additional time for executive intervention, management review etc etc etc. I’ve heard stories of a single large institution with 900 obsolete firewalls. These units are costing multi-millions in post-EOL support contracts but that’s cheaper than the cost of upgrading them.

 

The EtherealMind View

It’s not a billion dollar per annum market so the big vendors won’t be interested in developing this as a product. But I figure a smaller vendor could readily enter the OpenFlow application space and start attaching to high value customers with a low risk product. Because early stage products can rely on the engineer for manual validation, which would happen in a normal project, there is actually very little business risk.

After thinking a bit more, I’ve also realised that you don’t need an application – it would be possible to use the current versions of BigSwitch & Floodlight controllers and manually define flows. Not as simple as using an App but it would get the same result.

That’s my example of an SDN Use Case that is just one of many that SDN/OpenFlow makes possible.

Let me know if you are interested, maybe someone who works for venture capital firm is reading :)

 


  1. The maintenance cost on a CheckPoint firewall makes Cisco looks like a dodgy used car dealer.  ↩
  2. Well, highly simplified anyway. I’ve built arbitrarily complex firewall clusters in my time with three layers of firewalls and dozens of DMZ. The basic premise holds true here.  ↩
  3. Remember that the alternative is to do the whole thing manually anyway so a few manual rules isn’t a show stopper for the product.  ↩
  4. I’ve spent hundreds of hours researching whether a given piece of hardware supports the necessary features. Or determining products that can do this. ↩
  • http://twitter.com/cloudtoad Derick Winkworth

    Hey Greg… I was confused by your description of flows in firewalls. Are you suggesting OF might also communicate firewall state? Or are you just looking to migrate one application at a time over to the new firewall, not at all concerned with transferring state (i.e., sessions lossed)?

    • http://etherealmind.com Etherealmind

      I’m looking to migrate a service down to as granular as one firewall rule at a time.

      Not suggesting OpenFlow can communicate state. That’s simply not possible since state inspection algorithms are so different. Losing session state is not a problem – you have to migrate the firewall one way or the other (excepting corner conditions for 0.01% of people is which case this still isn’t a solution. There is no solution for this problem anyway).

  • http://twitter.com/northlandboy Lindsay Hill

    I can’t help but think that if we had tools like this, that 10-person project would have only needed…9 people. Yep, one of the engineers would get fired.

    I’ve done some decent-sized firewall migrations in my time too. I don’t know how much the tools would help, or how you would be able to prove they were “safe” enough to de-risk it for the ITSM people.

    Most of the time I’ve spent on firewall migrations has been more of the “Who knows what this does? Does anyone use this anymore?” The actual technical work is relatively straightforward. Of course the problem was with testing all the myriad service combinations, and I guess this could help. But maybe we would just end up with migrations dragging on for years, never getting completed? I’ve been involved in some big network migrations that took a long time – troubleshooting was a nightmare, especially for services that used some old and some new infrastructure.

    But at least someone is publicly thinking about use-cases for this sort of tech. Pleasing to see.

    • http://etherealmind.com Etherealmind

      I agree with your point, but the rollback is also very simple. Flip one service/rule, and rollback as needed. Thats the best you can hope for in this situation.

  • cryptochrome

    awesome post. you should startup a company that delivers that as a product. go!

    • http://etherealmind.com Etherealmind

      You can do this manually with a controller today. Use Floodlight (Opensource) and add flow rules manually.

      I’m available to help with a startup though.

      • Nimit Shishodia

        Hi etherealmind, we will be looking forward to bulid a FWV_Mig_App on our platform Evolve, which contains other cool apps like this..
        Ofcourse we are startup, so please get in touch
        nimit@ecodenetworks.com

  • http://twitter.com/neatherweb neatherweb

    I think one better would be a firewall that is controlled by OpenFlow. Accept/reject decisions can be made by the controller for new flows, then the controller can shift the flow down the pipeline to the ‘firewall’ table, which is where all the TCP state table and firewall’y features can be implemented in hardware. The controller might also be able to insert into other tables on a flow by flow basis for IPS or other tasks best done on-box or in hardware.
    That’s my wish list for SDN and firewalls. (Even if it belongs with the other unicorns. )

    • http://etherealmind.com Etherealmind

      I like the way you think.

  • Kallisti

    In a idealized SDN world, shouldn’t the firewall just speak directly to the controller as well? I mean, all the flows would be defined and documented there already, why is there a need to define flows on a firewall at all anymore? As I see it, SDN obsoletes firewalls to some degree (basic policy stuff). You would only need a firewall if you do not trust the switch on at least one side, to secure the underlay network and for additional security features like typical IDS functionality. Still, all the policy rules should come from the SDN controller, which also makes it easier to involve application people to the process, as they just need to define the flow from source to destination and application and the controller can configure _all_ firewalls along the path, not just one, spanning multiple security zones.

    A clever controller could even allow you to define applications objects and assign servers and security policies which would then allow you to validated whether a certain flow is allowed at all for certain security zones.

    There is pretty cool stuff you could do with software, once devices exist that can do all that with the required performance. :-)

    • http://etherealmind.com Etherealmind

      In a perfect world, you are correct. I don’t believe that firewall vendors are going to participate in OpenFlow though – they aren’t concerned about reducing operational costs. .

      In this article, I’m looking a migrating existing firewalls. Say CheckPoint FW-1 to Juniper SRX.

  • http://twitter.com/sjiveson What Lies Beneath

    I’m perhaps displaying my minimal knowledge of OpenFlow but I wonder how the return traffic will be directed back to the correct firewall? Would you need rules for the return traffic (application created or otherwise) or is there a stateful connection table/CBAC type functionality in OpenFlow?

    Regardless, (and now perhaps I’m being arrogant) but I’m pretty sure I could do this with a couple of load balancers and one programming language or another depending on the vendor. Other solutions are not the point of your article I know (and it’s a great use case) but I’d certainly like to at least highlight the fact that, like cloud and hosting, SDN (it’s intention and theory at least) isn’t necessarily new and some pretty mature SDN-like technologies already exist.

    • http://etherealmind.com Etherealmind

      You will, of course, need to program the forward and reverse flow path in both OpenFlow enabled switches. Programming an OpenFlow capable switch is a trivial task.

      You are drawing the comparison the a Load Balancer acts like a flow control. This is true however, a load balancer costs approximately 50 times more than an Ethernet switch. Even then, it’s not easy to do (I know, I’ve tried it).

      “some pretty mature SDN-like technologies already exist” – there are some flow technologies today but none of them are easy t consume as SDN. SDN is part flow management, and part tools for consumption. There is a big difference imho.

      • http://twitter.com/sjiveson What Lies Beneath

        I didn’t think cost would be an issue in this case but I see your point, particularly in an environment that had already moved to using OF.

        On the subject of the reverse flows, perhaps I’m being naive (or simply need to do my research before commenting) but handling that doesn’t sound too simple as you’d have to build your own state table for at least two points in the network and correlate the two. Still, that’s where the magic of the centralised controller (and the software it might run) and the startup come in I’m sure…

  • Dan Backman

    Greg,

    This is a cool use-case, and it actually brings up a better point: we don’t currently have a good way to steer traffic within a traditional packet based network. Yes, you can do creative things by hopping through VLANs, adding policy-based routing, or even doing full-blown MPLS/TE. But there are many cases like this where you really want the ability to steer traffic by exception to the standard forward rules.

    In an ideal world, this should be a property of the network itself, and shouldn’t have to be yet another device sandwich within the data path.

    SDN gives us a great way to look at adding this type of path control to the network, but also allows us to look at how we would effectively add this to the provisioning workflow as well. Adding specific forwarding behaviors with OpenFlow is cool, but we need a better way to actually integrate this with applications — so adding/changing/removing these forwarding rules doesn’t put us back to the same problem of managing firewall rules.

  • Nimit Shishodia

    awesome greg..you have given brilliant idea(s)..thanks

  • Pingback: Be The Steamroller Not The Road | NetworkStatic | Brent Salisbury's Blog

  • Pingback: Big Switch Introduces Switch Light

  • Pingback: SDN, OpenFlow and OpenDaylight Updates and Recordings