◎ SDN Use Case: Firewall Migration in the Enterprise

18th March 2013 by Greg Ferro

We all know replacing a firewall is really hard. Even using the same vendor but upgrading to a new platform is risky and time consuming. Switching firewalls that have hundreds or thousands of firewall rules can be an enormous risk and technical challenge where the devil is firmly located in the details. One wrong move and the entire data centre network “gets it in the neck”. If you’ve every owned a CheckPoint firewall, you will know that you want to get replace it as quickly as possible to improve the reliability of your network and reduce costs 1

It’s been while since I climbed onto my “software defined horse” and talked about Use Cases for SDN. Vendors continue to tell me that no one wants OpenFlow and SDN – they say they don’t need it. What vendors lack is imagination, so I’m here to help. This post is describes how SDN could solve the firewall migration challenge.

Of course, it’s change management/ITIL that makes firewall migrations the most miserable project to work on. Endless change meetings and discussions around risk that deliver zero benefit to the business. But, hey, that’s ITIL & ITSM for life defeating experiences.

Firewalls and SDN are about Flows

A firewall doesn’t handle packets, it’s handles flows. That is, the flow between a client & a server from the time the SYN starts until the FIN/ACK tears town the TCP Session. The firewall need to handle all flows in a sessions so that the stateful packet inspection

Lets start the design with a firewall “security locked” between a Router and some Servers in a standard mode for a Internet connection 2.

Lets say that some traffic is coming from the Internet. Each IP session flow would pass through the firewall to the servers and vice versa. And the return session also need to flow through that firewall ? Of course, otherwise the firewall inspection route cannot analyse the flow state of the session and would discard the frame. The yellow arrow shows the traffic flow.

What happens when you want to upgrade that firewall to another model or another vendor. Lets face the fact that you can put up with the CheckPoint FW-1 that causes weekly problems forever. You need to divert traffic flows to your new Juniper or Cisco firewall but the only tools you have is routing. Routing is based on destination addresses only and even using /32 routes can affect multiple firewall rules. Or maybe some policy based routing (which is actually flow based routing).

By far, the most common rule for a firewall is based on SRC IP, DST IP and DST PORT. Although other possibilities exist no one actually uses them much, if at all. Lets say that this “three tuple” is enough to describe 99% of firewall rules.

Firewall Flow Table
SRC IP	DEST IP	DST Port	Action
155.66.77.11	172.32.32.32	80	Drop
155.66.77.12	172.32.32.32	80	Drop
155.66.77.13	172.32.32.32	80	Permit

For this collection of “firewall rules” you can see that destination address flow redirection is the same. If you are using routing, even /32 routes, then you may need to move a complex set of rules for every /32. For very large firewall complexes, this is certainly true.

Is that too granular to match a single IP/Port ? Consider a “source load split” scenario. What if you move all traffic with a source address of 10.0.0.0/9 through one firewall, and 10.128.0.0/9 through another ? What about 10.0.0.0/12 ? What about a source of 10.0.0.0/12 with a destination of 192.168.0.0/26 ? This approach would allow testing of a new firewall configuration with some smaller fraction of the overall network and reduce the risk to the business.

Here how OpenFlow Helps

Now, you might have heard about OpenFlow. There is a key fact in the name that it’s also identifies traffic flows and also allows a OpenFlow device to manipulate the flows. Here is a sample OpenFlow table showing flow data and a typical set of actions on the data (screenshot from my upcoming book on OpenFlow).

In the context of a secure environment, you can see that OpenFlow has an extended set of criteria for managing network flows. Using OpneFlow on an OpenFlow switch we could selectively identify traffic flows and redirect those flows to a different Ethernet port. Lets add a couple of OpenFlow enabled switches (it could be just one switch, but I’ll show two here for simplicity).

The elements of the solution are:

OpenFlow capable switches (like PICA8 which are low priced and good quality.).
OpenFlow Controller to perform the OpenFlow / Switch integration. (BigSwitch for commercial or Floodlight for testing).
FW Migration App – this is what I’d like someone to deliver.

Here is how I see this working once you have the basic equipment in place

The FW Migration App reads the configuration from the old firewall.
App parses the configuration for firewall rules and build a flow table based on the rules.
App then loads the firewall rules into the new firewall.

The application would need to be able to read the configuration using an API or, more likely, by scraping the CLI. As a one off activity that focusses on the just the firewall rules, this should be less complex than attempting to parse the entire configuration including routing, inspection rules and other random firewall features. Configuration of those “advanced features” could be the second phase (or never phase) of product development.By keeping the configuration extraction simple the product remains simplifed.

The second phase of configuration is to start the cutover on a rule by rule basis. Here is rough outline of the flow:

FW Migration App GUI has selection button of available rules ready to cutover.
Select the rules for cutover from the list and select apply.
OpenFlow rules are sent to the OpenFlow switches.
Flows are now diverted to an alternate firewall.

This diagram summaries the process.

Read the configuration of the legacy CheckPoint firewall, parse the standard rule base. Highlight unrecognised rules or actions for user intervention 3
Push the rules into the new firewall. Engineer performs a manual verification.
Engineer selects a few rules for migration to the new firewall and OpenFlow rules are dispatched to your OpenFlow switches. All IP Flows matching the flow table are diverted into a different Ethernet port where the new firewall is connected.

Now you have forced the flow over to the alternate firewall while other flows continue to traverse the old firewall.

Once you have migrated all the rules, you can shut down the old CheckPoint and rejoice like these fine people did.

Scale

The OpenFlow flow direction is performed in silicon therefore forwarding performance is not consideration. Using policy based routing can be tricky since not all hardware supports line rate forwarding. 4 There are limits in the switch devices on the maximum number of OpenFlow entries in the TCAM that you may have to be careful. Some of todays switching can only hold a few thousand OpenFlow flow entries but is expected to increase in the next year or two as the next merchant silicon chips arrive.

How much is this worth ?

I’ve performed about a half a dozen large scale firewall migrations over the last 10 years. Each of these projects costs millions of $moneys in project time and resources. The real cost is change management, location of service owners and planning meeting which take thousands of man hours. The largest project I’ve worked had a 10 person FTE team for nine months to migrate a firewall cluster (only 2 engineers) plus additional time for executive intervention, management review etc etc etc. I’ve heard stories of a single large institution with 900 obsolete firewalls. These units are costing multi-millions in post-EOL support contracts but that’s cheaper than the cost of upgrading them.

The EtherealMind View

It’s not a billion dollar per annum market so the big vendors won’t be interested in developing this as a product. But I figure a smaller vendor could readily enter the OpenFlow application space and start attaching to high value customers with a low risk product. Because early stage products can rely on the engineer for manual validation, which would happen in a normal project, there is actually very little business risk.

After thinking a bit more, I’ve also realised that you don’t need an application – it would be possible to use the current versions of BigSwitch & Floodlight controllers and manually define flows. Not as simple as using an App but it would get the same result.

That’s my example of an SDN Use Case that is just one of many that SDN/OpenFlow makes possible.

Let me know if you are interested, maybe someone who works for venture capital firm is reading 🙂

The maintenance cost on a CheckPoint firewall makes Cisco looks like a dodgy used car dealer. ↩
Well, highly simplified anyway. I’ve built arbitrarily complex firewall clusters in my time with three layers of firewalls and dozens of DMZ. The basic premise holds true here. ↩
Remember that the alternative is to do the whole thing manually anyway so a few manual rules isn’t a show stopper for the product. ↩
I’ve spent hundreds of hours researching whether a given piece of hardware supports the necessary features. Or determining products that can do this. ↩

11 Things About Using A Transparent or Layer 2 Firewall ?

5th June 2012 by Greg Ferro

I often have discussions with people who want to deploy their firewalls in Layer 2 mode. This isn’t a decision to take lightly and needs a lot of careful planning.

Here are my notes both for and against Layer 2 Firewalls. Keep in mind that L2 Firewalls are most common in data centres and are typically retrofitted to provide security.

In general terms, I would only recommend using L3 firewalls for any new design or new build.

Reasons For Using Layer 2 Firewalls
Reason	Explanation
No change to existing IP addressing or Servers	This is the most common reason. You have a legacy infrastructure that you need to add new firewalls without changing IP addresses
Routing protocols can establish adjacencies through the firewall	Passing OSPF neighbours through a firewall is possible but a “bad idea”™
Protocols such as HSRP, VRRP, GLBP can pass	For certain designs and requirements, you might even want to do this.
Multicast streams can traverse the firewall	A major reason. Most firewalls are “feature incomplete” when it comes to Multicast and cause much pain in deployment
Non-IP traffic can be allowed (IPX, MPLS, BPDUs)	There is still legacy traffic (in the data centre especially) and you may be firewalling IP but passing legacy traffic.

Reasons Against Layer 2 Firewalls
Reason	Explanation
No visibility makes it hard to troubleshoot	Troubleshooting a L3 network is much easier since each step of the path is known
Layer 2 Firewalls are hard to detect	This isn’t a good reason. Security by obscurity is false security. If you need to obscure your security controls then you have a deficiency in your security process
Can easily insert loops into networks	Strong design and implementation discipline is necessary
Only allows for two interfaces, inside and outside (no DMZ interfaces)	Enforced design limitation may be functionally incomplete or need more firewall instances
NO dynamic routing protocol support or VPN support	The firewall control plane cannot insert itself into the IP circuit and cannot provide a lot of services
Specific design limitations	Most Layer 2 implementation lose a lot of features. Often unexpected features like QoS, VPN or GRE tunnels. You will need to research very carefully
Dual Homed Devices can creates L2 Path defects	Dual homed servers using active/standby are OK, but Active/Active can cause of lot of pain ( MLAG can help)

The EtherealMind View

As a general rule, I would not deploy Layer 2 Firewalls in a network. The negative aspects outweigh the positive features as a general topic. There are times when nothing else will do and a L2 solution is the only way. Just be careful to use your knowledge of first principles to consider the design you need to use, and then test it. You don’t want to find out what the real problems are after you have completed the deployment.

I imagine that many people will have different views. Lets hear them and I’ll try to respond.

Design: Cisco Firewall Services Module Virtualization Design Traps

13th August 2009 by Greg Ferro

[Read more…]

Blessay: Designing Enterprise DMZ and multilayer Firewall Clusters

2nd August 2009 by Greg Ferro

In modern Enterprise networks, you typically have many clusters of firewalls protecting assets in your network. Since we use two or more layers of firewalls, we can put our DMZ for intermediate security zones in different places in our network. Lets gather together the different options and consider the merits or not, and sometimes how they ‘self-build’.

Zones and Separation

For many networks, you need to separate different areas of the network. When separated for security reasons, these zones typically have a firewall put in to provide security and control that traffic that flows between these. It gets more complicated when there are services that needs to belong to two or more zones and so we have DMZ ((de-militarized zones)). DMZ have been around for along time, but there are many more choices for implementation.

Lets look into some of these implementation and design ideas.

Zones

For this document, I will focus on just two zones, an external and internal. The external zone will be untrusted and where evil must be stopped (like the Internet for example), and the internal zone is where the users are.

Of course you can have more than one Internal Zone or more than one External Zone.

For a normal network, the traffic flows could be demonstrated to go between all zones at any time.

But when you add a firewall, security is enforced.

Two Firewall Layers

For all large companies, it is policy to use two firewalls for the gateway to the Internet. This fundamental idea is a long held belief in the security community that firewalls aren’t really secure. It’s probably is based on the fact that Checkpoint firewalls, the only firewall vendor at the time, had a major problem in the late nineties which meant they didn’t actually work, and could be easily bypassed. Ever since this time, security policy has mandated the use of two firewalls to remove this risk [slider title=”but question of whether it is more secure remains unclear”]One of my favourite ways to upset security consultants from the major global firms, is to demand that clearly document how much more securetwo firewalls would be. For example, if I have a Cisco ASA and Nokia/Checkpoint, is it ten percent better ? Twenty percent ? Of course there is no answer for this, and the contortions that they will go to justify the statement is quite delightful. Sadly, most of these so-called security consultants have never even thought about it.[/slider]

What about product selection ?

For most people the choice has already been made as the firewalls have been in place for many years and then had the external firewall grafted in (or less commonly, the internal firewall). My recommendation is always to use a Cisco ASA as the external firewall and Juniper NetScreen as the internal firewall.

What ? No Checkpoint ?

Definitely not. The capital cost, operational cost and stupid-assed complexity of a Checkpoint/Nokia solution is horrendous. Not only are they expensive to buy, but very expensive to operate because they are so complicated. I haven’t seen a ‘proper and secure’ Checkpoint firewall in large companies because no one can make them work properly. As soon as problems start, all of the security features are turned off to get things working, and then they are never turned back on because of change control. This makes them the worst firewall product around.

And while it’s true that you COULD make Checkpoint secure, in the real world, you don’t want to be hiring Checkpoint experts just to manage firewalls, you need network people who are multi-skilled and part of cross-functional multidiscipline team. The days of the “firewalls only” team have passed. People who manage Checkpoint firewalls will need to be completely focussed on that one product, and will not usually have enough time to work on other parts of the network. This is poor value for money for most organisations.

Location – Where do you put the DMZ ?

Now DMZ are often specified by security people as a intermediate security zone for hosting systems that need to exposed to external parties, but also need to send data to the internal network. In a dual layer firewall, you could choose create a DMZ in a number of the different places.

To create a DMZ you need to

create some VLAN’s on your switch infrastructure
an IP interface on Firewalls with an address from a IP Subnet that you have allocated
firewall rules that permit / deby traffic to and from the DMZ

Lets look at the different options for where you might want to locate the DMZ

DMZ on the External Firewall

You could choose to put the DMZ on the external firewall, like so:

It is my opinion that this is probably not the right place. If you believe in having two firewalls for security, then putting your services behind a single firewall from the external and untrusted zone is not consistent. In fact, you see this reasonably often when the external firewall was the ‘original’ firewall, and the internal firewall was added later.

DMZ on the Internal Firewall

You could choose to the put the DMZ on the internal firewall, like so:

The reasons that I most prefer this design is:

traffic from the external and untrusted source passes through two firewalls thus meeting the intention of dual firewalls.
traffic to the internal network is always more complicated, and has more flows. Consider all of the administration traffic to the servers in the DMZ. Therefore, passing internal traffic through a single firewall reduces the cost of ownership by reducing the numbers rules needed in the firewalls.
its easier to understand. Because all external flows pass through the external firewalls, it is consistent with operational troubleshooting.

DMZ between the Firewall’s

This is starting to get clever, and is actually very common. You see, once you start adding DMZ’s to your network, you can’t have just one, you always end up with a quite few.

This type of DMZ Design looks really attractive, and people without a lot of design experience think that this is simply brilliant. I mean, it looks really marvellous when you draw it, and and it JUST LIKE THE RIGHT THING. Here is what is wrong with this idea:

Routing – the servers in the DMZ need to have routing tables to decide which interface to send traffic. My life is too short to spend time explaining routing to server engineers (even when Microsoft includes it in their curriculum).
Routing – on the firewall. Your firewall is not a router, and should’nt be used like one
Testing and Service – you cannot access the outside DMZ interfaces from the internal network without really painful procedures

Accessing the outside interfaces

Let me draw accessing the outside interface. After all, you are monitoring these services for availability aren’t you ?

As you can see, sending data to that external network means quite a lot of work. Routing, firewall rules and this all adds up to something that can easily go wrong and costs extra money to build and maintain.

DMZ bridges from the External to the Internal (Servers as a Firewall)

Now this idea usually comes from someone on the server team, because they sometimes think that servers are a firewall. In fact, the entire security world believes that servers are the very thing we are trying to protect and that operating systems such as Windows and Linux are not to be trusted. Still, I’ve seen it a few times and it’s always been a bad idea.

So, some recommendations.

For the reasons I have outlined, the DMZ on the Internal Firewall makes the most sense. It’s easy to operate, and keeps the complexity on the internal firewall. I haven’t discussed it here, but it also works better when you have HA firewalls.