Following from the previous post where we talked about the Design requirements, this post looks at the practical configuration
Load Balancing with Source NAT
The solution to the problem, im my opinion, is to use Source NAT on the load balancer so that all packets appear to originate from the Load Balancer itself.
- Client sends IP packet to Load Balancer VIP
- LB analyses server farm for that VIP, selects server, and uses that destination address
- LB determines which outbound interface to the server and uses the NAT pool on that interface.
- LB rewrites the IP Header and dispatches the packet.
- Server receives the packet, process the request and passes data to the application.
- server sends reply packet back to the LB.
- LB checks it’s state table to determine which client was the source.
- LB rewrites the packet and forwards out the interface.
Thus every packet received on the VIP has it’s Source and Destination address re-written. The Server will route the packet back to the load balancer, which will modify the addressing and forward the reply back to the client.
There are several advantages in big networks:
- the LB can be put anywhere in the existing network (subject to bandwidth concerns)
- the server can be anywhere (subject to bandwidth concerns)
- no change to server configuration
- limited change to routing (if any)
- the LB is only part of the failure domain for client data – plausible deniability for any server administration problems (very important this one)
In this example we look at how to configure this on Cisco ACE.
switch/ACE-TEST# sh run Generating configuration....
! Setup logging to console (not enabled by default)
logging monitor 7
!ACLs for the interface - traffic is not permitted by default.
access-list vip-traffic remark allow any any traffic to the vip
access-list vip-traffic line 1 extended permit icmp any any
access-list vip-traffic line 2 extended permit ip any any
! Setup the server probes. PING for Server up/down
!TCP 80 for application up/down
probe icmp PING
description simple ping monitor
passdetect interval 60 probe tcp TCP80
passdetect interval 10
passdetect count 2
!Define the probes that check the server. Use ICMP so that "show rserver" reflects actual OS status.
rserver host 169.254.0.100
ip address 169.254.0.100
rserver host 169.254.0.200
ip address 169.254.0.200
!Create the server farm with the two servers. Use a TCP80 probe so that "show serverfarm TESTFARM"
! shows the application status (not the server status)
serverfarm host TESTFARM
!Create a sticky database - your mileage may vary according to application.
sticky ip-netmask 255.255.255.255 address source stuck
!Create the class maps for service policy
!Allow the interface to reachable by ICMP !Allow the VIP to be reachable over HTTP on TCP80
class-map type management match-any REMOTE_ACCESS
description Remote access traffic match.
3 match protocol icmp any class-map match-all VIP_CLASS
2 match virtual-address 169.254.1.1 tcp eq www
class-map match-all VIP_CLASS_2
2 match virtual-address 169.254.1.2 tcp eq www
! Apply the policy map from the class-map
! policy-map type management first-match REMOTE_MGMT
!Define the load-balancing policy with sticky. Round robin default.
policy-map type loadbalance first-match POLICY_MAP
! The VIP Load balancing policy, with NAT pool defined.
policy-map multi-match VIP_FARM
loadbalance vip inservice
loadbalance policy POLICY_MAP
loadbalance vip icmp-reply active
nat dynamic 100 vlan 100
interface vlan 100
ip address 169.254.0.70 255.255.0.0
access-group input vip-traffic
access-group output vip-traffic
!Define NAT Pool for the VIP.
nat-pool 100 169.254.3.1 169.254.3.1 netmask 255.255.255.255 pat
!Apple the service policy for administration traffic and load balancing policy.
service-policy input REMOTE_MGMT
service-policy input VIP_FARM
You can see the state of the translation
switch/ACE-TEST# sh xlate
TCP PAT from vlan100:169.254.0.69/51548 to vlan100:169.254.3.2/1025
I think that many people do not take the time to consider the use of probes. In fact, clever use of probes can dramatically change the way that your LoadBal acts in your networks.
My general suggestions is to use two types of probes. An ICMP for verifying OS status, and an application probe such as performing an HTTP check on TCP80, or even a simple TCP Connection check on Port 80. To make this useful, attach the ICMP probe to the REAL Server, and the HTTP probe to the Server Farm. When you run the “show real servers” you will see that the servers are “up” and responding to ping, but when you perform a “show serverfarm” you are seeing the status of the web application on those servers.
That much more useful.
Think carefully about probe timers. For example, a 15 second delay in probe, with a three poll dead interval means that the detection can take between 45 and 59 seconds because the failure could occur immediately after the previous probe.
For the Cisco ACE it’s also important to consider TCP timeouts. The default timeout is 30 seconds (and the probe default is sixty seconds). When you change the timeouts to less than thirty seconds you should change the TCP timeout as well. Realistically, if a server cannot respond to a TCP SYN in less than 5 seconds then it’s got a serious problems.
A final consideration is the intervals for probes when the server is failed. It’s important to take a server out of service, but don’t try to bring it into service too quickly. Operating Systems and Applications can take minutes to start and stabilise. Therefore, configure your failure probes to have at least three successful probes about a minute apart to ensure that OS is ready for new connections. If you have high traffic volumes you need to go and research the “Slow Start” feature. This feature is configured using “passdetect” meaning detect the server passing the test to return to service.
Other Posts in A Series On The Same Topic
- Cisco ACE - Enterprise Load Balancing on a Stick using Source NAT - Part 3 (14th February 2011)
- Cisco ACE - Enterprise Load Balancing on a Stick using Source NAT - Part 2 (9th February 2011)
- Cisco ACE - Enterprise Load Balancing on a Stick using Source NAT - Part 1 (8th February 2011)