KeepalivedForNetworkReliability
Improving Network Reliability with Keepalived
by Philip Hollenback
originally published on LinuxDevCenter.
Redundancy is one of the key ways you can increase the reliability of your network. As the concept of RAID (redundant arrays of inexpensive disks) has shown, it can be much more cost effective to group a number of inexpensive components together than to spend much more money on one high-priced item. You can apply the same idea to your network: instead of investing in one very expensive proprietary router, why not install several redundant Linux routers made out of commodity parts and free software? This article shows how easy it is to do just that with Keepalived on Linux.
Concepts
The problem with routing is that most client computers do it in the simplest way possible: by using a default route. Any network traffic not destined for the local network goes happily onto the gateway router, assuming that it knows how to send it along appropriately.
This makes the gateway a single point of failure for your network. If it goes down, none of your client machines can communicate with the outside world.
One answer to this is to have your client machines run a routing protocol such as RIP or OSPF. Generally this isn't done, due to the increased complexity and overhead. Thus the only practical way to make routing more robust is to fool the clients into thinking they are always communicating with one gateway router. You can use VRRP (Virtual Router Redundancy Protocol) to do this; the Keepalived program provides one implementation of VRRP for Linux.
VRRP is an IETF protocol that allows two or more routers to act as one virtual router. According to the VRRP specification, the routers present a virtual IP address (VIP) that corresponds to a virtual MAC address (VMAC). Each router has a real hardware and IP address. Initially the master router handles the virtual IP and MAC addresses. If the master router fails, the backup then takes over the virtual addresses. The master and backup routers monitor each other with regular multicast advertisements, at a default rate of one per second.
Because Linux does not currently support VMACs, Keepalived implements only VIPs. In practice, this works fine on modern networks, although you should be aware that this can cause problems for older hardware that does not support gratuitous ARP requests.
In normal operation, the backup VRRP server monitors continually by listening for multicast advertisements from the master. If the master disappears, the backup sends a gratuitous ARP message out on the network, which says, in effect, "I own the hardware address that the master previously owned." This causes all other systems on the network to start using the backup VRRP server as their gateway. This continues until the master server reappears. The key point here is that no reconfiguration is necessary on the client machines; it all happens on the servers.
While this article covers the simple case of one master and one backup server, in reality there can be multiple backup servers for increased reliability. The VRRP protocol works on an election process: a failing master causes an election to happen, and the highest-priority backup takes over. If that backup fails, the next takes over, and so on.
Keepalived isn't the only VRRP implementation for Linux; for example, there is also a VRRPd project. However, it doesn't seem to be actively maintained; the last software release for that project was in 2002. There is also an IETF overview of the other VRRP implementations.
Installing the Software
Keepalived may come preinstalled on your Linux server, depending on which distribution you use. (It doesn't come with Fedora, at least up to FC3.) There have been many updates to Keepalived in the past few months, so you are probably best off downloading the latest version directly from the Keepalived web site.
Keepalived installs and builds in the standard Unix way: just unpack the tarball and follow the instructions in the install file. Make sure you put the Keepalived init file in your init directory; for example, /etc/rc.d/rc3.d/S99keepalived.
Remember that you also need to install Keepalived on both the master and backup routers. The installation is the same on all servers except for differences in the configuration file.
Configuration
The Keepalived configuration uses a single file, /etc/keepalived/keepalived.conf. This file can be intimidating, because there are many configuration options and the documentation is a bit scattered. Keepalived includes several other health-check mechanisms, and the documentation focuses on configurations such as web server farms. Luckily, you can ignore most of those configuration options if you are just configuring VRRP. One of my main goals in writing this article is to make people aware that Keepalived works perfectly fine as just a VRRP server.
Assume that your master router is at address 192.168.1.253 and your backup is at 192.168.1.254. Traditionally, the gateway on a network is on the .1 address, so set the VRRP virtual address to 192.168.1.1. That way if your existing client configurations use a default gateway on 192.168.1.1, you won't have to change the configuration on each client machine.
Edit keepalived.conf on the master so that it contains just the following:
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 1
priority 100
authentication {
auth_type PASS
auth_pass <password>
}
virtual_ipaddress {
192.168.1.1/24 brd 192.168.1.255 dev eth0
}
}
This first defines a new VRRP instance and calls it VI_1. If you want to run VRRP on multiple interfaces on a router, give each one a different instance name. Typical names are VI_1, VI_2, and so on, but you can name them anything you want.
The next line defines the state VRRP will be in when Keepalived starts. Because this is the master, VRRP should start in the master state so that it will control the virtual IP address.
The interface line defines which network interface this VRRP instance will operate on, so typically this is eth0 or eth1 or something similar. Keepalived works just fine with VLANs, so you can use VLAN addresses such as eth0.2 as well.
The virtual_router_id value matches up master and backup VRRP servers. All servers in a particular VRRP group (one master and one or more backup servers) should have the same vrrp_router_id.
VRRP uses an election mechanism to determine who is the master in a VRRP group, and the highest priority wins. The master should have a priority at least 50 higher than that of any of the backup servers, because the priority contributes to the failover time. In this example, if you set the master priority to 100, none of the backup servers should have a priority higher than 50. Remember, I said the state setting controls the state in which the VRRP server starts up. Immediately after startup, all VRRP routers in the same group (on the same network and with the same virtual_router_id) will hold an election. The server with the highest priority wins and becomes master, even if that machine has a lower priority than a system that started in the master state.
The VRRP specification describes several authentication mechanisms. Obviously there should be some way for the VRRP servers to communicate securely, because a rogue system could create a denial of service attack on your network by overriding your real VRRP master server. Keepalived supports both the password and IPSEC Authentication Header authentication methods, but password authentication is easier for normal use due to some implementation problems with IPSEC-AH authentication. Thus in my example I set the auth_type to password (plain password) and I specify a password on the auth_pass line. Note that this is a plain-text password that goes over the network very often (at least once per second at the standard advertisement interval), so this is really not a strong security measure.
Finally, the file sets the virtual router address in the virtual_ipaddress section. As I mentioned earlier, it probably makes sense to set your virtual IP address (VIP) to whatever your existing gateway was using, to minimize client configuration changes. You also have to specify the device this address is on. This should match the value of the interface setting above.
You can specify multiple addresses in the virtual_ipaddress section. This is useful if your VRRP server is on several different VLANs. In that case, each VIP goes on a separate line and the device entry corresponds to the VLAN of the virtual IP address.
That's an entire minimal master keepalived.conf file. See the keepalived.conf(5) man page for the other, optional settings.
The backup server keepalived.conf is almost identical. First, change the state setting to BACKUP, as you want this server to come up in the backup state. Then, change the priority to a lower number than that of the master. Remember that it should be at least 50 percent lower than the master priority, so 50 is a good choice in this case. Everything else in the configuration file (including, most importantly, the auth_pass) should have the same.
Time to Start Keepalived
Now that you have Keepalived configured on both the master and backup servers, start VRRP by running the Keepalived init script that came in the Keepalived source tarball (assuming you are on a Red Hat or Fedora system; adjust for other distros):
# /sbin/service keepalived start
Do this on both the master and slave servers. Then, check the syslog (/var/log/messages) on each machine. You should see messages indicating that Keepalived has started in mode MASTER on the master server and mode BACKUP on the slave. How can you tell if the master server is answering on the virtual IP address? The best way to check this is with the ip command. Run:
# ip addr show
on the master. Assuming that you're running Keepalived on eth0, you should see something like this:
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:e0:81:2b:aa:b5 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.253/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.1/24 brd 192.168.1.255 scope global secondary eth0
The output on the slave should be:
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:e0:81:2b:aa:c3 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth0
This shows that the master is answering both on its 192.168.1.253 address and the virtual IP of 192.168.1.1. This information is not available in the traditional ifconfig command output, so this is a good reason to bite the bullet and start using the ip command to view and or change your network settings if you haven't already been using it.
Testing
Testing Keepalived is straightforward: unplug the master from the network and see if the slave takes over. You can see the Keepalived state changes in syslog, however you won't see anything in the master syslog when you disconnect it from the network. The master actually does notice that the backup has disappeared and it transfers to a fault state--because the master also listens for multicast advertisements from the backup. In practice, the master doesn't do anything in the fault state except wait to hear from the backup.
The backup is more chatty. Its syslog will contain messages such as:
Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARP on eth0
This should be pretty easy to understand: the backup lost track of the master, so it decided to become master, take over the VIP, and send a gratuitous ARP to notify the clients. With the sample configuration, this will take 3.6 seconds at most.
Once this transition occurs, the backup is now in the MASTER state and controls the virtual IP address. This continues until the master comes back (well, or until the backup server goes down). Verify that the backup is in charge of the VIP by running ip addr show on the backup and verifying that the VIP is there (as in the previous section).
When the master comes back, you will see this in the backup server syslog:
Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs
Again, this is pretty easy to follow. The backup heard from another, more important VRRP server (the master, because that's the only other one in the example), so it went to the BACKUP state and deleted the virtual IP addresses.
At the same time, when the master comes back on to the network, it notices the backup and forces the backup to give up the VIP:
Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARP on eth0
At this point, the master is back in charge. Now you know that your Keepalived setup is working.
Failover Time-outs
Why is the maximum failover time in the example 3.6 seconds? This
comes from the advertisement interval and the skew time. The default
advertisement interval is 1 second (configurable in
keepalived.conf). The skew time helps to keep everyone from trying
to transition at once. It is a number between 0 and 1, based on the
formula (256 - priority) / 256
As defined in the RFC, the backup must receive an advertisement from
the master every (3 * advert_int) + skew_time
seconds. If it doesn't hear anything from the master, it takes over. With a backup router priority of 100 (as in the example), the failover will happen at most 3.6 seconds after the master goes down.
Closing Thoughts
Keepalived provides a rich set of tools for server monitoring. For our purposes of increasing router redundancy, the most interesting one is VRRP. Take a couple of Linux routers, add Keepalived with VRRP, and you have a much more redundant configuration.
Of course, it is important to note that this is not a complete solution. Consider the standard office setup of one T1 connected to one router. Even if you set a a backup router, you don't have full protection: if the one router with the T1 goes down, your clients will lose all their connectivity. Any complete redundancy solution must also consider external network links, not just internal VRRP routers. The IBM Redpaper on VRRP has some good information on designing a network with robust upstream routing.
In the past, some people have hesitated to consider using Keepalived for just a VRRP setup, as they perceive Keepalived as a large and complex system. I can assure you, based on my experience, that this isn't the case. If you are running a pool of systems such as web servers, you should check out the other features Keepalived has to offer. However, if you just want to add router redundancy to your Linux network, VRRP via Keepalived is just the ticket.
Special thanks to Keepalived developer Alexandre Cassen for reviewing this article and providing valuable feedback and corrections.
Phil Hollenback is a system administrator at a financial firm in Manhattan. When he's not upgrading Linux servers or skateboarding, Phil spends his time updating his web site, www.hollenback.net.
