High Availability Apache on Ubuntu 8.04
It's nice when your website keeps serving pages even after something catastrophic happens. Running two Apache nodes with Heartbeat gets you there -- if one server blows up, the other takes over in short order.
## Prelude
You'll need two boxes and *three* IP addresses. I'm using [virtual machines from Xeriom Networks](http://xeriom.net/). Both have been [firewalled](http://barkingiguana.com/2008/06/22/firewall-a-pristine-ubuntu-804-box), and I've opened the HTTP port to the world:
```bash
sudo iptables -I INPUT 3 -p tcp --dport http -j ACCEPT
sudo sh -c "iptables-save -c > /etc/iptables.rules"
```
For this post, let's assume the following IP addresses are available:
* 193.219.108.236 -- Node 1 (craig-02.vm.xeriom.net)
* 193.219.108.237 -- Node 2 (craig-03.vm.xeriom.net)
* 193.219.108.238 -- Not assigned (this becomes our floating IP)
## Simple service
First, install Apache on both boxes. Nothing fancy -- we just want to confirm we can serve *something* over HTTP.
Run this on both boxes:
```bash
sudo apt-get install apache2 --yes
```
Open a browser and hit the IP addresses for Node 1 and Node 2. You should see the default Apache page saying "It works!". If you don't, check your firewall allows `www` traffic. Your rules should look like this -- note the line ending `tcp dpt:www`:
```
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:www
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
```
## Adding resilience
Apache can serve pages from both machines now, which is great -- but it doesn't protect against one of them dying. For that, we use Heartbeat.
Install Heartbeat on both boxes:
```bash
sudo apt-get install heartbeat
```
Copy the sample configuration files to Heartbeat's config directory:
```bash
sudo cp /usr/share/doc/heartbeat/authkeys /etc/ha.d/
sudo sh -c "zcat /usr/share/doc/heartbeat/ha.cf.gz > /etc/ha.d/ha.cf"
sudo sh -c "zcat /usr/share/doc/heartbeat/haresources.gz > /etc/ha.d/haresources"
```
Lock down `authkeys` -- it's going to contain a password:
```bash
sudo chmod go-wrx /etc/ha.d/authkeys
```
Edit `/etc/ha.d/authkeys` and add a password of your choice:
```
auth 2
2 sha1 your-password-here
```
Configure `ha.cf` for your network. The node names **must** match the output of `uname -n` on each box:
```
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
bcast eth0
udpport 694
auto_failback on
node craig-02.vm.xeriom.net
node craig-03.vm.xeriom.net
```
Now tell Heartbeat to manage Apache. Edit `haresources` on both machines -- the contents must be identical on both nodes, and the hostname should be the output of `uname -n` on Node 1:
```
craig-02.vm.xeriom.net 193.219.108.238 apache2
```
The IP address here is the unassigned one from the prelude -- it becomes the floating virtual IP.
Since we told Heartbeat to use UDP port 694, we need to open it in the firewall on both boxes:
```bash
sudo iptables -I INPUT 2 -p udp --dport 694 -j ACCEPT
```
Your iptables rules should now look like:
```
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere udp dpt:694
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:www
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
```
Create a file on each box so we can tell which server is responding:
```bash
# Node 1 (craig-02.vm.xeriom.net)
echo "craig-02.vm.xeriom.net" > /var/www/index.html
```
```bash
# Node 2 (craig-03.vm.xeriom.net)
echo "craig-03.vm.xeriom.net" > /var/www/index.html
```
Hit each node's IP address in your browser to confirm the right content is showing. If it works, it's time to flip the switch.
## Bringing it to life
Start Heartbeat on the master (Node 1) first, then the slave (Node 2):
```bash
sudo /etc/init.d/heartbeat start
```
This takes a while to start up. Run `tail -f /var/log/ha-log` on both boxes to watch progress. After a bit, you should see Node 1 report something like:
```
heartbeat[6792]: 2008/06/24_11:06:21 info: Initial resource acquisition complete (T_RESOURCES(us))
IPaddr[6867]: 2008/06/24_11:06:22 INFO: Running OK
heartbeat[6832]: 2008/06/24_11:06:22 info: Local Resource acquisition completed.
```
## Testing for a broken heart
Check `ifconfig eth0:0` on both boxes. You should see output like this:
```
# Node 1
sudo ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:16:3e:3c:70:25
inet addr:193.219.108.238 Bcast:193.219.108.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
```
```
# Node 2
sudo ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:16:3e:92:ad:78
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
```
Node 1 has claimed the virtual IP address. If Node 1 dies, Node 2 takes over. Simulate a failure by stopping Heartbeat on Node 1:
```bash
# Node 1
sudo /etc/init.d/heartbeat stop
```
Check `ifconfig` again -- the virtual IP should now be on Node 2. Bring Node 1 back up and it should reclaim the IP.
If this all worked, congratulations -- Heartbeat is running and your web tier will survive a node failure. Skip ahead to see it in the browser.
If you see messages about the message queue filling up, the two nodes can't talk to each other. Double-check that UDP port 694 is open on *both* boxes:
```
heartbeat[6148]: 2008/06/24_11:05:09 ERROR: Message hist queue is filling up (500 messages in queue)
```
Verify the firewall rules -- the important line ends with `udp dpt:694`:
```
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere udp dpt:694
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:www
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
```
## The proof is in the pudding
Open your browser and hit the virtual IP address (193.219.108.238 in this example). You should see Node 1's page.
Stop Heartbeat on Node 1 (or shut it down entirely) and refresh. You should now see Node 2.
Bring Node 1 back up and refresh once more. You're back on Node 1.
That's high availability in action. If one server goes down, your users never notice.
Questions or thoughts? Get in touch.