Setting up HAST through CARP

In order to make a highly available (HA) server, FreeBSD can utilize techniques for a highly available storage (HAST) that syncs data between two hosts, essentially a raid1 cross-server. To make a virtual IP address that always follows the master node, common address redundancy protocol (CARP) is used. HAST can only be used for two hosts while CARP doesn’t have that restriction. For our purposes we thus set up two hosts with a virtual host that is always on the active one.

The carp interface hast-v must have its virtual IP-address on the same subnet as an existing physical one. Thus if a daemon serving requests shouldn’t respond on the physical interfaces, it must be set up to only listen for incoming requests on the carp interface. This is done either in its own configuration file or through IP filtering software. The hast vlan should also utilize jumbo frames.

/etc/rc.conf on hast-a

hostname="hast-a.local"
keymap="swedish.iso"
ifconfig_em0="inet 192.168.0.197/24"
ifconfig_em1="inet 192.168.100.197/24 mtu 9000"
sshd_enable="YES"
ntpd_enable="YES"

/etc/rc.conf on hast-b

hostname="hast-b.local"
keymap="swedish.iso"
ifconfig_em0="inet 192.168.0.198/24"
ifconfig_em1="inet 192.168.100.198/24 mtu 9000"
sshd_enable="YES"
ntpd_enable="YES"

Each machine has been given two disks. The operating system has been installed on the first one (da0) and the second one will be used for HAST (da1).

root@hast-a:/root # camcontrol devlist
<VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (pass1,da0)
<VMware Virtual disk 1.0>          at scbus2 target 1 lun 0 (pass2,da1)

HAST setup

/etc/hast.conf

This should look exactly the same on both servers. Refer to hast.conf(5) for specifics.

checksum sha256

resource hast0 {
  on hast-a {
    local /dev/da1
    remote 192.168.100.198
  }
  on hast-b {
    local /dev/da1
    remote 192.168.100.197
  }
}

Set up the initial hast device

On both servers run the following:

hastctl create hast0
/etc/rc.d/hastd onestart

On hast-a make it the master-node by the following:

hastctl role primary hast0

On hast-b make it the slave-node by the following:

hastctl role secondary hast0

Then verify on both servers that the initial sync has completed successfully. The default compression method is to only compress blocks that contain all zeroes, which is what initially an untouched disk more or less only consist of so this should complete almost instantaneously regardless of volume size.

root@hast-a:/ # hastctl status
hast0:
  role: primary
  provname: hast0
  localpath: /dev/da1
  extentsize: 2097152 (2.0MB)
  keepdirty: 64
  remoteaddr: 192.168.100.198
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  statistics:
    reads: 23
    writes: 0
    deletes: 0
    flushes: 0
    activemap updates: 0

On hast-a there should now be a /dev/hast/hast0 device that a filesystem can be created on top of.

newfs -U /dev/hast/hast0

Now that the hast part is complete, add it to /etc/rc.conf for automatic startup when booting up.

echo 'hastd_enable="YES"' >> /etc/rc.conf

CARP setup

This is all done on hast-a.

First load the carp kernel module

kldload /boot/kernel/if_carp.ko

We want to down the carp interface when any other interface goes down, thereby failing over the carp interface.

sysctl net.inet.carp.preempt=1

Create the carp interface. The vhid must be unique within the broadcast domain. The vhid can be in range of 0-255. The password given with pass must match on the opposite nodes, or carp won’t communicate with them.

ifconfig carp0 create
ifconfig carp0 vhid 1 pass hastpass 192.168.0.199/24

As to making all of this persistant through reboots.

echo 'if_carp_load="YES"' >> /boot/loader.conf
echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf
cat << EOF >> /etc/rc.conf
cloned_interfaces="carp0"
ifconfig_carp0="vhid 1 pass hastpass 192.168.0.199/24"
EOF

The setup is basically the same on both servers. The only difference on hast-b is the part making it the slave-node initially.

ifconfig_carp0="vhid 1 advskew 100 pass hastpass 192.168.0.199/24"

Verify the setup

By checking the output from ifconfig you should see hast-a is the master and hast-b is the backup (slave).

root@hast-a:/ # ifconfig carp0
carp0: flags=49 metric 0 mtu 1500
        inet 192.168.0.199 netmask 0xffffff00
        nd6 options=29
        carp: MASTER vhid 1 advbase 1 advskew 0
root@hast-b:/ # ifconfig carp0
carp0: flags=49 metric 0 mtu 1500
        inet 192.168.0.199 netmask 0xffffff00
        nd6 options=29
        carp: BACKUP vhid 1 advbase 1 advskew 100

By downing a physical interface, the carp0 interface should switch their roles within a default of three seconds and you should notice the following in /var/log/messages (merged from both servers) capturing a session of first taking the interface down then up again.

Mar 23 19:07:31 hast-a kernel: carp0: link state changed to DOWN
Mar 23 19:07:33 hast-b kernel: carp0: link state changed to UP
Mar 23 19:08:11 hast-a kernel: carp0: INIT -> BACKUP
Mar 23 19:08:11 hast-a kernel: carp0: link state changed to DOWN
Mar 23 19:08:11 hast-a kernel: carp0: BACKUP -> MASTER (preempting a slower master)
Mar 23 19:08:11 hast-a kernel: carp0: link state changed to UP
Mar 23 19:08:11 hast-b kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 23 19:08:11 hast-b kernel: carp0: link state changed to DOWN

Do note that the underlying nic whose IP-address is on the same subnet as the carp0 device must be allowed use of promiscious mode. If running a virtual machine this is not always so. On ESXi for instance, you may need to reconfigure the virtual distributed switch to accept promiscious mode (on the security tab). Ideally that should be configured as a port group allowed promiscious mode that’s only used by these servers, and another port group that’s not allowed promiscious mode (default) used by all the rest of the virtual hosts. You may also need to set Net.ReversePathFwdCheckPromisc to 1 on the in the ESXi-host advanced settings that these VM’s will run on.

Now that the carp part also is complete, all that’s left is to put an application on the servers that makes use of all this and tie carp and hast together. That’s the scope of another article.