Setting up a highly available (HA) DNS in FreeBSD 10

In Setting up HAST through CARP preparations were done for making the same storage volume be able to be accessed through a redundant IP-address. Now comes the time to tie this together with an actual application and make the failover automatic.

To the right is a graphical representation of all components tied together. In the middle there will be a DNS server that is either active on hast-a or hast-b, always through the common name hast-v. This can be applied to any simple network daemon that needs to be highly available.

In the previous post hast was setup but not with automatic failover. We’ll use the device state change daemon devd(8) to catch when the carp interface changes state. As CARP has changed in FreeBSD 10, so has the necessary bits in devd. Add the following to /etc/devd.conf

# The below is valid configuration for FreeBSD 9 and earlier.
# notify 0 {
#        match "system"          "IFNET";
#        match "type"            "LINK_*";
#        match "subsystem"       "carp*";
#        action "/root/carpcontrol.sh $type $subsystem";
#};

# The below is valid configuration for FreeBSD 10 and later.
notify 0 {
        match "system"          "CARP";
        match "subsystem"       "[0-9]+@[0-9a-z]+";
        match "type"            "(INIT|MASTER|BACKUP)";
        action "/root/carpcontrol.sh $type $subsystem";
};

CARP has changed quite a lot in FreeBSD 10. This is how it’s setup nowadays. (For FreeBSD 9 and earlier, refer to the previous post linked above.)

/etc/rc.conf on hast-a

ifconfig_vmx0="inet 192.168.0.197/24"
ifconfig_vmx0_alias0="inet 192.168.0.199/24 vhid 199 pass hastpass"
ifconfig_vmx1="inet 192.168.100.197/24"
ifconfig_vmx1_alias0="inet 192.168.100.199/24 vhid 199 pass hastpass"
ifconfig_vmx2="inet 192.168.50.1/30"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
pfsync_enable="YES"
pfsync_syncdev="vmx2"

/etc/rc.conf on hast-b

ifconfig_vmx0="inet 192.168.0.198/24"
ifconfig_vmx0_alias0="inet 192.168.0.199/24 vhid 199 pass hastpass advskew 100"
ifconfig_vmx1="inet 192.168.100.198/24"
ifconfig_vmx1_alias0="inet 192.168.100.199/24 vhid 199 pass hastpass advskew 100"
ifconfig_vmx2="inet 192.168.50.2/30"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
pfsync_enable="YES"
pfsync_syncdev="vmx2"

Minimal /etc/pf.conf

prod_if = "vmx0"
hast_if = "vmx1"
pfsync_if = "vmx2"
prod_carp_ip = "192.168.0.199"

set block-policy drop
set skip on lo0
set skip on pfsync0

scrub in

pass quick on $pfsync_if proto pfsync keep state (no-sync)
pass quick proto carp keep state (no-sync)

antispoof for $prod_if
antispoof for $hast_if

pass in quick proto { tcp, udp } from any to $prod_carp_ip port = 53
block in quick proto { tcp, udp } from any to $prod_if port = 53

pass out quick on $prod_if from any to any keep state
pass out quick on $hast_if from any to any keep state

/root/carpcontrol.sh

This is where the magic happens.

#!/bin/sh
#
# magnus.strahlert @ 140714
#   script to handle hast failover

_type=$1
_subsyst=$2

bn=${0##*/}
logger=/usr/bin/logger
hastctl=/sbin/hastctl
jot=/usr/bin/jot
fsck=/sbin/fsck

# Configurable variables below
hast_dev=/dev/hast/hast0
hast_if=vmx1
mp=/usr/local/hast

# What daemons to stop/start when failover takes place.
daemons="named"

logit() {
  ${logger} -p "$1" -t ${bn} "$2"
}

if [ "`echo ${_subsyst}|cut -d '@' -f2`" != "${hast_if}" ]; then
  logit kern.warning "Ignoring change to ${_type} on non-hast if ${_subsyst}"
  exit
fi

case ${_type} in
  "MASTER")
    logit kern.warning "Caught ${_type} from ${_subsyst}"

    ${hastctl} role primary ${hast_dev##*/}
    if [ $? -ne 0 ]; then
      logit kern.crit "Unable to change role to primary"
      exit 1
    fi

    for i in `${jot} 30`; do
      [ -c ${hast_dev} ] && break
      sleep 1
    done

    if [ ! -c ${hast_dev} ]; then
      logit kern.crit "Timeout waiting for device ${hast_dev}"
      ${hastctl} role secondary ${hast_dev##*/}
      exit 1
    fi

    logit kern.warning "HAST role changed to primary"

    logit kern.warning "Running fsck on ${hast_dev}"
    ${fsck} -y -t ufs ${hast_dev}

    if [ ! -d ${mp} ]; then
      mkdir -p ${mp}
    fi

    logit kern.warning "Mounting ${hast_dev}"
    mount -t ufs ${hast_dev} ${mp}

    for d in ${daemons}; do
      logit kern.warning "Starting ${d} in master-mode"
      service $d onestart
    done
  ;;

  "BACKUP")
    logit kern.warning "Caught ${_type} from ${_subsyst}"

    for d in ${daemons}; do
      service $d onestop
    done

    logit kern.warning "Unmounting ${hast_dev}"
    umount -f ${mp}

    ${hastctl} role secondary ${hast_dev##*/}

    if [ $? -ne 0 ]; then
      logit kern.crit "Unable to change role to secondary"
      exit 1
    fi

    logit kern.warning "HAST role changed to secondary"
  ;;
esac

All files mentioned above need of course to exist on both servers.

Daemons specified in the above script to be part of failover need to have their configuration saved on the hast volume. That might involve specifying specific rc-variables in /etc/rc.conf to point where the configuration files lies.

/etc/rc.conf

named_conf="/usr/local/hast/etc/namedb/named.conf"