{"id":202,"date":"2013-03-23T19:13:46","date_gmt":"2013-03-23T18:13:46","guid":{"rendered":"\/wordpress\/?p=202"},"modified":"2013-10-11T20:39:18","modified_gmt":"2013-10-11T18:39:18","slug":"setting-up-hast-through-carp","status":"publish","type":"post","link":"\/wordpress\/freebsd\/setting-up-hast-through-carp\/","title":{"rendered":"Setting up HAST through CARP"},"content":{"rendered":"

In order to make a highly available (HA) server, FreeBSD can utilize techniques for a highly available storage (HAST) that syncs data between two hosts, essentially a raid1 cross-server. To make a virtual IP address that always follows the master node, common address redundancy protocol (CARP) is used. HAST can only be used for two hosts while CARP doesn’t have that restriction. For our purposes we thus set up two hosts with a virtual host that is always on the active one.<\/p>\n\n\n\n\n\n
Name<\/th>\nProd VLAN<\/th>\nHAST VLAN<\/th>\n
hast-a<\/td>\n192.168.0.197<\/td>\n192.168.100.197<\/td>\n
hast-b<\/td>\n192.168.0.198<\/td>\n192.168.100.198<\/td>\n
hast-v<\/td>\n192.168.0.199<\/td>\n <\/td>\n<\/table>\n

The carp interface hast-v<\/strong><\/code> must have its virtual IP-address on the same subnet as an existing physical one. Thus if a daemon serving requests shouldn’t respond on the physical interfaces, it must be set up to only listen for incoming requests on the carp interface. This is done either in its own configuration file or through IP filtering software. The hast vlan should also utilize jumbo frames.<\/p>\n

\/etc\/rc.conf on hast-a<\/h3>\n

<\/p>\n

hostname=\"hast-a.local\"\r\nkeymap=\"swedish.iso\"\r\nifconfig_em0=\"inet 192.168.0.197\/24\"\r\nifconfig_em1=\"inet 192.168.100.197\/24 mtu 9000\"\r\nsshd_enable=\"YES\"\r\nntpd_enable=\"YES\"\r\n<\/pre>\n
<\/code><\/p>\n
\/etc\/rc.conf on hast-b<\/h3>\n
<\/p>\n
hostname=\"hast-b.local\"\r\nkeymap=\"swedish.iso\"\r\nifconfig_em0=\"inet 192.168.0.198\/24\"\r\nifconfig_em1=\"inet 192.168.100.198\/24 mtu 9000\"\r\nsshd_enable=\"YES\"\r\nntpd_enable=\"YES\"\r\n<\/pre>\n
<\/code><\/p>\n
Each machine has been given two disks. The operating system has been installed on the first one (da0) and the second one will be used for HAST (da1).
\n<\/p>\n
root@hast-a:\/root # camcontrol devlist\r\n<VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (pass1,da0)\r\n<VMware Virtual disk 1.0>          at scbus2 target 1 lun 0 (pass2,da1)\r\n<\/pre>\n
<\/code><\/p>\n
HAST setup<\/h2>\n
\/etc\/hast.conf<\/h3>\n
This should look exactly the same on both servers. Refer to hast.conf(5)<\/a> for specifics.
\n<\/p>\n
checksum sha256\r\n\r\nresource hast0 {\r\n  on hast-a {\r\n    local \/dev\/da1\r\n    remote 192.168.100.198\r\n  }\r\n  on hast-b {\r\n    local \/dev\/da1\r\n    remote 192.168.100.197\r\n  }\r\n}\r\n<\/pre>\n
<\/code><\/p>\n
Set up the initial hast device<\/h3>\n
On both servers run the following:
\n<\/p>\n
hastctl create hast0\r\n\/etc\/rc.d\/hastd onestart\r\n<\/pre>\n
<\/code><\/p>\n
On hast-a<\/strong><\/code> make it the master-node by the following:
\n<\/p>\n
hastctl role primary hast0<\/pre>\n
<\/code><\/p>\n
On hast-b<\/strong><\/code> make it the slave-node by the following:
\n<\/p>\n
hastctl role secondary hast0<\/pre>\n
<\/code><\/p>\n
Then verify on both servers that the initial sync has completed successfully. The default compression method is to only compress blocks that contain all zeroes, which is what initially an untouched disk more or less only consist of so this should complete almost instantaneously regardless of volume size.
\n<\/p>\n
root@hast-a:\/ # hastctl status\r\nhast0:\r\n  role: primary\r\n  provname: hast0\r\n  localpath: \/dev\/da1\r\n  extentsize: 2097152 (2.0MB)\r\n  keepdirty: 64\r\n  remoteaddr: 192.168.100.198\r\n  replication: fullsync\r\n  status: complete<\/strong>\r\n  dirty: 0 (0B)\r\n  statistics:\r\n    reads: 23\r\n    writes: 0\r\n    deletes: 0\r\n    flushes: 0\r\n    activemap updates: 0\r\n<\/pre>\n
<\/code><\/p>\n
On hast-a<\/strong><\/code> there should now be a \/dev\/hast\/hast0<\/code> device that a filesystem can be created on top of.
\n<\/p>\n
newfs -U \/dev\/hast\/hast0\r\n<\/pre>\n
<\/code><\/p>\n
Now that the hast part is complete, add it to \/etc\/rc.conf<\/code> for automatic startup when booting up.
\n<\/p>\n
echo 'hastd_enable=\"YES\"' >> \/etc\/rc.conf\r\n<\/pre>\n
<\/code><\/p>\n
CARP setup<\/h2>\n
This is all done on hast-a<\/strong><\/code>.<\/p>\n
First load the carp kernel module
\n<\/p>\n
kldload \/boot\/kernel\/if_carp.ko\r\n<\/pre>\n
<\/code><\/p>\n
We want to down the carp interface when any other interface goes down, thereby failing over the carp interface.
\n<\/p>\n
sysctl net.inet.carp.preempt=1\r\n<\/pre>\n
<\/code><\/p>\n
Create the carp interface. The vhid must be unique within the broadcast domain. The vhid can be in range of 0-255. The password given with pass must match on the opposite nodes, or carp won’t communicate with them.<\/p>\n
ifconfig carp0 create\r\nifconfig carp0 vhid 1 pass hastpass 192.168.0.199\/24\r\n<\/code><\/pre>\n
As to making all of this persistant through reboots.<\/p>\n
echo 'if_carp_load=\"YES\"' >> \/boot\/loader.conf\r\necho 'net.inet.carp.preempt=1' >> \/etc\/sysctl.conf\r\ncat << EOF >> \/etc\/rc.conf\r\ncloned_interfaces=\"carp0\"\r\nifconfig_carp0=\"vhid 1 pass hastpass 192.168.0.199\/24\"\r\nEOF\r\n<\/code><\/pre>\n
The setup is basically the same on both servers. The only difference on hast-b<\/strong><\/code> is the part making it the slave-node initially.<\/p>\n
ifconfig_carp0=\"vhid 1 advskew 100<\/strong> pass hastpass 192.168.0.199\/24\"\r\n<\/code><\/pre>\n
Verify the setup<\/h2>\n
By checking the output from ifconfig you should see hast-a<\/strong><\/code> is the master and hast-b<\/code><\/strong> is the backup (slave).<\/p>\n
root@hast-a:\/ # ifconfig carp0\r\ncarp0: flags=49 metric 0 mtu 1500\r\n        inet 192.168.0.199 netmask 0xffffff00\r\n        nd6 options=29\r\n        carp: MASTER vhid 1 advbase 1 advskew 0\r\nroot@hast-b:\/ # ifconfig carp0\r\ncarp0: flags=49 metric 0 mtu 1500\r\n        inet 192.168.0.199 netmask 0xffffff00\r\n        nd6 options=29\r\n        carp: BACKUP vhid 1 advbase 1 advskew 100\r\n<\/code><\/pre>\n
By downing a physical interface, the carp0 interface should switch their roles within a default of three seconds and you should notice the following in \/var\/log\/messages<\/code> (merged from both servers) capturing a session of first taking the interface down then up again.<\/p>\n
Mar 23 19:07:31 hast-a kernel: carp0: link state changed to DOWN\r\nMar 23 19:07:33 hast-b kernel: carp0: link state changed to UP\r\nMar 23 19:08:11 hast-a kernel: carp0: INIT -> BACKUP\r\nMar 23 19:08:11 hast-a kernel: carp0: link state changed to DOWN\r\nMar 23 19:08:11 hast-a kernel: carp0: BACKUP -> MASTER (preempting a slower master)\r\nMar 23 19:08:11 hast-a kernel: carp0: link state changed to UP\r\nMar 23 19:08:11 hast-b kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)\r\nMar 23 19:08:11 hast-b kernel: carp0: link state changed to DOWN\r\n<\/code><\/pre>\n
\"vswitch_promiscious\"<\/a>Do note that the underlying nic whose IP-address is on the same subnet as the carp0 device must be allowed use of promiscious mode. If running a virtual machine this is not always so. On ESXi for instance, you may need to reconfigure the virtual distributed switch to accept promiscious mode (on the security tab). Ideally that should be configured as a port group allowed promiscious mode that’s only used by these servers, and another port group that’s not allowed promiscious mode (default) used by all the rest of the virtual hosts. You may also need to set Net.ReversePathFwdCheckPromisc<\/code> to 1<\/code> on the in the ESXi-host advanced settings that these VM’s will run on.<\/p>\n
Now that the carp part also is complete, all that’s left is to put an application on the servers that makes use of all this and tie carp and hast together. That’s the scope of another article.<\/p>\n","protected":false},"excerpt":{"rendered":"
In order to make a highly available (HA) server, FreeBSD can utilize techniques for a highly available storage (HAST) that syncs data between two hosts, essentially a raid1 cross-server. To make a virtual IP address that always follows the master node, common address redundancy protocol (CARP) is used. HAST can only be used for two …<\/span> Read More →<\/span><\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[14],"tags":[44,45,12,31,43,7,34,8],"_links":{"self":[{"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/posts\/202"}],"collection":[{"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/comments?post=202"}],"version-history":[{"count":19,"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/posts\/202\/revisions"}],"predecessor-version":[{"id":229,"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/posts\/202\/revisions\/229"}],"wp:attachment":[{"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/media?parent=202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/categories?post=202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp.strahlert.net\/wordpress\/wp-json\/wp\/v2\/tags?post=202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}