Installing FreeBSD 9.1 using root on ZFS and GPT disks

This is partly based on the excellent guide at http://daemon-notes.com/articles/system/install-zfs/begin.

Boot with FreeBSD 9.1 installer and select Live CD when the installer starts.

Identify the disks you want to install to. SATA disks are named /dev/adaX and USB pendrives and SCSI drives are named /dev/daX.

dmesg|egrep (ada|da)

Of which output could look like this:

da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 scbus2 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)

Partitioning the drives using GPT

Create the partitions on the disk, making sure to properly align them using 4k blocks. The first partition containing the bootblock won’t need that.

gpart create -s gpt ada0
gpart add -b 34 -s 64k -t freebsd-boot ada0
gpart add -s 2G -a 4k -t freebsd-swap -l swap0 ada0
gpart add -a 4k -t freebsd-zfs -l disk0 ada0

Then do the same for the second disk.

gpart create -s gpt ada1
gpart add -b 34 -s 64k -t freebsd-boot ada1
gpart add -s 2G -a 4k -t freebsd-swap -l swap1 ada1
gpart add -a 4k -t freebsd-zfs -l disk1 ada1

Verify the disks are partitioned correctly.

gpart show

They should now look like this:

=>      34  16777149  da0  GPT  (8.0G)
        34       128    1  freebsd-boot  (64k)
       162         6       - free -  (3.0k)
       168   4194304    2  freebsd-swap  (2.0G)
   4194472  12582704    3  freebsd-zfs  (6G)
  16777176         7       - free -  (3.5k)

=>      34  16777149  da1  GPT  (8.0G)
        34       128    1  freebsd-boot  (64k)
       162         6       - free -  (3.0k)
       168   4194304    2  freebsd-swap  (2.0G)
   4194472  12582704    3  freebsd-zfs  (6G)
  16777176         7       - free -  (3.5k)

Add bootcode to the disks.

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

Creating the ZFS filesystems

Set up the disks as two-way mirror, essentially raid-1.

zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/disk0 /dev/gpt/disk1

The zpool then looks like this:

  pool: zroot
 state: ONLINE
  scan: none requested
config:

	NAME           STATE     READ WRITE CKSUM
	zroot          ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    gpt/disk0  ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0

errors: No known data errors

ZFS filesystems

zfs create -o compression=on -o setuid=off zroot/tmp
chmod 1777 /mnt/tmp
zfs create zroot/usr
zfs create zroot/usr/home
cd /mnt
ln -s usr/home home
cd -
zfs create zroot/usr/local
zfs create -o compression=on -o exec=off -o setuid=off zroot/usr/src
zfs create zroot/var
zfs create -o exec=off -o setuid=off zroot/var/backups
zfs create -o compression=on -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=on -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=on -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=on -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp

In my environment /usr/ports is mounted via NFS from a package building server. For a local /usr/ports the following need also to exist.

zfs create -o compression=on -o setuid=off zroot/usr/ports
zfs create -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o exec=off -o setuid=off zroot/usr/ports/packages

Then set it bootable and copy the important zpool.cache onto the filesystem, that it will find the zpool upon boot.

zpool set bootfs=zroot zroot
mkdir -p /mnt/boot/zfs
cp -p /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

Install FreeBSD

Extract the distfiles.

cd /mnt
for dist in base.txz kernel.txz lib32.txz; do
  tar -xzf /usr/freebsd-dist/$dist
done

Create a couple of necessary configuration files

cat << EOF > /mnt/boot/loader.conf
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
EOF
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf

Only the swap devices need to exist in the /etc/fstab. All other filesystems are mounted by zfs.

cat << EOF > /mnt/etc/fstab
# Device         Mountpoint  FStype  Options  Dump    Pass
/dev/gpt/swap0   none        swap    sw       0       0
/dev/gpt/swap1   none        swap    sw       0       0
EOF

zfs set readonly=on zroot/var/empty
shutdown -r now

You now have a mirrored ZFS-only FreeBSD system with GPT disks!