Setting up a PXE bootserver

Notes on installing a PXE bootserver that boots diskless terminalservers by use of nfsroot. Three machines have been set up for this:

bootserver

A typical install of FreeBSD have been used. Only including base, generic kernel and lib32. After it’s installed, freebsd-update have been applied to fetch the latest patches. That leaves us with a base from which to build the terminalservers.

By creating the dump on a different filesystem, it won’t be included (in the dump). If another filesystem cannot be mounted by nfs or usb or whatnot, the nodump flag can be set on the resulting dump by use of for instance chflags nodump /tmp/bootserver.root

Create nfsroot filesystem

This entire process can be copied to create additional terminalservers.

dump -0Laf /mnt/bootserver.root /
mkdir /ts-rootfs/TS1
cd /ts-rootfs/TS1
restore -xvf /mnt/bootserver.root

/var needs to be mounted on a virtual ramdisk device so we don’t need the contents of /var on the terminalserver. /var itself still must exist in order to be able to mount a filesystem on top of it.

rm -rf /ts-rootfs/TS1/var/*

Back to the bootserver and configure necessary daemons.

Set up NFS server

/etc/exports

/ts-rootfs/TS1 -maproot=root -alldirs -network=192.168.65.11/32
/ts-rootfs/TS2 -maproot=root -alldirs -network=192.168.65.12/32

/etc/rc.conf

rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"
mountd_enable="YES"

Set up TFTP server

mkdir /tftpboot

/etc/rc.conf

I have also set up a NIC for management on another network so this is in order for inetd to only listen on the pxeboot lan.

inetd_enable="YES"
inetd_flags="-a 192.168.65.1"

/etc/inetd.conf

Uncomment the tftp line for ipv4

tftp   dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /tftpboot

Then create a link to the bootloader

cd /tftpboot
ln /boot/pxeboot

Set up DHCP server

This is a port that needs to be installed. I simply fetched it from my package building server.

pkg_add http://esxi-v4/All/isc-dhcp42-server-4.2.4_2.tbz

/usr/local/etc/dhcpd.conf

default-lease-time 3600;
max-lease-time 86400;
ddns-update-style none;

subnet 192.168.65.0 netmask 255.255.255.0 {
option domain-name "ts.local";

next-server 192.168.65.1;

filename "pxeboot";

deny unknown-clients;
}

host ts01 {
hardware ethernet 00:0c:29:eb:52:78;
fixed-address 192.168.65.11;
option root-path "192.168.65.1:/ts-rootfs/TS1/";
}

host ts02 {
  hardware ethernet 00:0c:29:2d:6b:e3;
  fixed-address 192.168.65.12;
  option root-path "192.168.65.1:/ts-rootfs/TS2/";
}

Set up NTP server

In distributed computing it’s important that the time is the same on all involved servers. Thus bootserver will serve time to the terminalservers. If it’s also connected to the Internet, it can ask other ntp servers for time. If not, it doesn’t matter if the time is correct. What matters is that it is the same on the terminalservers.

/etc/rc.conf

ntpd_enable="YES"

Now the terminalservers are ready for booting but a couple of preparations need to be done on them before that. These preparations need to be done on all terminalservers that have been created.

Prepare /ts-rootfs/TS1

chroot /ts-rootfs/TS1

/etc/rc.conf

hostname="ts01.ts.local"
defaultroute_delay=0     # (The terminalservers are setup without a defaultroute)
sshd_enable="YES"
ntpd_enable="YES"

/etc/fstab

192.168.65.1:/ts-rootfs/TS1 / nfs rw 0 0

The /var filesystem needs to be put on a virtual ramdrive device by the following. If you find your terminalservers /var filesystem growing you need to increase its size. I’ve chosen 64Mb which is plenty thus far.

mkdir -p /conf/base/var
echo "64m" > /conf/base/var/md_size

/etc/rc.shutdown.local

(run this to create the initial snapshot)
All contents of /var will be lost when the terminalserver reboots. To remedy this I’ve added /bin/sh /etc/rc.shutdown.local to root’s crontab on a daily schedule.

mv -f /conf/base/var.cpio.gz /conf/base/var.cpio.gz.old
tar -cf /conf/base/var.cpio.gz -C / --format cpio --gzip var

/etc/periodic.conf

daily_output="/var/log/daily.log"
weekly_output="/var/log/weekly.log"
monthly_output="/var/log/monthly.log"
daily_status_security_output="/var/log/dailysecurity.log"

/etc/ntp.conf

server 192.168.65.1

In order to access the terminal servers, ssh keys should be set up. It's also advisable to put all hostnames in /etc/hosts on bootserver and the terminalservers.

Setup SSH keys

On bootserver

ssh-keygen -t dsa
ssh 192.168.65.11 "mkdir .ssh; chmod 700 .ssh"
scp .ssh/id_dsa.pub 192.168.65.11:.ssh/authorized_keys
ssh 192.168.65.12 "mkdir .ssh; chmod 700 .ssh"
scp .ssh/id_dsa.pub 192.168.65.12:.ssh/authorized_keys

/etc/hosts looks like this that need to be synced to all servers:

192.168.65.1     bootserver bootserver.ts.local
192.168.65.11    ts01 ts01.ts.local
192.165.65.12    ts02 ts02.ts.local

Now you have a terminalserver farm that can easily be expanded with endless possobilities of distributed computing! 🙂