Fully Automatic Installation of Debian Etch with Root on LVM using FAI

At work we have been making extensive use of FAI to rapidly deploy Debian hosts. It allows us to take a virgin server and network install a fully functioning Debian system in under four minutes (less than three minutes if you discount the time our hosts spend performing their BIOS tests). It also provides a extensively customisable disaster recovery system into which you can netboot any problematic hosts to investigate and repair them. I thoroughly recommend it to anyone that manages more than a few Debian hosts and that is willing to spend the couple of days or so that it takes to configure and explore all of the options.

Unfortunately for me the current version does not natively support the Linux Logical Volume Management. There are at least two community contributed helper scripts that provide this functionality by means of well-placed hooks provided by FAI, but neither of these have support for root on LVM; in fact one actively discourages it by displaying a warning! Anyhow, placing the root filesystem on LVM is enormously beneficial since amongst other things it allows you to make a consistent point-in-time snapshot of the entire filesystem with no disruption to running services.

In the past, placing root on LVM has indeed been problematic adding extra complexity to the boot process. It requires the bootloader to pass an initial ramdisk to the kernel which contains the necessary modules, libraries, binaries and scripts to discover and enable the Logical Volume that contains the root filesystem, without which the system of no use at all. It must deal with the activation of inactive volumes and handle special volumes such as those under snapshot, mirroring, or even cryptographically secured.

Historically, support for this has been poor due to the old-style initial ramdisks built by Debian Sarge and prior being incapable of dealing with a variety of abnormal conditions, causing the startup scripts within the ramdisk to quietly and uninterruptedly hang with no simple means by which to inspect the state of the locked up system to determine the cause of failure. This periodically rendered unbootable hosts that were not cleanly shut down thus requiring the aid of a live CD distro or netboot disaster recovery image in order to fix the problems. (Usually it would be that the snapshots had failed having ran out of room, requiring that they be first removed in order for the live volume to be made accessible.)

Thankfully this has now all changed because of a vast improvement called initramfs - the new system for creating initial ramdisks in Etch. initramfs has bundled the BusyBox Bash shell clone which is launched if there is any problem in mounting the root filesystem thereby allowing you to repair it without the need for an external recovery system. Together with the big improvements that have made the startup scripts much more robust, this now mostly solves any problems with putting root on LVM. Accordingly, you can take reassurance from the fact that the new Debian Installer provides root on LVM functionality as an installation option for any new Debian system, making it no longer an exceptional setup but a well-trodden path.

Despite not supporting LVM, the FAI system can be easily modified by writing custom hook scripts that are run at designated points during the host installation process. Peter Gervai has created a script to provide support for software RAID and LVM by extending a previous script by Michal Svamberg that provides support for software RAID. The RAID and LVM hook enables the FAI installer to create the necessary LVM entities according to user provided configuration files and it is well documented in on its wiki page.

However there are some steps that must be taken in addition to the setup instructions documented in the wiki in order to successfully FAI build LVM-enabled hosts.

Firstly, ensure that the netboot installer itself is LVM capable by loading device-mapper kernel module before the partitioning task starts:

Simply append dm_mod to the kernelmodules variable in the config file class/20-hwdetect.source in the NFS-exported configuration directory.

Secondly, ensure that the fully installed target system is independantly bootable by recreating the initial ramdisk to have LVM capability and by fixing an erroneous line that finds its way into /etc/fstab due to a minor bug in the FAI partitioning scripts.

Create a new config file called package_config/SW_RAID in the NFS-exported config directory:

lvm-common
lvm2
mdadm

Also create a new config file called scripts/SW_RAID/10-misc:

#! /bin/bash
error=0 ; trap "error=$((error|1))" ERR
cat <<EOF >> $target/etc/initramfs-tools/modules
dm_mod
dm_mirror
dm_snapshot
EOF
$ROOTCMD update-initramfs -u
# Fix the incorrect /dev/ entry that SW_RAID helper makes in fstab
sed -ie '/^\/dev\/ /d' /tmp/target/etc/fstab
exit $error

Those basic changes should be all that is necessary to get any host that is part of the SW_RAID class to FAI install as a root on LVM system. Hopefully we can expect FAI to natively support LVM sometime soon, which will make the process even more painless!

Finally, here are some sample disk configuration files to give you some idea of how easy it should be to maintain.

Sample disk_config/FAIBASE file:

disk_config disk1
primary /boot 200 rw ; ext2 boot
primary swap 2000 rw
logical - 1024-


Sample class/SW_RAID.var file:

SW_RAID_CONFIG="
"

LVM_VG_CONFIG="
vg00[]=/dev/cciss/c0d0p5
"

LVM_LV_CONFIG="
vg00/root[-n root ]=2G:/:ext3:
vg00/home[-n home ]=5G:/home:ext3:
vg00/srv[-n srv ]=10G:/srv:ext3:
vg00/var[-n var ]=5G:/var:ext3:
vg00/varlog[-n varlog]=5G:/var/log:ext3:
"