Diskless Debian

MAJOR CAVEAT

MAJOR CAVEAT ALERT: Lenny DOES NOT work diskless as shipped. The NFS in kernels 2.6.25 to 27 is fubar as a part of adding clusterf*** (sorry, NFSv4 idmap) functionality. Actual Debian BUG. Kernel git entry. If using Lenny, at present you need to use it with a 2.6.18 kernel from etch or update to the latest security updates which fix this bug as a regression. It works flawlessly in that case.

Introduction

First of all - why diskless?

Noise

While the disks are not as noisy as they used to be prior to the introduction of fluid bearings in the mass market around year 2000-2001, they still emit some noise. Even if the noise from the disks themselves is negligible it is nearly impossible to cool them down to correct operational temperature without a lot of noise from the fans.

Heat

Without correct cooling the temperature of a modern disk will quickly exceed 50C. This will drop the MTBF of a typical disk to under 2 years (or even less). While disks themselves are no longer a major contributor to the thermals of most systems they are a contributor none the less. Removing them allows to pack the system in a smaller case and use less cooling which is important if you want to stick it behind a TV or have more desk real estate.

Cost

It is possible to take care of all noise issues and thermal problems by using silent enclosures and fans. The cost per PC to drop the noise under 24db and disk temperature under 30C in a residential environment is at least 25-30£, usually more.

Latency

While disks offer superior transfer rates compared to NFS, their seek latency is considerably higher compared to retrieving data from a server cache. It is high enough to cause serious problems in some applications like for example a QoS firewall or a media PC.

Flash Specifics

Cheap Flash tends to extremely bad on write speed and latency. I have a couple of 2-4 channel FDM designed as IDE replacements for industrial Embedded systems and they are horrible. Its cheapness is still relative. It is actually fairly expensive. Some of the end-of-2009 offerings like Kingston V series are starting to get "where I like them", but they are still nowhere near the cost of diskless.

In the past I used to build custom flash systems using a combination of CRAMFS and multi-layer config stored in gzipped CPIO format. Unfortunately, the limitations of CRAMFS and the lack of support and interest in it in the mainstream linux kernel make the approach more or less useless now. One might as well bight the bullet and go run-from-RAM all-CPIO with a "date-based" overlay to maintain config.

Summary

All in all, diskless still makes sense for a lot of applications. This document has two sections. The first one deals with "conventional" diskless installs which run Debian off NFS booting via PXE or from dedicated boot media. The second one deals with how to cut down this system even further so it fits into flash.

Debian Over NFS

Creating a Debian install to run over NFS is trivial:

• It is necessary to have a suitable system for netbooting. It is not necessary for it to be a Debian system. In fact any Unix can be used for netboot provided that it has working NFS, as well as dhcp and a tftpd daemon which support the necessary options and extensions.
• The next step is to install a minimal Debian installation onto a disk on a test client. The minimal Debian install is slightly under 0.5G for Debian sarge so nearly any old disk and PC should be usable for this stage.
• Once the client is installed and running it is necessary to apt-get install initrd-netboot on the test client. This package contains udhcpc and supporting scripts essential for booting over network.
• Once initrd-netboot has been installed the next step is to dpkg-reconfigure the kernel image on the test client. This will rebuild the initrd and include udhcpc in it. The kernel and initrd are now ready for use to boot a root over NFS image. They can be either booted over tftp using PXE or used from a dedicated boot media (small disk that is later put to sleep, flash, floppy, CD).
• If PXE will be used for booting the next step will be to set up dhcp on the netboot server. If the only boot environment will be pxelinux and the dhcp server can double up as a tftp server the only thing necessary is to add filename "pxelinux.0" to the relevant host or subnet declarations in /etc/dhcp3/dhcpd.conf. If the dhcp server does not double up as a tftp server, the setup is considerably more difficult because it has to supply a next-server dhcp option which some PXE clients have some problem with. See the dhcpd and pxelinux documentation for more details.
• The easiest way to provide unique NFS roots per client is by setting up static host declarations in /etc/dhcp3/dhcpd.conf for every diskless host. The declarations must contain a root-path which will be used by udhcpc to pivot root from the initial initrd. An important caveat is that there is no name resolution at this stage so the root path cannot contain a server name. IP addresses are still allowed. For example:

host vorlon {         
   hardware ethernet 00:11:85:e1:fb:32; # HP Thin Client 5515
   filename "pxelinux.0";   # bootstrap - actual location on debian /var/lib/tftpboot/pxelinux.0
   fixed-address 192.168.3.18;         
   option root-path "/exports/boot/vorlon"; # no IP so DHCP server is assumed to provide NFS
} 

It is necessary to set up tftpd. PXE requires a tftpd server which supports some extensions to the protocol (not-found extension ). I use tftpd-hpa because when I tested it the first time, out of all tftpd(s) shipped with Debian, it was the only one that supported the behaviour required by pxelinux (ENOTFOUND error code on inexistant files instead of timeout). From there on I still use it because it is "the devil I know". Unfortunately it is quite broken for some configurations. This is actually a long standing kernel udp bug which nobody bothers to fix. It will affect only some multihomed systems. It is quite easy to detect this failure condition. If the clients fail to boot and the tftp packets observed via tcpdump have the wrong source address, the bug is definitely showing its ugly head. I have patches for some version so if you are running into the problem feel free to drop me a line. Alternatively, replace all occurrences of send with sendto in tftpd.c passing the correct destination address. I recently looked at the netkit tftpd source and it has the same problem as well.

It is also necessary to set up the host configuration space. pxelinux.0 once loaded will try accessing files called pxelinux.cfg/XX-XX-XX-XX-XX-XX, pxelinux.cfg/AABBCCDD where AABBCCDD is the IP address in Hex, followed by pxelinux.cfg/AABBCCD, pxelinux/AABBCC, ..., pxelinux.cfg/A, pxelinux/default. For example for the above host declaration 192.168.3.18 will correspond to C0A80312.

The files from the previous point are expected to contain a suitable pxelinux config. For example this is the config for the diskless HP thin client HP Thin Client used to type this document (it is taken from pxelinux.cfg/C0A80312). The locations of all files are relative to the tftpd working directory (defaults to /var/lib/tftpboot).

default linux 
label linux         
  kernel diskless/i386/vmlinuz-2.6.14-1-crusoe         
  append vga=normal initrd=diskless/i386/initrd.img-2.6.14-1-crusoe root=/dev/nfs ip=dhcp rw  --

The kernel and initrd in the above config file are the kernel image installed on the initial test system and the initrd created for it after initrd-netboot has been installed. It is a good test to test these at this stage without changing anything on the disk itself. The system should boot using this kernel and image and proceed to mount the root filesystem off the disk.

Once everything is ready and the system boots via PXE, the small debian installation can be copied onto the target NFS filesystem using cp -rax. udhcpc will correctly set up the root path and mount it so fstab needs to have only the remaining filesystems.

# /etc/fstab: static file system information. 
# <file system> <mount point>   <type>  <options>       <dump>  <pass> 
proc            /proc           proc    defaults        0       0 
none            /tmp           tmpfs    defaults        0       0 
none            /var/tmp       tmpfs    defaults        0       0 

The kernel image and the initrd have to be copied to their target locations under /var/lib/tftpboot.

After that the client can be rebooted. If everything works according to plan it will load pxelinux.0 via tftp, load the kernel and the initrd and after that mount the NFS filesystem from the server.

Once there is a working minimal image it is a good idea to tar.gz it and save it for future use. From there on it is trivial to add extra clients by untarring the image and adding configs to pxelinux.cfg and /etc/dhcp3/dhcpd.conf.

This method has one major disadvantage. It is not very efficient on disk space and causes excessive disk-cache exhaustion on the server. Still, it is quite usable for a setup with several diskless clients and a single server. The cost savings in the client are enough to pay for the extra disk space, RAID and proper cooling in the server. They are also quite easy to maintain, because many operations can be done by simply chroot-ing into the correct directory on the server or editing the files on it.

Hopefully, one day extXfs will grow up enough to have deduplication and then it will become the perfect way of running it. In the meantime if cache pollution and disk wastage becomes an issue one can dedupe "manually" by replacing identical files with hard-links.

Application Specific Settings, Caveats and Bugs

SSH

If you run multiple machines with a common $HOME exported over NFS you are bound at some point to screw your .Xauthority file. While I never had this happening before upgrading to Lenny with Lenny it became a common occurence. The tell tale symptom are messages like "KDE Cannot Launch" and inability to start new programs.

The fix is to make sure that every host on the network uses its hostname when writing to .Xauthority by putting the following into every /etc/ssh/sshd_config

X11UseLocalhost no

This is especially valid from Lenny onwards which split the entry for localhost from the entry for the hostname in /etc/hosts. Bug