Using NFS on Linux - how to do it efficiently on Linux.

This writeup assumes Debian. Other Linux distros do not differ significantly. BSD and Ssolaris are quite different as they have different autofs syntax and semantics. They however still follow the same principles.

Introduction

For the time being using NFSv4 on Debian is not safe in an environment set-up "by the book". In this write-up up to the end of the NFSv3 section everything is safe, tested and run in production for years. There seems to be a major bug with using v4 in an autofs environment: Debian Bug ID 576405. So the v4 section can be used only "at your own risk" (following this bug may be a good idea).

NFSv3

The important thing to remember when dealing with NFSv3 is that it was built to the Sun perception of the world.

From a Sun perspective the world is flat (no classless interdomain routing) and sits on top of four turtles whose names are NIS, Autofs, NFS and rpc.bootparams.

While Linux makes this more palatable by replacing rpc.bootparams with DHCP, the picture is still the same - If you yank one of the turtles out your world starts to wobble.

• If you yank NIS, you have to go to every machine to change elementary things. LDAP does not provide some of the functionality in NIS like netgroups. So it is not a solution for all NIS problems and it brings enough problems of its own. In fact in most small networks NIS is a better option.
• If you yank autofs and use static mounts instead, moving any of the exported volumes becomes a major nightmare. While linux is supposed to support soft mounts the quality of the support leaves a lot to be desired. Autofs makes this much more pallatable by keeping mounted only things that are accessed at any given time. For example if you distribute software or shared data via NFS if it is static mounted you have to declare an outage and update the NFS server or manually move each machine to the new version upon upgrade. With autofs you can install new version side by side with the old one into a different directory, change the map to point to the new version and the machines will pick it up as they start using it. Most importantly - as far as the clients are concerned not a single of the paths and locations will change. This is quite important when dealing with software that tends to write full "resolved" paths all over its config files. So in a network of 100 machines instead of dealing with a 100 you end up dealing only with the 3-4 holdouts that have failed to pick up the new export. Similarly, a rollback is a mere map change. All machines go back to using the old version with minimal fuss. End of the day - it takes one BOFH to manage a few hundreds of user workstations with autofs and NIS and a whole brigade of PFYs to do the same job without.
• If you yank NFS out of the picture and try to use another fs a whole lot of features stop working as expected. In fact even moving from v3 to v4 has its surprises. While NFS is one of the favourite pet hates and bugbears of all Unix sysadmins the reality is that it is implied in one place too many when trying to manage Unix clients in a network.
• If you do not have centralised boot management rebuilding machines becomes a major nightmare. Additionally, Linux does not do NIS+ and does not rely on hosts propagated via NIS so someone has to ensure that all forward and reverse DNS mappings are correct and persistent in the long term. DHCP is the best way to do this on Linux.

If you try to make this contraption hold a non-flat network with multiple subnets you lose features, have to do some really "strange things" and the entire system goes into primadonna mode. While Linux fixes some of that it cannot fully compensate for all of Sun's sillies that show up in v3. The way to make it work is to move to v4.

NFSv4 (when it works)

On the positive side NFSv4 acknowledges that the world is round. No more flat network assumptions. Multiple subnets are OK and it is even designed to work reasonably well across firewalls. Halleluyah, finally, about bloody time.

However, in order to compensate for all this "goodness" it also brings in the assumption that the filesystem layout must strictly follows Solaris rules.

Solaris file system layout traditionally assumes that anything a Solaris host has to offer to the network sits under /exports and is mounted to the correct locations (even on the local machine) using autofs. That is not the case in the average Linux or BSD network. Linux (and to a lesser extent BSD) nfs use is non-systematic and chaotic. Stuff is exported from all over the place and not from a single hierarchy. If you are trying to convert such a network to NFSv4 you are up to a very bad surprise - you may end up having to export your ROOT (yes, I am not joking - ROOT). That is of course untenable. So step number one in making a network being able to use NFSv4 efficiently is to bring any exports to the Solaris convention - they should sit under /exports.

Setting up a maintainable NFS v3 and v4 system in a multiuser Debian network.

Decide which exports (I hate using the Redmondian term shares) will be moved to NFSv3. There are a few bugs related to NFSv4 in Linux. You cannot chown a file to an unknown UID/GID on a NFSv4 mount. This situation is most commonly encountered when building/untarring things as root and they come from another system. For day to day use when any chown will be to a "known" network user NIS should take care of this problem. This bug however is one of the reasons why NIS is a must in an NFSv4 environment. Otherwise adding/deleting/changing users becomes a major nightmare. idmapd was supposed to solve it, but quite obviously for some use cases it fails to do so.

Set up NIS

• Install NIS apt-get install nis and set local domain on master
• Uncomment the relevant lines in /var/yp/Makefile on the NIS master to ensure that shadow and other important databases are built and propagated
• Set the /etc/default/nis settings to enable the ypserv on the master
• Build the databases by issuing make in /var/yp
• Add +::::: compat lines to the end of /etc/passwd, /etc/shadow, /etc/group. Make sure that the number of : is correct in each case
• Install NIS on clients and set local domain
• Promote any clients you need to secondaries by changing the settings in /etc/default/nis and issuing an /etc/init.d/nis restart. Use /usr/lib/yp/ypinit -s MASTERNAME to init the database as needed and update the ypservers list and the Makefile in /var/yp on the master to enable map push.

Set up autofs

• Install autofs on all machines. You may need to make corrections for a couple of debian bugs by adding an extra rc2.d entry to rerun autofs one more time after nis has started. I add an /etc/init.d/autofs restart to the end of the start function in /etc/init.d/nis. This solves the chicken and egg problem of who should start first autofs or nis.
• Create a map for HOME and LOCAL - for example /etc/auto.home and /etc/auto.local. Home is obvious. LOCAL is for shared data and non-packaged executables distributed via nfs. Good examples are java, bleeding edge openoffice, build tools and environments, etc. While smacking an autofs on top of /usr/local/ seems like a very tempting solution it is actually better to symlink only things that are needed instead of trying to maintain an autofs-ed /usr/local/.
• Add all entries for home directories to the /etc/auto.home map (this is still a v3 setup for now).

mediahd -rsize=4096,wsize=4096,rw pacem:/exports/home/mediahd
aivanov  rsize=4096,wsize=4096,rw eden:/exports/md3/home/aivanov

• Create a simple /etc/auto.master

/var/autofs/misc /etc/auto.misc
/home yp:auto.home
/var/autofs/local yp:auto.local

• Add auto.home, auto.local and auto.master to the /var/yp/Makefile (there are entries for some of them already in the distribution one). Rebuild NIS by issuing Make.
• Move home on your server with all subdirectories to be under /exports/. This is best done in single user mode.
• restart autofs. You should see nothing under home. if you access let's say /home/aivanov it will magically appear and be mounted. When NFS v3 is in use the mount on a local machine is bind-mounted so you can have thousands of these if you want to. At the same time on a machine in the network only home dirs for users which use it will be mounted. This way no machine will exceed the 256 max mounts limits and no machine will hold a mount unless it really needs it allowing for dirs to be moved and maintained.
• install and start autofs on clients. Replace the distribution /etc/master with a single line like +auto.master to pick up the NIS one

It should all work now and will probably be enough for most networks.

Set up NFS (including v4)

There is a number of false statements about NFSv4 setups around the network

• You will not need to specify paths any more. Sure you do. But they will be relative to the NFS export root (/exports on a Solaris like install)
• You need gssapi and kerberous. Sure you do not. You need them only if you want per-user and per-machine authentication. If you are moving up from NFSv3 you do not really need that.

In reality all you need to do is:

1. Reorganise your exports so all of them sit under an /export
2. Put them in the right place using autofs and/or symlinks where autofs is not enough

Once this has been done the v3 setup can be upgraded to v4 in two simple steps. However, as NFSv4 does not work properly on 2.6.26 at present it is better to stop here. From here onwards - this is the way it should work (and it kind'a works). However bugs prevent it from working properly.

• Add fsid=root in /etc/exports for the /exports entry. This enables v4 and makes all v4 exports relative to that point. Add "nohide" to any filesystems mounted under /exports to ensure that they are visible even when a single hierarchy is being mounted as is the case in v4. As v4 does not have the bind shortcut a 127.0.0.1 entry is also required. Alternatively the files for the map on the server exporting them can be local and use v3 (in reality bind) and v4 be used only for clients reading the map from NIS.

/exports            192.168.0.0/16(rw,async,no_root_squash,no_subtree_check,nohide,fsid=root) 127.0.0.1(rw,async,no_root_squash,no_subtree_check,nohide,fsid=root)
# separate fs - needs nohide
/exports/md3        192.168.0.0/16(rw,async,no_root_squash,no_subtree_check,nohide) 127.0.0.1(rw,async,no_root_squash,no_subtree_check,nohide)
# separate fs - needs nohide)
/exports/md4        192.168.0.0/16(rw,async,no_root_squash,no_subtree_check,nohide) 127.0.0.1(rw,async,no_root_squash,no_subtree_check,nohide)
# separate fs - needs nohide)

• Entries in auto.home and other autofs maps are changed to the v4 format. The paths become relative to whatever is declared as root in /etc/exports. Note - these are still full paths.

mediahd -fstype=nfs4 pacem:/home/mediahd
aivanov  -fstype=nfs4 eden:/md3/home/aivanov

• hit autofs on the head using killall -HUP automount followed by killall -USR1 automount. This forces it to reread the maps.

Done - you have now moved to an NFSv4 system. This is sufficient for most LANs. It is possible to go further from here and have encryption and authentication between machines using kerberos and gssapi. This however is probably unnecessary for most small LANs which comply to the Pitr Paradigm:

Setting up DHCP and booting

To be continued.