Mounting filesystems

This section uses filenames and command names specific to Solaris. Note that you are better off using the automounter (see "The Automounter") to mount filesystems, rather than using the mount utility described in this section. However, understanding the automounter, and why it is better than mount, requires understanding mount. Thus, we will discuss the concept of NFS filesystem mounting in the context of mount. Solaris has different component names from non-Solaris systems. Table 6-3 shows the rough equivalents to non-Solaris systems.

Table 6-3. Correspondence of Solaris and non-Solaris mount components

Description Solaris Non-Solaris
List of filesystems /etc/vfstab /etc/fstab
List of mounted filesystems /etc/mnttab /etc/mtab
RPC program number to network address mapper (portmapper) rpcbind portmap
MOUNT daemon mountd rpc.mountd

NFS clients can mount any filesystem, or part of a filesystem, that has been exported from an NFS server. The filesystem can be listed in the client's /etc/vfstab file, or it can be mounted explicitly using the mount(1M) command. (Also, in Solaris, see the mount_nfs(1M) manpage, which explains NFS-specific details of filesystem mounting.)NFS filesystems appear to be "normal" filesystems on the client, which means that they can be mounted on any directory on the client. It's possible to mount an NFS filesystem over all or part of another filesystem, since the directories used as mount points appear the same no matter where they actually reside. When you mount a filesystem on top of another one, you obscure whatever is "under" the mount point. NFS clients see the most recent view of the filesystem. These potentially confusing issues will be the foundation for the discussion of NFS naming schemes later in this chapter.

Using /etc/vfstab

Adding entries to /etc/vfstab is one way to mount NFS filesystems. Once the entry has been added to the vfstab file, the client mounts it on every reboot. There are several features that distinguish NFS filesystems in the vfstab file: Some typical vfstab entries for NFS filesystems are: 
ono:/export/ono

-

/hosts/ono

nfs

-

yes

rw,bg,hard



onaga:/export/onaga

-

/hosts/onaga

nfs

-

yes

rw,bg,hard



wahoo:/var/mail

-

/var/mail

nfs

-

yes

rw,bg,hard


The yes in theabove entries says to mount the filesystems whenever the system boots up. This field can be yes or no, and has the same effect for NFS and non-NFS filesystems.Of course, each vendor is free to vary the server and filesystem name syntax, and your manual set should provide the best sample vfstab entries.

Using mount

While entries in the vfstab file are useful for creating a long-lived NFS environment, sometimes you need to mount a filesystem right away or mount it temporarily while you copy files from it. The mount command allows you to perform an NFS filesystem mount that remains active until you explicitly unmount the filesystem using umount, or until the client is rebooted. As an example of using mount, consider building and testing a new /usr/local directory. On an NFS client, you already have the "old" /usr/local, either on a local or NFS-mounted filesystem. Let's say you have built a new version of /usr/local on the NFS server wahoo and want to test it on this NFS client. Mount the new filesystem on top of the existing /usr/local: 

# mount wahoo:/usr/local /usr/local


Anything in the old /usr/local is hidden by the new mount point, so you can debug your new /usr/local as if it were mounted at boot time. From the command line, mount uses a server name and filesystem name syntax similar to that of the vfstab file. The mount command assumes that the type is nfs if a hostname appears in the device specification. The server filesystem name must be an absolute pathname (usually starting with a leading /), but it need not exactly match the name of a filesystem exported from the server. Barring the use of the nosub option on the server (see "Exporting options" earlier in this chapter), the only restriction on server filesystem names is that they must contain a valid, exported server filesystem name as a prefix. This means that you can mount a subdirectory of an exported filesystem, as long as you specify the entire pathname to the subdirectory in either the vfstab file or on the mount command line. Note that the rw and hard suboptions are redundant since they are the defaults (in Solaris at least). This tutorial often specifies them in examples to make it clear what semantics will be. For example, to mount a particular home directory from /export/home of server ono, you do not have to mount the entire filesystem. Picking up only the subdirectory that's needed may make the local filesystem hierarchy simpler and less cluttered. To mount a subdirectory of a server's exported filesystem, just specify the pathname to that directory in the vfstab file: 

ono:/export/home/stern - /users/stern nfs - yes rw,bg,hard


Even though server ono exports all of /export/home, you can choose to handle some smaller portion of the entire filesystem.

Mount options

NFS mount options are as varied as the vendors themselves. There are a few well-known and widely supported options, and others that are added to support additional NFS features or to integrate secure remote procedure call systems. As with everything else that is vendor-specific, your system's manual set provides a complete list of supported mount options. Check the manual pages for mount(1M), mount_nfs(1M), and vfstab(4).
TIP: For the most part, the default set of mount options will serve you fine. However, pay particular attention to the nosuid suboption, which is described in "Network Security". The nosuid suboption is not the default in Solaris, but perhaps it ought to be.
The Solaris mount command syntax for mounting NFS filesystems is: 

mount [ -F nfs ] [-mrO] [ -o suboptions ] server:pathname mount [ -F nfs ] [-mrO] [ -o suboptions ] mount_point mount [ -F nfs ] [-mrO] [ -o suboptions ] server:pathname mount_point mount [ -F nfs ] [-mrO] [ -o suboptions ] server1:pathname1,server2:pathname2,...serverN:pathnameN mount_point mount [ -F nfs ] [-mrO] [ -o suboptions ] server1,server2,...serverN:pathname mount_point


The first two forms are used when mounting a filesystem listed in the vfstab file. Note that server is the hostname of the NFS server. The last two forms are used when mounting replicas. See "Naming schemes" later in this chapter. The -F nfs option is used to specify that the filesystem being mounted is of type NFS. The option is not necessary because the filesystem type can be discerned from the presence of host:pathname on the command line. The -r option says to mount the filesystem as read-only. The preferred way to specify read-only is the ro suboption to the -o option. The -m option says to not record the entry in the /etc/mnttab file. The -O option says to permit the filesystem to be mounted over an existing mount point. Normally if mount_point already has a filesystem mounted on it, the mount command will fail with a filesystem busy error. In addition, you can use -o to specify suboptions. Suboptions can also be specified (without -o) in the mount options field in /etc/vfstab. The common NFS mount suboptions are:
TIP: It is a popular misconception that NFS Version 3 and NFS over TCP are synonymous. As noted previously, the NFS protocol version is independent of the transport protocol used. You can have NFS Version 2 clients and servers that support TCP and UDP (or just TCP, or just UDP). Similarly, you can have NFS Version 3 clients that support TCP and UDP (or just TCP, or just UDP). This misconception arose because Solaris 2.5 introduced both NFS Version 3 and NFS over TCP at the same time, and so NFS mounts that previously used NFS Version 2 over UDP now use NFS Version 3 over TCP.


onaga:/export/home/mre - /users/mre nfs - yes rw,proto=udp,retrans=6,timeo=11


TIP: It is a popular misconception that if the minimum attribute timeout is set to 30 seconds, that the NFS client will issue a request to get new attributes for each open file every 30 seconds. Marketing managers for products that compete with NFS use this misconception to claim that NFS is therefore a network bandwidth hog because of all the attribute requests that are sent around. The reality is that the attribute timeouts are checked only whenever a process on the NFS client tries to access the file. If the attribute timeout is 30 seconds and the client has not accessed the file in five hours, then during that five-hour period, there will be no NFS requests to get new attributes. Indeed, there will be no NFS requests at all. For files that are being continuously accessed, with an attribute timeout of 30 seconds, you can expect to get new attribute requests to occur no more often than every 30 seconds. Given that in NFS Version 2, and to an even higher degree in NFS Version 3, attributes are piggy-backed onto the NFS responses, attribute requests would tend to be seen far less often than every 30 seconds. For the most part, attribute requests will be seen most often when the NFS client opens a file. This is to guarantee cache consistency. See "File attribute caching" for more details.
The nointr, intr, retrans, rsize, wsize, timeo, hard, soft, and ac* options will be discussed in more detail in the "Client-Side Performance Tuning", since they are directly responsible for altering clients' performance in periods of peak server loading.

Backgrounding mounts

The mount protocol used by clients is subject to the same RPC timeouts as individual NFS RPC calls. When a client cannot mount an NFS filesystem during the allotted RPC execution time, it retries the RPC operation up to the count specified by the retry mount option. If the bg mount option is used, mount starts another process that continues trying to mount the filesystem in the background, allowing the mount command to consider that request complete and to attempt the next mount operation. If bg is not specified, mount blocks waiting for the remote fileserver to recover, or until the mount retry count has been reached. The default value of 10,000 may cause a single mount to hang for several hours before mount gives up on the fileserver. You cannot put a mount in the background of any system-critical filesystem such as the root ( / ) or /usr filesystem on a diskless client. If you need the filesystem to run the system, you must allow the mount to complete in the foreground. Similarly, if you require some applications from an NFS-mounted partition during the boot process -- let's say you start up a license server via a script in /etc/rc2.d -- you should hard-mount the filesystem with these executables so that you are not left with a half-functioning machine. Any filesystem that is not critical to the system's operation can be mounted with the bg option. Use of background mounts allows your network to recover more gracefully from widespread problems such as power failures. When two servers are clients of each other, the bg option must be used in at least one of the server's /etc/vfstab files. When both servers boot at the same time, for example as the result of a power failure, one usually tries to mount the other's filesystems before they have been exported and before NFS is started. If both servers use foreground mounts only, then a deadlock is possible when they wait on each other to recover as NFS servers. Using bg allows the first mount attempt to fail and be put into the background. When both servers finally complete booting, the backgrounded mounts complete successfully. So what if you have critical mounts on each client, such that backgrounding one is not appropriate? To cope, you will need to use the automounter (see "The Automounter") instead of vfstab to mount NFS filesystems. The default value of the retry option was chosen to be large enough to guarantee that a client makes a sufficiently good effort to mount a filesystem from a crashed or hung server. However, if some event causes the client and the server to reboot at the same time, and the client cannot complete the mount before the retry count is exhausted, the client will not mount the filesystem even when the remote server comes back online. If you have a power failure early in the weekend, and all the clients come up but a server is down, you may have to manually remount filesystems on clients that have reached their limit of mount retries.

Hard and soft mounts

The hard and soft mount options determine how a client behaves when the server is excessively loaded for a long period or when it crashes. By default, all NFS filesystems are mounted hard, which means that an RPC call that times out will be retried indefinitely until a response is received from the server. This makes the NFS server look as much like a local disk as possible -- the request that needs to go to disk completes at some point in the future. An NFS server that crashes looks like a disk that is very, very slow. A side effect of hard-mounting NFS filesystems is that processes block (or "hang") in a high-priority disk wait state until their NFS RPC calls complete. If an NFS server goes down, the clients using its filesystems hang if they reference these filesystems before the server recovers. Using intr in conjunction with the hard mount option allows users to interrupt system calls that are blocked waiting on a crashed server. The system call is interrupted when the process making the call receives a signal, usually sent by the user typing CTRL-C (interrupt) or using the kill command. CTRL-\ (quit) is another way to generate a signal, as is logging out of the NFS client host. When using kill only SIGINT, SIGQUIT, and SIGHUP will interrupt NFS operations. When an NFS filesystem is soft-mounted, repeated RPC call failures eventually cause the NFS operation to fail as well. Instead of emulating a painfully slow disk, a server exporting a soft-mounted filesystem looks like a failing disk when it crashes: system calls referencing the soft-mounted NFS filesystem return errors. Sometimes the errors can be ignored or are preferable to blocking at high priority; for example, if you were doing an ls -l when the NFS server crashed, you wouldn't really care if the ls command returned an error as long as your system didn't hang. The other side to this "failing disk" analogy is that you never want to write data to an unreliable device, nor do you want to try to load executables from it. You should not use the soft option on any filesystem that is writable, nor on any filesystem from which you load executables. Furthermore, because many applications do not check return value of the read(2) system call when reading regular files (because those programs were written in the days before networking was ubiquitous, and disks were reliable enough that reads from disks virtually never failed), you should not use the soft option on any filesystem that is supplying input to applications that are in turn using the data for a mission-critical purpose. NFS only guarantees the consistency of data after a server crash if the NFS filesystem was hard-mounted by the client. Unless you really know what you are doing, neveruse the soft option. We'll come back to hard- and soft-mount issues in when we discuss modifying client behavior in the face of slow NFS servers in "Client-Side Performance Tuning".

Resolving mount problems

There are several things that can go wrong when attempting to mount an NFS filesystem. The most obvious failure of mount is when it cannot find the server, remote filesystem, or local mount point. You get the usual assortment of errors such as "No such host" and "No such file or directory." However, you may also get more cryptic messages like: 

client# mount orion:/export/orion /hosts/orion mount: orion:/export/orion on /hosts/orion: No such device.


If either the local or remote filesystem was specified incorrectly, you would expect a message about a nonexistent file or directory. The device hint in this error indicates that NFS is not configured into the client's kernel. The device in question is more of a pseudo-device -- it's the interface to the NFS vnode operations. If the NFS client code is not in the kernel, this interface does not exist and any attempts to use it return invalid device messages. We won't discuss how to build a kernel; check your documentation for the proper procedures and options that need to be included to support NFS. Another cryptic message is "Permission denied." Often this is because the filesystem has been exported with the options rw=client_list or ro=client_list and your client is not in client_list. But sometimes it means that the filesystem on the server is not exported at all. Probably the most common message on NFS clients is "NFS server not responding." An NFS client will attempt to complete an RPC call up to the number of times specified by the retrans option. Once the retransmission limit has been reached, the "not responding" message appears on the system's console (or in the console window): 

NFS server bitatron not responding, still trying


followed by a message indicating that the server has responded to the client's RPC requests: 

NFS server bitatron OK


These "not responding" messages may mean that the server is heavily loaded and cannot respond to NFS requests before the client has had numerous RPC timeouts, or they may indicate that the server has crashed. The NFS client cannot tell the difference between the two, because it has no knowledge of why its NFS RPC calls are not being handled. If NFS clients begin printing "not responding" messages, a server have may have crashed, or you may be experiencing a burst of activity causing poor server performance. A less common but more confusing error message is "stale filehandle." Because NFS allows multiple clients to share the same directory, it opens up a window in which one client can delete files or directories that are being referenced by another NFS client of the same server. When the second client goes to reference the deleted directory, the NFS server can no longer find it on disk, and marks the handle, or pointer, to this directory "invalid." The exact causes of stale filehandles and suggestions for avoiding them are described in "Stale filehandles". If there is a problem with the server's NFS configuration, your attempt to mount filesystems from it will result in RPC errors when mount cannot reach the portmapper (rpcbind) on the server. If you get RPC timeouts, then the remote host may have lost its portmapper service or the mountd daemon may have exited prematurely. Use ps to locate these processes: 

server% ps -e | grep -w mountd 274 ? 0:00 mountd server% ps -e | grep -w rpcbind 106 ? 0:00 rpcbind


You should see both the mountd and the rpcbind processes running on the NFS server. If mount promptly reports "Program not registered," this means that the mountd daemon never started up and registered itself. In this case, make sure that mountd is getting started at boot time on the NFS server, by checking the /etc/dfs/dfstabfile. See "Setting up NFS" earlier in this chapter. Another mountd-related problem is two mountd daemons competing for the same RPC service number. On some systems (not Solaris), there might be a situation when one mount daemon can be started in the boot script and one configured into /etc/inet/inetd.conf; the second instance of the server daemon will not be able to register its RPC service number with the portmapper. Since the inetd-spawned process is usually the second to appear, it repeatedly exits and restarts until inetd realizes that the server cannot be started and disables the service. The NFS RPC daemons should be started from the boot scripts and not from inetd, due to the overhead of spawning processes from the inetd server (see "Internet and RPC server configuration"). There is also a detection mechanism for attempts to make "transitive," or multihop, NFS mounts. You can only use NFS to mount another system's local filesystem as one of your NFS filesystems. You can't mount another system's NFS-mounted filesystems. That is, if /export/home/bob is local on serverb, then all machines on the network must mount /export/home/bob from serverb. If a client attempts to mount a remotely mounted directory on the server, the mount fails with a multihop error message. Let's say NFS client marble has done: 

# mount serverb:/export/home/bob /export/home/bob


and marble is also an NFS server that exports /export/home. If a third system tries to mount marble:/export/home/bob, then the mount fails with the error: 

mount: marble:/export/home/bob on /users/bob: Too many levels of remote in path


"Too many levels" means more than one -- the filesystem on the server is itself NFS-mounted. You cannot nest NFS mounts by mounting through an intermediate fileserver. There are two practical sides to this restriction: We've seen how to export NFS filesystems on a network and how NFS clients mount them. With this basic explanation of NFS usage, we'll look at how NFS mounts are combined with symbolic links to create more complex -- and sometimes confusing -- client filesystem structures.