The qf File Internals

The qf file holds all the information that is needed to perform delivery of a queued mail message. The information contained in that file, and its appearance, changes from release to release of sendmail. Here, we document the qf file that is used with V8.8 sendmail. Note that as of V8.7 a V version line has been introduced to enable future versions to correctly process older version's queue files.

This section must be taken with a proverbial grain of salt. The internals of the qf file are essentially an internal interface to sendmail and, as such, are subject to change without notice. The information offered here is intended only to help debug sendmail problems. It is not intended (and we strongly discourage its use) as a guide for writing files directly to the queue.

The qf file is line-oriented, containing one item of information per line. Each line begins with a single uppercase character (the code letter), which specifies the contents of the line. Each code letter is followed, with no intervening space, by the information appropriate to the letter. The complete list of code letters is shown in Table 23.2.

qf File Code Letters
Code Meaning Version How Many
B Body type and above At most one
C Controlling user and above At most one per R line
D line Data filename Obsolete as of V8.7 Exactly one
E line Errors to and earlier Many
F line Flag bits and above Many
H line Header definition All versions Many
I line df file's inode number and above Exactly one
K Time last processed and above Exactly one
M Message (why queued) All versions At most one
N line Number times tried and above At most one
P Priority (current) All versions At most one
Q line Original recipient and above At most one per R line
R Recipient address All versions Many
S Sender address All versions Exactly one
T Time created All versions Exactly one
V line Version and above Exactly one
Z line DSN envelope ID and above At most one
$ $ line Restore macro value and above At most one each
line End of qf file and above At most one

Some code letters may appear only once in a qf file; others may appear many times. Any line that begins with a tab or space character is joined to the line above it. Empty lines are ignored. The order in which these lines appear in the qf file is important for the mailq command to work properly.

We discuss the individual lines in the qf file by code letters. Each letter is presented in alphabetical order rather than the order in which they should appear in the qf file.

line

Message body type(V8.6 and above)

The message body type is described under the -B command-line switch (see ). The B line in the qf file stores whatever the body type was set to, either from the command line or by the SMTP MAIL command. The two usual body types are BITMIME or BIT.

The form of the B line is 

Btype

There must be no space between the B and the type. If the type is missing, the body type becomes the character value zero. If the entire B line is missing, the default is BIT. If type is longer than MAXNAME as defined in conf.h (see MAX...) when compiling sendmail, it is truncated to MAXNAME-1 characters when the qf file is read.

Note that the type must be either bit or bitmime. Anything else will not be detected when the qf file is read and may eventually cause the ESMTP dialogue to fail: 

<sender>... Unknown BODY type badtype

This error will be reproduced at every MX site for the recipient until a site that does not speak ESMTP is found or until the MX list is exhausted.

line

Set controlling user(V5.62 and above)

To ensure secure handling of delivery, recipient addresses that are either a file or a program require that sendmail perform delivery as the owner of the file or program rather than as root. A file address is one that begins with a / character. A program address is one that begins with a | character. Both characters are detected after quotation marks have been stripped from the address.

To prevent potential security violations, sendmail must take special precautions when addresses in the qf file result from reading a ~/.forward or :include: file. When such an address is to be placed into the qf file (whether as a recipient's address in an R line or as an error recipient's address in an E line), sendmail first places a C line (for Controlling user) into the file and then the recipient's address. The C line specifies the owner of the ~/.forward or :include: file: 

Cgeorge RPF:/u/users/george/mail/archive Cben RPF:|/u/users/ben/bin/mailfilter

Here, when sendmail later delivers to the recipients in this qf file, it first converts its user identity to that of the user george, then resets itself back to being root again. The same process repeats with the next recipient, except that sendmail changes from root to ben and back again. If there is no C line preceding a R line, the previous C line's value is carried down: 

Cgeorge RPF:/u/users/george/mail/archive RPF:|/u/users/ben/bin/mailfilter <-controlling user is george

The form of the C line in the qf file is 

Cuser <-Prior to V8 Cuser:eaddr <-V8.1 through V8.7.5 Cuser:uid:gid:eaddr <-V8.7.6 and V8.8

The C must begin the line and be immediately followed by user, with no intervening space. If no user follows the C, any prior controlling user is cleared and the identity that is used reverts to that specified by the DefaultUser (u) option (see DefaultUser (g)(u)). If present, the user is the login name of the owner of the ~/.forward or :include: file that yielded the address in the next following R or E line. If user is the name of a user who is unknown to the system, prior to V8.7.6 and prior to V8.8 the effect was the same as if it were missing. Beginning with V8.8 and V8.7.6, an unknown user causes the identity to become that of the uid and gid. Beginning with V8 sendmail, an optional eaddr may be last. If present, the eaddr gives the address to use for error messages.

There may be only one C line immediately preceding each R and E line. Two C lines in a row have the effect of the second superseding the first.

line

Data filename(Obsolete as of V8.7)

Beginning with V8.7 sendmail looks for its data file (the file containing the message body) under the same name as its qf file, but with the q changed into a d. Prior to V8.7, the D line in the qf file contained the name of the file that contained the message body. If the D line was missing, there was no message body. The form of the qf file D line was 

Dfile

The D must begin the line. The file must immediately follow with no intervening space. All text, from the first character following the D to the end of the line, is taken as the name of the file. There is no default for file; either it must be present, or the entire D line must be absent.

The sendmail program opens the df file for reading. If that open fails, sendmail syslog(3)'s the following error message at LOG_CRIT and continues to process the qf file: 

readqf: cannot open dfAA12345

Be aware that sendmail attempts to remove the file after it has been delivered to all recipients. If sendmail is unable to remove the file, and if the LogLevel (L) option (see LogLevel (L)) is greater than 97, sendmail syslog(3)'s the following warning at LOG_DEBUG: 

file: unlink-fail #

The file is the name of the file that could not be removed. The # is the error number, as defined in /usr/include/errno.h.

The df file is opened only when processing the queue file, not when printing it.

line

Send errors to(V8.6 and earlier)

Notification of errors often requires special handling by sendmail. When mail to a mailing list fails, for example, sendmail looks for the owner of that list. If it finds one, the owner, not the sender, receives notification of the error. To differentiate error notification addresses from ordinary sender and recipient addresses, pre-V8.7 sendmail stored error addresses separately in the qf file, one per E line. Beginning with V8.7, this E line is no longer used. Instead, sendmail uses the S line.

The form of the E line in the qf file looks like this: 

Eaddr <-V8.6 and earlier

The E must begin the line. One or more addresses may be entered on that same line. Whitespace and commas may surround the individual addresses. Note, however, that sendmail places only a single address on each E line. There may be multiple E lines. Each is processed in turn.

Each line is fully processed as it is read. That is, the line is scanned for multiple addresses. Each address that is found is alias-expanded. Each resulting new address is processed by rule sets 3 and 0 to resolve a delivery agent for each.

If an alias expands to a program or a file (text that begins with a / or | character), that text is sent out in the delivered message's Errors-To: line in that form. This can cause confusion when the message is later processed and bounced at the receiving site.

line

Saved flag bits(V8.1 and above)

Under V8 sendmail the Timeout.queuewarn option (see ) can specify an interval to wait before notifying the sender that a message could not immediately be delivered. To keep track of whether such a notification has been sent, sendmail stores the state of its EF_WARNING envelope flag in the qf file. If that flag is set, notification has already been sent.

Error mail messages sent by sendmail may occasionally be queued, rather than immediately delivered. The Timeout.queuewarn option notification should not be sent for such mail. If such mail remains in the queue too long, it should be canceled rather than bounced. V8 sendmail saves the state of the EF_RESPONSE envelope flag in the qf file. If that flag is set, the message is an error notification.

Beginning with V8.8, sendmail also records the state of the EF_HAS8BIT flag (the message body contains 8-bit data) and the EF_DELETE_BCC flag (delete empty Bcc: headers; see Bcc:).

All envelope flags are listed in Table 37.3 of -d2.1. The F line is used to save envelope flags for later restoration. Its form looks like this: 

Fflags

Here, the flags are any combination of w, which restores (sets) the EF_WARNING flag; r, which restores the EF_RESPONSE flag; , which restores the EF_HAS8BIT flag; and b, which restores the EF_DELETE_BCC flag. Only those letters are recognized. Other letters are silently ignored. Note that these flags may be done away with in later versions of sendmail and new flags may be added without notice.

For security protection, V8 sendmail rejects and logs the following flag sequence: 

From -^ a space here

See "Funny Flag Bits in qf File" for more information about this.

line

Header line(All versions of sendmail)

The lines of text that form the message header are saved to the qf file, one per H line. Any header lines added by sendmail are also saved to H lines in the qf file.

The form of the H line is: 

Hdefinition

The H must begin the line, and the definition must immediately follow with no intervening space. The definition is exactly the same as, and obeys the same rules as, the H commands in the configuration file (see "The H Configuration Command").

When sendmail writes header lines to the qf file, it pre-expands macros (replaces expressions such as $x with their values) and preresolves conditionals ($?, $!, and $.).

The order in which H lines appear in the qf file is exactly the same as the order in which they appear in the delivered message.

line

Inode and device information for the df file(V8.7 and above)

When a machine crashes under UNIX, files in a directory may become detached from that directory. When this happens, those orphaned files are saved in a directory called lost+found. Because file names are saved only in directories, orphaned files are nameless. Consequently, UNIX stores them in lost+found using their inode numbers as their names.

To illustrate, consider finding these four files in lost+found after a crash: 

#1528 #1200 #3124 #3125

Two of these are qf files, and two are df files. Beginning with V8.7 sendmail the qf files contain a record of the inode numbers for their corresponding df files. That information is stored in the I line: 

Imajor/minor/ino

Here, the major and minor are the major and minor device numbers for the disk device that the df file was stored on. The ino is the inode number for the df file. In our above lost+found example the following command could be run to pair up the orphaned files: 

% grep "^I.*/.*/" * #1200:I123/45/3124 #1325:I123/45/1528

This shows that the qf file #1200 has the df file #3124 and that the qf file #1325 has the df file #1528.

The sendmail program does not check the inode number in the I line against the actual inode number of the df file. Instead, the I line is generated afresh each time the qf file is processed.

line

The time last processed from the queue(V8.7 and above)

The MinQueueAge option (see ) sets the length of time a queued message must remain queued before delivery can again be tried. Each time sendmail processes a qf file, it subtracts the time stored in the K line from the current time and compares the result to the MinQueueAge. If sufficient time has not passed, the rest of processing is skipped. (Note that this test is performed only if the qf file has been processed at least once; see the N line in ).

The time stored in the K line looks like this: 

K703531020

This number represents the date and time in seconds. Every time the qf file is processed (delivery is attempted), the K line is updated with the current time.

line

The reason message was queued(All versions of sendmail)

When a mail message is placed into the queue because of an error during the delivery attempt, the nature of that error is stored in the M line of the qf file. The error is usually prefixed with Deferred: 

Deferred: reason

Delivery can be deferred until a later queue run because of a temporary lack of services. For example, the reason may be "remote host is down."

The form of the qf file M line is: 

Mmsg

The M must begin the line. It is immediately followed by the msg with no intervening space. The text of msg is everything up to the end of the line. The msg created by sendmail may include the word Deferred:, followed by a reason. The M line should appear before the S line.

If the msg is missing, sendmail simply prints a blank line rather than a reason when showing the queue with mailq or the -bp command-line switch. If the M line is entirely missing, sendmail prints nothing.

The maximum number of characters in msg is defined by MAXLINE in conf.h (see ). There should be only one M line in a qf file. If there are multiple M lines, only the last is used. If multiple recipients produced error messages, only the last one is stored in an M line.

line

Number of times tried(V8.7 and above)

Each time delivery is attempted for a message, the number stored in its qf file's N line is incremented by one. This number always begins at zero.

When delivering many messages to a single host, sendmail remembers failures. If one message fails to make it all the way through an SMTP dialogue, all the following messages to that same host will be deferred (not attempted during the current queue run). For those deferred messages the number of tries is correctly incremented as though the delivery was actually attempted.

The value in this N line is used to determine whether the delay of the MinQueueAge option (see ) should be triggered. This value, when zero, can also be used to enable a special first-time connection timeout (see "Timeout.iconnect").

line

Priority when processed from queue(All versions of sendmail)

Not all messages need to be treated equally. Messages that have failed often, for example, tend to continue to fail. When sendmail processes the messages in its queue, it sorts them by priority that was and attempts to deliver those most likely to succeed first.

When a mail message is first placed into the queue, it is given an initial priority calculated when it was first created (see ), which is stored in the P line: 

P640561

This number in the qf file is really a cost. The lower it is, the more preferentially the message is treated by sendmail. Each time the qf file is read, the number in the P line is incremented. The size of that increment is set by the value of the RetryFactor (Z) option (see RetryFactor (Z)). If that option is negative, the logic of "what fails will continue to fail" is inverted.

The form of the qf file P line is 

Ppri

The P must begin the line. The pri is a text representation of an integer value. The pri must immediately follow the P with no intervening space. The text in pri is converted to an integer using the C library routine atol(3). That routine allows pri to be represented in text as a signed decimal number, an octal number, or a hexadecimal number.

If pri is absent, the priority value used is that of the configuration file RetryFactor (Z) option. If the entire P line is absent, the priority value begins as zero.

There should be only one P line in any qf file. Multiple P lines cause all but the last to be ignored.

line

The DSN ORCPT address(V8.7 and above)

When a mail message arrives that includes an ORCPT parameter for the ESMTP RCPT command (see RFC1891), sendmail needs to save that parameter's information separately from the RCPT recipient address: 

RCPT TO:<gw@wash.dc.gov> ORCPT=rfc822;gw@wash.dc.gov -^ -^ recipient address parameter's information

Not all sites understand ESMTP. If sendmail forwards the message to such a site, it needs to omit the ORCPT parameter. Consequently, sendmail must not store that parameter with the RCPT address.

The Q line is used to separately store the ORCPT parameter information: 

Qtype;addr

The type;addr is defined by RFC1891. The sendmail program checks the validity of addr when that information is received but otherwise merely stores type;addr as is in the Q line.

There must be only a single Q line for each recipient R line, and each such Q line must precede its corresponding R line.

line

Recipient's address(All versions of sendmail)

The qf file lists all the recipients for a mail message. There may be one recipient or many. When sendmail creates the qf file, it lists each recipient address on an individual R line. The form of the R line in the qf file looks like this: 

Rflags:addr

The R must begin the line. Only a single address may appear on each R line. There may be multiple R lines. Each is processed in turn.

If the colon is present and if the version of the qf file is greater than 0, the characters between the R and the colon are interpreted as flags that further define the nature of the address: P (primary) Addresses can undergo many transformations prior to delivery. When expanding aliases, for example, the address george might be transformed into two addresses via a ~/.forward file: george@here and george@there. In this instance, george is the primary address, and the aliases are secondary addresses. If aliasing yields only a single transformation, the single new address is considered primary. Addresses that are received via a RCPT SMTP command are always considered primary, as are all other recipient addressees prior to aliasing. N (notify) Recipient addresses can lead to various kinds of notification based on the nature of the DSN NOTIFY extension to the RCPT SMTP command. That notification can be either NEVER or some combination of SUCCESS, FAILURE, or DELAY. Internally, sendmail uses the absence of the latter three to imply NEVER. This N flag simply says that the DSN NOTIFY extension appeared in the message. If the N is absent, but an S, F, or D is present, DSN information will not be propagated. Note that NOTIFY can also be specified by using the -N command line switch (see -N).

Each R line is fully processed as it is read. That is, the line is scanned for multiple addresses. Each address that is found is alias-expanded. Each resulting new address is processed by rule sets 3 and 0 to resolve a delivery agent for each.

line

Sender's address(All versions of sendmail)

Each mail message must have a sender. The sendmail program can determine the sender in four ways:

The form of the S line in the qf file looks like this: 

Saddr

The S must begin the line. Exactly one address must follow on that same line. Whitespace may surround that address. There may be only one S line in the qf file.

If the addr is missing, sendmail sets the sender to be the user who ran sendmail. If that user is not known in the passwd file (or database), sendmail syslog(3)'s the following message and sets the sender to be postmaster: 

Who are you?

The resulting address is then processed to extract the user's full name into $x (see $x). Finally, the sender's address is rewritten by rule sets 3, 1, and then 4.

Under all versions of sendmail the address in the S line will include any RFC822 comment text that appeared with the original message. Under V8.7, if the F=c flag (see F=c) is set for the sender's delivery agent, all comment text is stripped from the address.

If sendmail is compiled with USERDB defined (see USERDB), the sender address can optionally be rewritten by the User Database before it is placed into the S line. Such rewriting is allowed only if the delivery agent for the sender includes the F=i flag (see F=i).

line

Creation time(All versions of sendmail)

To limit the amount of time a message can remain in the queue before being bounced, sendmail must know when that message was first placed in the queue. That time of first placement is stored in the T line in the qf file. For example, the following number represents the date and time in seconds: 

T703531020

Each time sendmail fails to deliver a message from the queue, it checks to see whether too much time has passed. It adds the T line value to the value specified in the Timeout.queuereturn (T) option (see ). If that sum is less than the current time, the message is bounced instead of being left in the queue.

Messages are occasionally left in the queue for longer than the normal timeout period. This might happen, for example, if a remote machine is down but you know that it will eventually be brought back up. There are two ways to lengthen the amount of time a message may remain in the queue.

The preferred way is to create a temporary separate queue directory and move the necessary queued file to that temporary holding place. When the remote site comes back up, you can later process the files in that other queue by running sendmail with an artificially long Timeout.queuereturn value (see ).

A second way to extend the life of messages in the queue is to edit the qf file and change the value stored in the T line. Just add 86400 to that value for each day you want to extend. Care is required to avoid editing a file that is currently being processed by sendmail. [8]

[8] The nvi(1) editor uses the same file locking as sendmail and so can safely be used to edit qf files.

There is currently no plan to give sendmail the ability to rejuvenate queued messages (make old messages appear young).

The form of the T line in the qf file is: 

Tsecs

The T begins the line, and the secs must immediately follow with no intervening space. The numeric text that forms secs is converted to an integer using the C library routine atol(3). That routine allows secs to be represented in text as a signed decimal number, an octal number, or a hexadecimal number.

If secs is absent or the entire T line is absent, the time value is zero. A zero value causes the mail message to time out immediately.

There should be only one T line in any qf file. Multiple T lines cause all but the last to be ignored.

line

Version of the qf file(V8.7 and above)

s sendmail evolves, it will continue to add new abilities to the qf file. To protect old versions of sendmail from wrongly misinterpreting new configuration files, the V line has been introduced. Prior to V8.7 sendmail there was no V line. For versions prior to 8.7.6 the version for the V line is 1: 

V1 <-V8.7.5 and earlier

For V8.7.6 and above, including V8.8, the version is 2 (where version 2 changes the layout of the C line in the qf file): 

V2 <-V8.7.6 and above and V8.8

If the version is greater than that currently supported, sendmail will log this error and exit: 

Version number in qf (bad) greater than max (max)

A version 1 qf file allows flags to follow the R line.

line

DSN ENVID envelope identifier(V8.7 and above)

The MAIL ESMTP command may optionally be followed by an RFC1891 ENVID envelope identifier: 

MAIL From: <address> ENVID=envelopeID

ENVID is used to propagate a consistent envelope identifier (distinct from the Message-ID: header see Message-ID:) that is permanently associated with a message.

The Z line holds that ENVID envelope identifier information: 

ZenvelopeID

The ENVID information needs to be held separately from the S sender line because sendmail has no way to determine in advance whether a recipient host speaks ESMTP.

There must be only a single Z line in any qf file. The ${envid} macro (see ${envid}) also stores the ENVID value.

$ line

Restore macro value(V8.6 and above)

The sendmail program uses the $r macro (see $r) to store the protocol used when sendmail first received a mail message. If the message was received by using SMTP, that protocol is smtp. Otherwise, it is NULL.

The sendmail program uses the $s macro (see $s) to store the full canonical name of the sender's machine.

The sendmail program uses the $_ macro (see $-) to store RFC1413 identd(8) information and IP source routing information.

When sendmail creates a qf file, it saves the values of the $r, $s, and $_ macros in lines that begin with $.

The form of the $ line in the qf file looks like this: 

$Xvalue

The $ must begin the line, and the macro's single-character name (the X) must immediately follow with no intervening space. The X is followed (again with no intervening space) by the value of the macro.

If value is missing, the value given to the macro is NULL. If the X and value are missing, the macro is given a value of NULL. If both are present, the macro that is specified (X) is given the value specified (value).

There may be multiple $ lines. The need to quickly process a qf file requires that only single-character macro names be used.

line

Mark EOF in qf file(V8.7 and above)

One form of attack against sendmail involves appending information to an existing qf file. To prevent such attacks, V8.7 introduced the dot line. In a qf file, any line that begins with a single dot: 

followed by anything

is considered to mark the end of the file's useful information. Upon encountering that dot, sendmail continues to read the qf file. If any line follows the dot line, sendmail logs the following message and changes the qf file into a Qf file (see "Extra Data at End of qf File"): 

SECURITY ALERT: extra data in qf: bogus line here