Random thoughts, tips & tricks about Slackware-Linux, Lego and Star Wars

Qemu, snapshots, temporary files, where they go and how to change that

March 14th, 2011 by Niels Horn in , ,

I use for several tasks, but its primary use is for testing programs. It has a wonderful feature called "snapshots", where you can save a known state of the Virtual Machine you can fall back to if anything goes wrong.
For my recent adventure with keyboard layouts I researched several sources on Qemu and noticed that there were many users having problems or doubts about these snapshots and the temporary files that are created. So I decided to have a go at explaining a few things that are a bit obscure…

How I use snapshots

To make things clear, I should explain how I use the snapshot function myself and how I was puzzled as well in the beginning.
I normally create a "clean" virtual machine, doing a full install of my favorite operating system, , without any extra packages and without any special configurations. After finishing the installations, this becomes my "clean machine", that I only change in case of security updates etc. I have "clean machines" for Slackware 13.0, 13.1 and 13.37-RC1 at the moment. (the last one is being updated for all patches that come out but will be reinstalled from scratch as soon as 13.37 becomes official).

From this point on I use a script to start my VM in Qemu that uses the "-snapshot" option, which mean that everything I changed, installed, deleted, etc., is forgotten when I close the VM. So I can build packages, install them, install needed dependencies, and - when finished - I end the VM and will have my "clean machine" back exactly the way it was when I started testing.
This is better than magic! :) I can test whatever I want, make a mess, install all kinds of good or bad packages and simply go back to the original environment without any worries.

TINSTAAFL (*)

But, there is no real magic going on, of course… All the changes are saved in a temporary file, that is read as a change-log on top of the hard-disk image. When closing down the virtual machine, the temporary file with all the changes is deleted and the hard-disk image becomes the original and only source of data again, as it was when I installed the operating system.

Some time ago, when I started to build *BIG* packages, like brlcad and OpenCASCADE, that need e few GB's of disk space, I ran into problems…
brlcad is the champion, needing about 5GB of space in /tmp/SBo to build the package. As I said, all these changes are not saved on the virtual hard-disk image, but in that temporary file. And my host computer that runs the VMs started to complain that the /tmp partition was out of space. Then I started looking for these temporary files….

Where are they?

My disk space in /tmp was running low, but I could not find any large file there. That was puzzling…
But looking in the source code, I understood the trick.
Qemu creates the temporary file and then "unlinks" it. This unlinking basically means that the file is deleted, but Linux keeps it as long as it is in use by the process that created it (in other words: Qemu). So the temporary file is not visible, because it has been deleted. But it is still there, occupying disk space, until the Qemu process ends.

Well, of course it is visible, if you know how / where to look for it…
First, let's look for the Qemu process: 

root@niels-sw:~# ps -ef | grep qemu
root 16291 1 98 21:38 pts/7 00:59:25 qemu -localtime -boot c -hda hda.img -m 1024M -name slack-1337 -net nic,macaddr=52:54:00:12:34:90 -net tap,ifname=tap0 -snapshot

So Qemu is there, running as root, with the "-snapshot" option, process ID ("pid") 16291.
Now let's check the files this process has open: 

root@niels-sw:~# lsof -p 16291
...
qemu 16291 root 8u REG 253,2 1636237312 14 /tmp/vl.W2tkAX (deleted)
...

There are many more files that are opened, but I just wanted to show the one we are looking for.
It sits in /tmp and is a "deleted" file and already occupies 1.6GB (building OpenCASCADE here as a test).

So the file is there, it can be detected with some special commands, but will fill up my /tmp partition in a while!
OK, so we simply redirect the file to another partition, where I have lots of space left!
Eh… well, but how do we do that?

How to change the location of the Qemu temporary files

No, there is no command-line option for it… I really think the Qemu developers should create this option, letting us start Qemu like "qemu -tmpdir /var/whatever/" but I have seen this feature request since Qemu 8.0 and it still is not there.
There is however a hidden (I never found this in the documentation) feature…
Browsing the code, there seems to be a solution. This is the part in block.c that defines the filename for the temporary file: 

void get_tmp_filename(char *filename, int size)
{
 int fd;
 const char *tmpdir;
 /* XXX: race condition possible */
 tmpdir = getenv("TMPDIR");
 if (!tmpdir)
 tmpdir = "/tmp";
 snprintf(filename, size, "%s/vl.XXXXXX", tmpdir);
 fd = mkstemp(filename);
 close(fd);
}

"Oh, that looks simple enough!" I thought when I first saw that code… Simply set a "TMPDIR=/var/whatever" and run Qemu!
No, not really…

Most people start Qemu from a non-privileged account (that is: not as root), using sudo.
It *is* possible to run Qemu without using sudo, but it is much more complicated to setup, especially if you want to use the KVM kernel module, use tap devices, use ifconfig commands, etc.
But if you use sudo, the environment variable you set as a non-privileged user won't be known to root who's going to start Qemu.
This is why the few people who know about the TMPDIR variable complain that it is "not working".

So how did I solve this?

Actually, I didn't. I use Qemu as a normal user 99% of the time, but when I need to build one of the bigger packages, I use "su -" to run as root and start Qemu like this: 

TMPDIR=/mnt/spare qemu -hda hda.img -m 1024M -snapshot ....

where /mnt/spare is a 100G drive I have mounted for these occasions.
(this is all done by a small script, actually).

Conclusion

Qemu creates invisible temporary files in /tmp by default.
They can be redirected to another location using the TMPDIR environment variable, but not if you're using sudo to start Qemu. In this case you will need to login as root or become root with "su".

Now what about that acronym?

(*) TINSTAAFL = There Is No Such Thing As A Free Lunch
This is true for about everything in life…