Lguest: linux.conf.au 2008 Tutorial

Pretty Picture of Task Dependencies

Tasks in Detail

Add a LGUEST_DEVICE_S_ACTIVE bit and set it in the network driver

Every device has a descriptor in the lguest_devices page. We want to add a new bit to the 'status' field for the driver to indicate it's actually active.

See lguest_dev_probe() how to manipulate a devices' descriptor status field. Set this new status bit on successful lguest_netopen(), and remove it again on lguest_netclose().

You can use the 'status' command to check it's working as you ifconfig up and down the network device in the Guest.

• include/linux/lguest_launcher.h (define LGUEST_DEVICE_S_ACTIVE)
• drivers/net/lguest_net.c (modify lguestnet_open()/lguestnet_close())
• drivers/lguest/lguest_bus.c (lguest_dev_probe())

Write a new hypercall which prints your name using printk().

Call the hypercall at the bottom of lguest_init(), just before start_kernel().

• include/linux/lguest.h (hypercall numbers)
• drivers/lguest/hypercalls.c (hypercall implementations)
• drivers/lguest/lguest.c (lguest_init())

Reboot hook in Guest

Currently the Guest crashes if you try to reboot, and not gracefully.

Create a function called lguest_reboot() which calls the "LHCALL_CRASH" hcall. Set machine_ops.restart to point to your function.

• drivers/lguest/lguest.c (where to put lguest_reboot)
• arch/i386/xen/enlighten.c (to see how Xen does this already)

Fix lguest_reboot() to use new LHCALL_STOP

Change lguest_reboot() to use the LHCALL_STOP hcall with the LHCALL_STOP_REBOOT argument.

• drivers/lguest/lguest.c (lguest_reboot())

Create a "kill" ioctl which kills the guest with reason "killed by launcher".

A little example so we can test our ioctl handler is working. Let's call it LGKILL, using letter 'L'. Call it instead of exiting when the user hits three ^C in handle_console_input().

• include/linux/lguest_launcher.h (to define LGKILL ioctl)
• include/asm-i386/ioctls.h (to see how others define ioctls)
• drivers/lguest/lguest_user.c (handle_console_input)

On LHCALL_STOP reboot, set lg->dead to ERR_PTR(-ELOOP).

Any error will cause an error to be returned to the Guest. ELOOP seems appropriate for reboot.

On LHCALL_STOP w/ LHCALL_STOP_SHUTDOWN, set lg->dead to an empty string.

This will allow the launcher to tell the Guest exited, rather than crashing.

• drivers/lguest/hypercalls.c

Rename LHCALL_CRASH to LHCALL_STOP, and add an extra "reason" argument.

Reasons:
0 == LHCALL_STOP_CRASH
1 == LHCALL_STOP_SHUTDOWN
2 == LHCALL_STOP_REBOOT

LHCALL_CRASH takes a string argument (in "edx"). Use "ebx". You can reuse the same hypercall number, since LHCALL_CRASH currently gives a 0 arg for "ebx" anyway.

• include/linux/lguest.h (to define LHCALL_STOP_CRASH etc)
• drivers/lguest/hypercalls.c (to replace LHCALL_CRASH with LHCALL_STOP)
• drivers/lguest/lguest.c (to fixup LHCALL_CRASH users)

Write a close_all() function for the Launcher to close all file descriptors

(Don't close STDOUT_FILENO, STDERR_FILENO or STDIN_FILENO). Test this function by calling it before exit().

For testing I recommend inserting a sleep(30) at the end and looking at /proc//fd to see what filedescriptors are open.

• drivers/lguest/lguest_user.c (add close_all())

Add a --control= arg for controlling the guest

This will be useful to send control messages to the Launcher: the argument is the filename of a fifo to create (see mkfifo(3)).

Write a function setup_control() which takes the filename, tries to mkfifo it and then opens it. If mkfifo fails with errno equal to EEXIST, try to open the file anyway.

Call the function when they specify --control, and ignore the opened fd for the moment.

• Documentation/lguest/lguest.c (add setup_control(), add --control)

Listen to the control fd, and print out anything which comes in.

Create a dummy "control" device in setup_control() using new_device(): set the type to 0xFFFF (which should be ignored by the Guest), and ask for 0 pages. The output handler should NULL, and the input handler (say, handle_control_input) should simply read dev->fd into a buffer and print out the result.

• Documentation/lguest/lguest.c (setup_control()

Suspend when told to by the control fd

Modify handle_control_input() to respond to the string "suspend" by sending itself a SIGSTOP.

• Documentation/lguest/lguest.c (handle_control_input()

Write a reboot() routine for the launcher which execs itself

Re-execing is the simplest way to "reboot" a userspace program.

• drivers/lguest/lguest_user.c (add reboot())

Put a header on the memory file

Create a function open_memfile() which opens the memory file and puts a simple header with two fields at the beginning of it. This function should be used by map_zeroed_pages(), instead of opening itself.

Make the header a page long, so the rest can still be mmaped.

The two fields should be: a version number, and the header length. There'll be more later.

• Documentation/lguest/lguest.c (open_memfile(), map_zeroed_pages())

Allow user to specify name of memory file

The easiest way to do this is to create a global char * variable "memfile", and use it in map_zeroed_pages().

Initialize memfile to $HOME/.lguest/. If they specify the "--memfile" option then call a new function use_memfile() which overrides it with that value. Files:

• Documentation/lguest/lguest.c (map_zeroed_pages(), main())

Fix Launcher to warn if discarding packets when LGUEST_DEVICE_S_ACTIVE

Currently the Launcher uses a horrible hack to decide whether to print a warning when it can't send network packets to the host: it keeps a boolean flag in the network device's priv member, which is set when the Guest sends a packet out. After this, we assume the Guest network is ready.

Instead, change the test in handle_tun_input() to warn only if the network device's LGUEST_DEVICE_S_ACTIVE bit is set, and remove the (bool *)dev->priv hack from handle_tun_output(), handle_tun_input() and setup_tun_net().

• Documentation/lguest/lguest.c (handle_tun_input() etc)

Have the Launcher "reboot" itself when the final read returns -ELOOP.

Remember to call close_exec() before calling reboot(); you should reboot the Guest a few times and check that the number of files in /proc//fd doesn't increase.

• drivers/lguest/lguest_user.c (run_guest())

Extend memory file header to contain room for register state.

• Documentation/lguest/lguest.c (open_memfile())

Launcher clean exit on Guest shutdown.

Modify the Launcher to exit with status 0 on a zero-length read after -ENOENT.

• Documentation/lguest/lguest.c

Print out device status when 'status' comes in the control fd

Modify handle_control_input() to respond to the string "status" by iterating through the devices and printing out useful information, particularly the flags each device has set in the 'status' field in its descriptor.

You might want to add a 'const char *name' field to 'struct device' (and initialize it in new_device()) to make the output more readable.

• Documentation/lguest/lguest.c (handle_control_input()

Implement a new control sequence ^Z^Z^Z

This is similar to the ^C^C^C logic, except it sends a SIGSTOP (see man kill(2)) to itself to "suspend".

(For bonus points, do the tcsetattr() on return from SIGSTOP to restore the terminal to raw mode).

• Documentation/lguest/lguest.c (handle_console_input())

Ensure that Launcher doesn't set evil flags

Make sure you set the 0x202 bits in eflags (interrupts enabled and reserved bit set). Also the lower two bits of the segment registers indicate the privilege level: set that to GUEST_PL if it's 0.

lguest/drivers/lguest_user.c (initialize())

Create a new write() LHREQ_FLUSHPAGES to flush page mappings.

The current code assumes that the Launcher will never remap pages accessible by the Guest: once the Guest accesses them it can simply put the pages into the Guest page tables.

If the Launcher is going to change page mappings, it needs a way to tell the kernel to forget about the old mappings. We do this by adding a new command to enum lguest_req, called LHREQ_FLUSHPAGES: there's already a function called guest_pagetable_clear_all() which does just what we want.

If you hack the Launcher to do this write on every console output, you should see a marked degredation in Guest performance.

• include/linux/lguest_launcher.h (enum lguest_req)
• drivers/lguest/lguest_user.c (write())

Extend /dev/lguest with an ioctl handler, which always returns -ENOTTY.

For more complicated Guest manipulation (such as getting Guest state), read and write are awkward; we need an ioctl. Call it lg_ioctl().

ioctls are supposed to return -ENOTTY when they don't understand a value. Modify the Launcher to call the ioctl before run_guest() and make sure it returns -1 with errno=ENOTTY.

• drivers/lguest/lguest_user.c (lg_ioctl() and lguest_fops.ioctl)
• include/linux/fs.h (ioctl prototype)
• Documentation/lguest/lguest.c (to test ioctl)

Fix lguest I/O bug for shared pages

lguest's I/O system uses memory addresses as locations for I/O: if one Guest does a LHCALL_BIND_DMA to an address, another Guest can send I/O using LHCALL_SEND_DMA to that address (which it could only do if they share memory).

Unfortunately, the I/O code assumes that I/O aimed at a shared page is always going to another Guest, not the Launcher. It should send the I/O to the Launcher if no Guest is found, so we can use shared memory for normal Guest memory without breaking I/O.

The code is in send_dma(). You can fix this most easily by adding a "found_guest" flag, setting it to 1 if we call dma_transfer and doing the "Guest is sending to its Launcher" path if it's not set.

• drivers/lguest/io.c (send_dma())

Create and map an actual file in $HOME/.lguest/, instead of /dev/zero.

To access Guest memory from other processes, we need to memory map a file rather than an anonymous mapping. So make map_zeroed_pages() map the empty file, rather than /dev/zero.

The first time it's called, it should use the O_TRUNC to open the file in case it already exists. Use a static integer to track the file length. If they want to map past the current end of file, use ftruncate() to extend it and increase the length.

• Documentation/lguest/lguest.c (map_zeroed_pages())

Restore Guest from memory file

When --memfile is specified, use register state from file header to feed to LHREQ_INITIALIZE, instead of starting fresh.

• Documentation/lguest/lguest.c (use_memfile())

Suspend Guest into memory file

On suspend, use the LGGETREG to save the register state into the memory file header, then print a message about suspending, and exit.

• Documentation/lguest/lguest.c (handle_console_input())

Implement /dev/rtc for the Guest

hwclock in the guest looks for /dev/rtc before trying (and failing) to program the clock directly. It would be nice to have a working /dev/rtc in the Guest to fix this message in the kernel logs:

hwclock[939] general protection eip:804b7b3 esp:bf8f5fe0 error:0

• include/linux/rtc.h (struct rtc_class_ops)
• drivers/rtc/rtc-test.c (example driver)

LGGETREG ioctl to retrieve the entire register

Implement an ioctl LGGETREG to retrieve the entire register state and the value of lguest_data. Make sure this matches the format in A-hsetregs.

(You don't need the LGKILL ioctl any more, BTW, it was just an example).

• drivers/lguest/lguest_user.c (lg_ioctl())

Supply all register data in LHREQ_INITIALIZE

Extend the Launcher's LHREQ_INITIALIZE call to give the kernel the entire Guest register state, and the location of lguest_data (or 0, to mean it's not set). Make sure the format mathces A-hgetreg.

This effectively means moving setup_regs() into the Launcher: a stepping stone towards suspend/resume.

• drivers/lguest/lguest_user.c (setup_regs())
• Documentation/lguest/lguest.c (tell_kernel())

Export matching memory

When you find matching memory, call a new function export_memory() which copies it into $HOME/.lguest/shareable/