########################################### Xen HOWTO University of Cambridge Computer Laboratory http://www.cl.cam.ac.uk/netos/xen ############################# Get Xen Source Code ============================= The public master BK repository for the 1.2 release lives at: 'bk://xen.bkbits.net/xeno-1.2.bk' To fetch a local copy, first download the BitKeeper tools at: http://www.bitmover.com/download with username 'bitkeeper' and password 'get bitkeeper'. Then install the tools and then run: # bk clone bk://xen.bkbits.net/xeno-1.2.bk Under your current directory, a new directory named 'xeno-1.2.bk' has been created, which contains all the necessary source codes for the Xen hypervisor and Linux guest OSes. To get newest changes to the repository, run # cd xeno-1.2.bk # bk pull Build Xen ============================= Hint: To see how to build Xen and all the control tools, inspect the tools/misc/xen-clone script in the BK repository. This script can be used to clone the repository and perform a full build. To build Xen manually: # cd xeno-1.2.bk/xen # make clean # make This will (should) produce a file called 'xen' in the current directory. This is the ELF 32-bit LSB executable file of Xen. You can also find a gzip version, named 'xen.gz'. To install the built files on your server under /usr, type 'make install' at the root of the BK repository. You will need to be root to do this! Hint: There is also a 'make dist' rule which copies built files to an install directory just outside the BK repo; if this suits your setup, go for it. Build Linux as a Xen guest OS ============================== This is a little more involved since the repository only contains a "sparse" tree -- this is essentially an 'overlay' on a standard linux kernel source tree. It contains only those files currently 'in play' which are either modified versions of files in the vanilla linux tree, or brand new files specific to the Xen port. So, first you need a vanilla linux-2.4.24 tree, which is located at: http://www.kernel.org/pub/linux/kernel/v2.4 Then: # mv linux-2.4.24.tar.gz /xeno-1.2.bk # cd /xeno-1.2.bk # tar -zxvf linux-2.4.24.tar.gz You'll find a new directory 'linux-2.4.24' which contains all the vanilla Linux 2.4.24 kernel source codes. Hint: You should choose the vanilla linux kernel tree that has the same version as the "sparse" tree. Next, you need to 'overlay' this sparse tree on the full vanilla Linux kernel tree: # cd /xeno-1.2.bk/xenolinux-2.4.24-sparse # ./mkbuildtree ../linux-2.4.24 Finally, rename the buildtree since it is now a 'xenolinux' buildtree. # cd /xeno-1.2.bk # mv linux-2.4.24 xenolinux-2.4.24 Now that the buildtree is there, you can build the xenolinux kernel. The default configuration should work fine for most people (use 'make oldconfig') but you can customise using one of the other config tools if you want. # cd /xeno-1.2.bk/xenolinux-2.4.24 # ARCH=xen make oldconfig { or menuconfig, or xconfig, or config } # ARCH=xen make dep # ARCH=xen make bzImage Assuming the build works, you'll end up with /xeno-1.2.bk/xenolinux-2.4.24/arch/xen/boot/xenolinux.gz. This is the gzip version of XenoLinux kernel image. Build the Domain Control Tools ============================== Under '/xeno-1.2.bk/tools', there are three sub-directories: 'balloon', 'xc' and 'misc', each containing a group of tools. You can enter any of the four sub-directories and type 'make' to compile the corresponding group of tools. Or you can type 'make' under '/xeno-1.2.bk/tools' to compile all the tools. In order to compile the control-interface library in 'xc' you must have zlib and development headers installed. Also you will need at least Python v2.2. 'make install' in the tools directory will place executables and libraries in /usr/bin and /usr/lib. You will need to be root to do this! As noted earlier, 'make dist' installs files to a local 'install' directory just outside the BK repository. These files will then need to be installed manually onto the server. The Example Scripts =================== The scripts in tools/examples/ are generally useful for administering a Xen-based system. You can install them by running 'make install' in that directory. The python scripts (*.py) are the main tools for controlling Xen domains. 'defaults' and 'democd' are example configuration files for starting new domains. 'xendomains' is a Sys-V style init script for starting and stopping Xen domains when the system boots / shuts down. These will be discussed below in more detail. Installation ============================== First: # cp /xen-1.2.bk/xen/xen.gz /boot/xen.gz # cp /xen-1.2.bk/xenolinux-2.4.24/arch/xen/boot/xenolinux.gz /boot/xenolinux.gz Second, you must have 'GNU Grub' installed. Then you need to edit the Grub configuration file '/boot/grub/menu.lst'. A typical Grub menu option might look like: title Xen 1.2 / XenoLinux 2.4.24 kernel /boot/xen.gz dom0_mem=131072 com1=115200,8n1 noht module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0 The first line specifies which Xen image to use, and what command line arguments to pass to Xen. In this case we set the maximum amount of memory to allocate to domain0, and enable serial I/O at 115200 baud. We could also disable smp support (nosmp) or disable hyper-threading support (noht). If you have multiple network interface you can use ifname=ethXX to select which one to use. If your network card is unsupported, use ifname=dummy The second line specifies which XenoLinux image to use, and the standard linux command line arguments to pass to the kernel. In this case, we're configuring the root partition and stating that it should (initially) be mounted read-only (normal practice). The following is a list of command line arguments to pass to Xen: ignorebiostables Disable parsing of BIOS-supplied tables. This may help with some chipsets that aren't fully supported by Xen. If you specify this option then ACPI tables are also ignored, and SMP support is disabled. noreboot Don't reboot the machine automatically on errors. This is useful to catch debug output if you aren't catching console messages via the serial line. nosmp Disable SMP support. This option is implied by 'ignorebiostables'. noacpi Disable ACPI tables, which confuse Xen on some chipsets. This option is implied by 'ignorebiostables'. watchdog Enable NMI watchdog which can report certain failures. noht Disable Hyperthreading. ifname=ethXX Select which Ethernet interface to use. ifname=dummy Don't use any network interface. com1=,DPS[,,] com2=,DPS[,,] Xen supports up to two 16550-compatible serial ports. For example: 'com1=9600,8n1,0x408,5' maps COM1 to a 9600-baud port, 8 data bits, no parity, 1 stop bit, I/O port base 0x408, IRQ 5. If the I/O base and IRQ are standard (com1:0x3f8,4; com2:0x2f8,3) then they need not be specified. console= Specify the destination for Xen console I/O. This is a comma-separated list of, for example: vga: use VGA console and allow keyboard input com1: use serial port com1 com2H: use serial port com2. Transmitted chars will have the MSB set. Received chars must have MSB set. com2L: use serial port com2. Transmitted chars will have the MSB cleared. Received chars must have MSB cleared. The latter two examples allow a single port to be shared by two subsystems (eg. console and debugger). Sharing is controlled by MSB of each transmitted/received character. [NB. Default for this option is 'com1,tty'] dom0_mem=xxx Set the maximum amount of memory for domain0. tbuf_size=xxx Set the size of the per-cpu trace buffers, in pages (default 1). Note that the trace buffers are only enabled in debug builds. Most users can ignore this feature completely. Boot into Domain 0 ============================== Reboot your computer; After selecting the kernel to boot, stand back and watch Xen boot, closely followed by "domain 0" running the XenoLinux kernel. Depending on which root partition you have assigned to XenoLinux kernel in Grub configuration file, you can use the corresponding username / password to log in. Once logged in, it should look just like any regular linux box. All the usual tools and commands should work as per usual. Start New Domains ============================== You must be 'root' to start new domains. Make sure you have successfully configured at least one physical network interface. Then: # xen_nat_enable The xc_dom_create.py program is useful for starting Xen domains. You can specify configuration files using the -f switch on the command line. The default configuration is in /etc/xc/defaults. You can create custom versions of this to suit your local configuration. You can override the settings in a configuration file using command line arguments to xc_dom_create.py. However, you may find it simplest to create a separate configuration file for each domain you start. xc_dom_create.py will print the local TCP port to which you should connect to perform console I/O. A suitable console client is provided by the Python module xenctl.console_client: running this module from the command line with and parameters will start a terminal session. This module is also installed as /usr/bin/xencons, from a copy in tools/misc/xencons. An alternative to manually running a terminal client is to specify '-c' to xc_dom_create.py, or add 'auto_console=True' to the defaults file. This will cause xc_dom_create.py to automatically become teh console terminal after starting the domain. Boot-time output can be directed to this 'virtual console' by specifying 'console=xencons0' as a boot parameter. It is also possible to log in via the virtual console. To do this, you must run a mingetty on the virtual console, which you can achieve as follows: # mkdir -p /dev/xen # mknod /dev/xen/cons c 4 123 # echo "c:2345:respawn:/sbin/mingetty --noclear xen/cons" >>/etc/inittab If you wish to permit root logins via the virtual console then you must also add 'xen/cons' to the list of trusted ttys in /etc/securetty. Manage Running Domains ============================== You can see a list of existing domains with: # xc_dom_control.py list In order to stop a domain, you use: # xc_dom_control.py stop To shutdown a domain cleanly use: # xc_dom_control.py shutdown To destroy a domain immediately: # xc_dom_control.py destroy There are other more advanced options, including pinning domains to specific CPUs and saving / resuming domains to / from disk files. To get more information, run the tool without any arguments: # xc_dom_control.py There is more information available in the Xen README files, the VBD-HOWTO and the contributed FAQ / HOWTO documents on the web. Other Control Tasks using Python ================================ A Python module 'Xc' is installed as part of the tools-install process. This can be imported, and an 'xc object' instantiated, to provide access to privileged command operations: # import Xc # xc = Xc.new() # dir(xc) # help(xc.domain_create) In this way you can see that the class 'xc' contains useful documentation for you to consult. A further package of useful routines (xenctl) is also installed: # import xenctl.utils # help(xenctl.utils) You can use these modules to write your own custom scripts or you can customise the scripts supplied in the Xen distribution. Automatically start / stop domains at boot / shutdown ===================================================== A Sys-V style init script for RedHat systems is provided in tools/examples/xendomains. When you run 'make install' in that directory, it should be automatically copied to /etc/init.d/. You can then enable it using the chkconfig command, e.g.: # chkconfig --add xendomains By default, this will start the boot-time domains in runlevels 3, 4 and 5. To specify a domain is to start at boot-time, place its configuration file (or a link to it) under /etc/xc/auto/. The script will also stop ALL domains when the system is shut down, even domains that it did not start originally. You can also use the "service" command (part of the RedHat standard distribution) to run this script manually, e.g: # service xendomains start Starts all the domains with config files under /etc/xc/auto/. # service xendomains stop Shuts down ALL running Xen domains.