############################### __ __ ____ ___ \ \/ /___ _ __ |___ \ / _ \ \ // _ \ '_ \ __) || | | | / \ __/ | | | / __/ | |_| | /_/\_\___|_| |_| |_____(_)___/ ############################### XenDemoCD 2.0 University of Cambridge Computer Laboratory 28 Aug 2004 http://www.cl.cam.ac.uk/netos/xen Welcome to the Xen Demo CD! Executive Summary ================= This CD is a standalone demo of the Xen Virtual Machine Monitor (VMM), Linux-2.4 and Linux-2.6 OS port (Xenlinux). It runs entirely off the CD, without requiring hard disk installation. This is achieved using a RAM disk to store mutable file system data while using the CD for everything else. The CD can also be used for installing Xen/Xenlinux to disk, and includes a source code snapshot along with all of the tools required to build it. Booting the CD ============== It should be possible to get Xen working with any relatively modern hardware supported by standard Linux. However, the version of XenLinux built for the DemoCD is fairly h/w specific. If you need other hardware, you'll have to configure and build your own xenlinux kernel. Xen does require an 'i686'-class CPU or newer, so won't work on 486's or plain Pentiums. We have compiled in drivers for the following hardware: CPU: Pentium Pro/II/III/IV/Xeon, Athlon (i.e. P6 or newer) SMP supported IDE: Intel PIIX chipset, others will be PIO only (slow) SCSI: Adaptec / Dell PERC Raid (aacraid), fusion MPT, megaraid, Adaptec aic7xxx Net: Recommended: Intel e1000, Broadcom BCM57xx (tg3), 3c905 (3c59x) Also supported: pcnet32, Intel e100, tulip Because of the demo CD's use of RAM disks, make sure you have plenty of RAM (256MB+). To try out the Demo, boot from CD (you may need to change your BIOS configuration to do this), then select one of the four boot options from the Grub menu: Xen / linux-2.4.27 Xen / linux-2.4.27 using cmdline IP configuration Xen / linux-2.4.27 in "safe mode" linux-2.4.22 The last option is a plain linux kernel that runs on the bare machine, and is included simply to help diagnose driver compatibility problems. The "safe mode" boot option might be useful if you're having problems getting Xen to work with your hardware, as it disables various features such as SMP, and enables some debugging. If you are going for a command line IP config, hit "e" at the grub menu, then edit the "ip=" parameters to reflect your setup e.g. "ip=:::::eth0:off". It shouldn't be necessary to set either the nfs server or hostname parameters. Alternatively, once Xenlinux has booted you can login and setup networking with 'dhclient' or 'ifconfig' and 'route' in the normal way. To make things easier for yourself, it's worth trying to arrange for an IP address which is the first in a sequential range of free IP addresses. It's useful to give each VM instance its own public IP address (though it is possible to do NAT or use private addresses), and the configuration files on the CD allocate IP addresses sequentially for subsequent domains unless told otherwise. After selecting the kernel to boot, stand back and watch Xen boot, closely followed by "domain 0" running the Xenlinux kernel. The boot messages can also sent to the serial line by specifying the baud rate on the Xen cmdline (e.g., 'com1=9600,8n1'); this can be very useful for debugging should anything important scroll off the screen. Xen's startup messages will look quite familiar as much of the hardware initialisation (SMP boot, apic setup) and device drivers are derived from Linux. If everything is well, you should see the linux rc scripts start a bunch of standard services including sshd. Login on the console or via ssh:: username: user root password: xendemo xendemo Once logged in, it should look just like any regular linux box. All the usual tools and commands should work as per usual. However, because of the poor random access performance of CD drives, the machine will feel very slugish, and you may run out of memory if you make significant modifications to the ramfs filesystem -- for the full experience, install a Xen and Xenlinux image on you hard drive :-) You can configure networking, either with 'dhclient' or manually via 'ifconfig' and 'route', remembering to edit /etc/resolv.conf if you want DNS to work. You can start an X server with 'startx'. It defaults to a conservative 1024x768, but you can edit the script for higher resoloutions. The CD contains a load of standard software. You should be able to start Apache, PostgreSQL, Mozilla etc in the normal way, but because everything is running off CD the performance will be very sluggish and you may run out of memory for the 'tmpfs' file system. You may wish to go ahead and install Xen/Xenlinux on your hard drive, either dropping Xen and the Xenlinux kernel down onto a pre-existing Linux distribution, or using the file systems from the CD (which are based on RH9). See the installation instructions later in this document. If your video card requires 'agpgart' then it unfortunately won't yet work with Xen, and you'll only be able to configure a VGA X server. We're working on a fix for this for the next release. If you want to browse the Xen / Xenlinux source, it's all located under /usr/local/src/xen-2.0.bk, complete with BitKeeper repository. We've also included source code and configuration information for the various benchmarks we used in the SOSP paper. Starting other domains ====================== The first thing you need to do is to start the "xend" control daemon with "xend start". You may wish to add an appropriate link to xend in you /etc/rcX.d directory e.g. "ln -sf ../init.d/xend S97xend" If you're not intending to configure the new domain with an IP address on your LAN, then you'll probably want to use NAT. The 'xen_nat_enable' installs a few useful iptables rules into domain0 to enable NAT. [NB: We plan to support RSIP in future] Xen has a management interface that can be manipulated from domain0 to create new domains, control their CPU, network and memory resource allocations, allocate IP addresses, grant access to disk partitions, and suspend/resume domains to files, etc. The management interface is implemented as a set of library functions (implemented in C) for which there are Python language bindings. We have developed a simple set of example python tools for manipulating the interface, with the intention that more sophisticated high-level management tools will be developed in due course. Within the source repository the tools live in tools/examples/ but are installed in /usr/local/bin/ on the CD. Starting a new domain is achieved using the command 'xm create' which allocates resources to a new domain, populates it with a kernel image (and optionally a ramdisk) and then starts it. It parses a configuration file written in the Python language, the default location of which is "/etc/xc/defaults", but this may be overridden with the "-f" option. For the Demo CD, the defaults file will cause domains to be created with ram-based root file systems, and mount their /usr partition from the CD, just like domain0. (If you are writing your own config file, the "example" script may be a better starting point) Variables can be initialised and passed into configuration files. Some of these may be compulsory, others optional. The 'defaults' file on the CD requires the 'ip' variable to be set to tell Xen what IP address(es) should be routed to this domain. Xen will route packets to the domain if they bear one of these addresses as a destination address, and will also ensure that packets sent from the domain contain one of the addresses as a source address (to prevent spoofing). If multiple IP addresses are to be assigned to a domain they can be listed in a comma separated list (with no whitespace). The 'mem' variable can be used to change the default memory allocation of 64MB. For example to start a domain with two IP addresses and 72MB: xm create ip=128.23.45.34,169.254.1.1mem=72 When invoked with the '-n' option 'xm create' will do a dry run and just print out what resources and configuration the domain will have e.g.: [root@xendemo]# xm create -n ip=commando-1.xeno,169.254.2.3 mem=100 Parsing config file 'defaults' VM image : "/boot/xenlinux.gz" VM ramdisk : "/boot/initrd.gz" VM memory (MB) : "100" VM IP address(es) : "128.232.38.51:169.254.2.3" VM block device(s) : "phy:cdrom,hdd,r" VM cmdline : "ip=128.232.38.51:169.254.1.0:128.232.32.1:255.255.240.0::eth0:off root=/dev/ram0 rw init=/linuxrc 4 LOCALIP=169.254.2.3" xm create will print the local TCP port to which you should connect to perform console I/O. A suitable console client is provided by the Python module xenctl.console_client: running this module from the command line with and parameters will start a terminal session. This module is also installed as /usr/bin/xencons, from a copy in tools/misc/xencons. An alternative to manually running a terminal client is to specify '-c' to xm create, or add 'auto_console=True' to the defaults file. This will cause 'xm create' to automatically become the console terminal after starting the domain. The 169.254.x.x network is special in that it is the 'link local' subnet, and is isolated from the external network and hence can only be used for communication between virtual machines. By convention, we usually give each domain a link local address. The startup scripts on the CD have been modified to accept a LINKLOCAL= parameter on the kernel command line and initialise an IP alias accordingly (see /etc/sysinit/network-scripts/ifcfg-eth0). Linux only allows one IP address to be specified on the kernel command line, so if you specify multiple IP addresses you'll need to configure the new Linux VM with the other addresses manually (using ifconfig) having logged in. If you inspect the 'defaults' config script you'll see that the new domain was started with a '4' on the kernel command line to tell 'init' to go to runlevel 4 rather than the default of 3 used by domain0. This is done simply to suppress a bunch of harmless error messages that would otherwise occur when the new (unprivileged) domain tried to access physical hardware resources to try setting the hwclock, system font, run gpm etc. After it's booted, you should be able to ssh into your new domain from domain0 using the link local 19.254.x.x address you assigned. If you assigned a further IP address you should be able to ssh in using that address too. If you ran the xen_enable_nat script, a bunch of port redirects have been installed to enable you to ssh in to other domains remotely even if you didn't assign an externally routeable address. To access the new virtual machine remotely, use: ssh -p2201 root@IP.address.Of.Domain0 # use 2202 for domain 2 etc. You can manipulate running domains using the xm tool. Invoking it without arguments prints some usage information. To see what domains are running, run 'xm list'. Using the tool you can change scheduling parameters, pause a domain, send it a shutdown request, or blow it away with the 'destroy' command. You can even suspend it to disk (but you probably won't have enough memory to do the latter if you're running off the demo CD). To find usage information for xm, run the script with no arguments or with the 'help' argument. To get help on a particular xm command, use 'xm cmdname help'. Troubleshooting Problems ======================== If you have problems booting Xen, there are a number of boot parameters that may be able to help diagnose problems: ignorebiostables Disable parsing of BIOS-supplied tables. This may help with some chipsets that aren't fully supported by Xen. If you specify this option then ACPI tables are also ignored, and SMP support is disabled. noreboot Don't reboot the machine automatically on errors. This is useful to catch debug output if you aren't catching console messages via the serial line. nosmp Disable SMP support. This option is implied by 'ignorebiostables'. noacpi Disable ACPI tables, which confuse Xen on some chipsets. This option is implied by 'ignorebiostables'. watchdog Enable NMI watchdog which can report certain failures. noht Disable Hyperthreading. badpage=[,]* Specify a list of pages not to be allocated for use because they contain bad bytes. For example, if your memory tester says that byte 0x12345678 is bad, you would place 'badpage=0x12345' on Xen's command line (i.e., the last three digits of the byte address are not included!). com1=,DPS[,,] com2=,DPS[,,] Xen supports up to two 16550-compatible serial ports. For example: 'com1=9600,8n1,0x408,5' maps COM1 to a 9600-baud port, 8 data bits, no parity, 1 stop bit, I/O port base 0x408, IRQ 5. If the I/O base and IRQ are standard (com1:0x3f8,4; com2:0x2f8,3) then they need not be specified. console= Specify the destination for Xen console I/O. This is a comma-separated list of, for example: vga: use VGA console and allow keyboard input com1: use serial port com1 com2H: use serial port com2. Transmitted chars will have the MSB set. Received chars must have MSB set. com2L: use serial port com2. Transmitted chars will have the MSB cleared. Received chars must have MSB cleared. The latter two examples allow a single port to be shared by two subsystems (eg. console and debugger). Sharing is controlled by MSB of each transmitted/received character. [NB. Default for this option is 'com1,vga'] conswitch= Specify how to switch serial-console input between Xen and DOM0. The required sequence is CTRL- pressed three times. Specifying '`' disables switching. The specifies whether Xen should auto-switch input to DOM0 when it boots -- if it is 'x' then auto-switching is disabled. Any other value, or omitting the character, enables auto-switching. [NB. Default for this option is 'a'] nmi= Specify what to do with an NMI parity or I/O error. 'nmi=fatal': Xen prints a diagnostic and then hangs. 'nmi=dom0': Inform DOM0 of the NMI. 'nmi=ignore': Ignore the NMI. [NB. Default is 'dom0' ('fatal' for debug builds).] dom0_mem=xxx Set the initial amount of memory for domain0. pdb=xxx Enable the pervasive debugger. See docs/pdb.txt xxx defines how the gdb stub will communicate: com1 use com1 com1H use com1 (with high bit set) com2 use on com2 com2H use com2 (with high bit set) It's probably a good idea to join the Xen developer's mailing list on Sourceforge: http://lists.sourceforge.net/lists/listinfo/xen-devel About The Xen Demo CD ===================== The purpose of the Demo CD is to distribute a snapshot of Xen's source, and simultaneously provide a convenient means for enabling people to get experience playing with Xen without needing to install it on their hard drive. If you decide to install Xen/Xenlinux you can do so simply by following the installation instructions below -- which essentially involves copying the contents of the CD on to a suitably formated disk partition, and then installing or updating the Grub bootloader. This is a bootable CD that loads Xen, and then a Linux 2.4.27 OS image ported to run on Xen. The CD contains a copy of a file system based on the RedHat 9 distribution that is able to run directly off the CD ("live ISO"), using a "tmpfs" RAM-based file system for root (/etc /var etc). Changes you make to the tmpfs will obviously not be persistent across reboots! Because of the use of a RAM-based file system for root, you'll need plenty of memory to run this CD -- something like 96MB per VM. This is not a restriction of Xen : once you've installed Xen, Xenlinux and the file system images on your hard drive you'll find you can boot VMs in just a few MBs. The CD contains a snapshot of the Xen and Xenlinux code base that we believe to be pretty stable, but lacks some of the features that are currently still work in progress e.g. OS suspend/resume to disk, and various memory management enhancements to provide fast inter-OS communication and sharing of memory pages between OSs. We'll release newer snapshots as required, making use of a BitKeeper repository hosted on http://xen.bkbits.net (follow instructions from the project home page). We're obviously grateful to receive any bug fixes or other code you can contribute. We suggest you join the xen-devel@lists.sourceforge.net mailing list. Installing from the CD ====================== If you're installing Xen/Xenlinux onto an existing linux file system distribution, just copy the Xen VMM (/boot/image.gz) and Xenlinux kernels (/boot/xenlinux.gz), then modify the Grub config (/boot/grub/menu.lst or /boot/grub/grub.conf) on the target system. It should work on pretty much any distribution. Xen is a "multiboot" standard boot image. Despite being a 'standard', few boot loaders actually support it. The only two we know of are Grub, and our modified version of linux kexec (for booting off a XenoBoot CD -- PlanetLab have adopted the same boot CD approach). If you need to install grub on your system, you can do so either by building the Grub source tree /usr/local/src/grub-0.93-iso9660-splashimage or by copying over all the files in /boot/grub and then running /sbin/grub and following the usual grub documentation. You'll then need to edit the Grub config file. A typical Grub menu option might look like: title Xen 2.0 / Xenlinux 2.4.27 kernel /boot/xen.gz dom0_mem=131072 com1=115200 noht watchdog module /boot/vmlinuz-2.4.27-xen0 root=/dev/sda4 ro The first line specifies which Xen image to use, and what command line arguments to pass to Xen. In this case we set the maximum amount of memory to allocate to domain0, and enable serial I/O on COM1 at 115200 baud. We could also disable smp support (nosmp) or disable hyper-threading support (noht). The second line specifies which xenlinux image to use, and the standard linux command line arguments to pass to the kernel. In this case, we're configuring the root partition and stating that it should initially be mounted read-only (normal practice). If we were booting with an initial ram disk (initrd), then this would require a second "module" line. Installing the Xen tools and source =================================== The tools and source live in the /usr/local/src/xen-2.0.bk directory on the CD (and may also be downloaded from the project downloads page). You'll need to copy them to some mutable storage before using them. If you have the BitKeeper BK tools installed you can check the repository is up to date by cd'ing into the xeno-2.0.bk directory and typing 'bk pull' (assuming you have an Internet connection). You can rebuild Xen, the tools and XenLinux by typing 'make world'. You can install them to the standard directories with 'make install', or into the ./install subtree with 'make dist'. Modifying xc_mycreatelinuxdom1.py ================================= xc_mycreatelinuxdom1.py.py can be used to set the new kernel's command line, and hence determine what it uses as a root file system, etc. Although the default is to boot in the same manner that domain0 did (using the RAM-based file system for root and the CD for /usr) it's possible to configure any of the following possibilities, for example: * initrd=/boot/initrd init=/linuxrc boot using an initial ram disk, executing /linuxrc (as per this CD) * root=/dev/hda3 ro boot using a standard hard disk partition as root !!! remember to grant access in createlinuxdom.py. * root=/dev/xvda1 ro boot using a pre-configured 'virtual block device' that will be attached to a virtual disk that previously has had a file system installed on it. * root=/dev/nfs nfsroot=/path/on/server ip= Boot using an NFS mounted root file system. This could be from a remote NFS server, or from an NFS server running in another domain. The latter is rather a useful option. A typical setup might be to allocate a standard disk partition for each domain and populate it with files. To save space, having a shared read-only usr partition might make sense. Block devices should only be shared between domains in a read-only fashion otherwise the linux kernels will obviously get very confused as the file system structure may change underneath them (having the same partition mounted rw twice is a sure fire way to cause irreparable damage)! If you want read-write sharing, export the directory to other domains via NFS from domain0. Installing the file systems from the CD ======================================= If you haven't got an existing Linux installation onto which you can just drop down the Xen and Xenlinux images, then the file systems on the CD provide a quick way of doing an install. However, you would be better off in the long run doing a proper install of your preferred distro and installing Xen onto that, rather than just doing the hack described below: Choose one or two partitions, depending on whether you want a separate /usr or not. Make file systems on it/them e.g.: mkfs -t ext3 /dev/hda3 [or mkfs -t ext2 /dev/hda3 && tune2fs -j /dev/hda3 if using an old version of mkfs] Next, mount the file system(s) e.g.: mkdir /mnt/root && mount /dev/hda3 /mnt/root [mkdir /mnt/usr && mount /dev/hda4 /mnt/usr] To install the root file system, simply untar /usr/XenDemoCD/root.tar.gz: cd /mnt/root && tar -zxpf /usr/XenDemoCD/root.tar.gz You'll need to edit /mnt/root/etc/fstab to reflect your file system configuration. Changing the password file (etc/shadow) is probably a good idea too. To install the usr file system, copy the file system from CD on /usr, though leaving out the "XenDemoCD" and "boot" directories: cd /usr && cp -a X11R6 etc java libexec root src bin dict kerberos local sbin tmp doc include lib man share /mnt/usr If you intend to boot off these file systems (i.e. use them for domain 0), then you probably want to copy the /usr/boot directory on the cd over the top of the current symlink to /boot on your root filesystem (after deleting the current symlink) i.e.: cd /mnt/root ; rm boot ; cp -a /usr/boot . The XenDemoCD directory is only useful if you want to build your own version of the XenDemoCD (see below). Debugging ========= Xen has a set of debugging features that can be useful to try and figure out what's going on. Hit 'h' on the serial line (if you specified a baud rate on the Xen command line) or ScrollLock-h on the keyboard to get a list of supported commands. If you have a crash you'll likely get a crash dump containing an EIP (PC) which, along with an 'objdump -d image', can be useful in figuring out what's happened. Debug a Xenlinux image just as you would any other Linux kernel. We supply a handy debug terminal program which you can find in /usr/local/src/xen-2.0.bk/tools/misc/miniterm/ This should be built and executed on another machine that is connected via a null modem cable. Documentation is included. Alternatively, if the Xen machine is connected to a serial-port server then we supply a dumb TCP terminal client: 'tools/xenctl/lib/console_client.py ' Installing Xen / Xenlinux on a RedHat distribution =================================================== When using Xen / Xenlinux on a standard Linux distribution there are a couple of things to watch out for: The first Linux VM that is started when Xen boots start (Domain 0) is given direct access to the graphics card, so it may use it as a console. Other domains don't have ttyN consoles, so attempts to run a 'mingetty' against them will fail, generating periodic warning messages from 'init' about services respawning too fast. They should work for domain0 just fine. IMPORTANT: To prevent warning messages when running RH9 you'll need to remove ttyN from /etc/inittab for domains>0. Due to a bug in the RH9 /etc/rc.sysinit script #'ing the lines out of /etc/inittab won't work as it ignores the '#' and tries to access them anyway. Every Xenlinux instance owns a bidirectional 'virtual console'. The device node to which this console is attached can be configured by specifying 'xencons=' on the OS command line: 'xencons=off' --> disable virtual console 'xencons=tty' --> attach console to /dev/tty1 (tty0 at boot-time) 'xencons=ttyS' --> attach console to /dev/ttyS0 The default is to attach to /dev/tty1, and also to create dummy devices for /dev/tty2-63 to avoid warnings from many standard distro startup scripts. The exception is domain 0, which by default attaches to /dev/ttyS0. Note that, because domains>0 don't have any privileged access at all, certain commands in the default boot sequence will fail e.g. attempts to update the hwclock, change the console font, update the keytable map, start apmd (power management), or gpm (mouse cursor). Either ignore the errors, or remove them from the startup scripts. Deleting the following links are a good start: S24pcmcia S09isdn S17keytable S26apmd S85gpm If you want to use a single root file system that works cleanly for domain0 and domains>0, one trick is to use different 'init' run levels. For example, on the Xen Demo CD we use run level 3 for domain 0, and run level 4 for domains>0. This enables different startup scripts to be run in depending on the run level number passed on the kernel command line. Xenlinux kernels can be built to use runtime loadable modules just like normal linux kernels. Modules should be installed under /lib/modules in the normal way. If there's some kernel feature that hasn't been built into our default kernel, there's a pretty good change that if its a non-hardware related option you'll just be able to enable it and rebuild. If its not on the xconfig menu, hack the arch/xen/config.in to put the menu back in. If you're going to use the link local 169.254.1.x addresses to communicate between VMs, there are a couple of other issues to watch out for. RH9 appears to have a bug where by default it configures the loopback interface with a 169.254 address, which stops it working properly on eth0 for communicating with other domains. This utterly daft RH9 behaviour can be stopped by appending "NOZEROCONF=yes" to /etc/sysconfig/networking-scripts/ifcfg-lo If you're going to use NFS root files systems mounted either from an external server or from domain0 there are a couple of other gotchas. The default /etc/sysconfig/iptables rules block NFS, so part way through the boot sequence things will suddenly go dead. If you're planning on having a separate NFS /usr partition, the RH9 boot scripts don't make life easy, as they attempt to mount NFS file systems way to late in the boot process. The easiest way I found to do this was to have a '/linuxrc' script run ahead of /sbin/init that mounts /usr: #!/bin/bash /sbin/ipconfig lo 127.0.0.1 /sbin/portmap /bin/mount /usr exec /sbin/init "$@" <>/dev/console 2>&1 The one slight complication with the above is that /sbib/portmap is dynamically linked against /usr/lib/libwrap.so.0 Since this is in /usr, it won't work. I solved this by copying the file (and link) below the /usr mount point, and just let the file be 'covered' when the mount happens. In some installations, where a shared read-only /usr is being used, it may be desirable to move other large directories over into the read-only /usr. For example, on the XenDemoCD we replace /bin /lib and /sbin with links into /usr/root/bin /usr/root/lib and /usr/root/sbin respectively. This creates other problems for running the /linuxrc script, requiring bash, portmap, mount, ifconfig, and a handful of other shared libraries to be copied below the mount point. I guess I should have written a little statically linked C program... Description of how the XenDemoCD boots ====================================== 1. Grub is used to load Xen, a Xenlinux kernel, and an initrd (initial ram disk). [The source of the version of Grub used is in /usr/local/src] 2. the init=/linuxrc command line causes linux to execute /linuxrc in the initrd. 3. the /linuxrc file attempts to mount the CD by trying the likely locations : /dev/hd[abcd]. 4. it then creates a 'tmpfs' file system and untars the 'XenDemoCD/root.tar.gz' file into the tmpfs. This contains hopefully all the files that need to be mutable (this would be so much easier if Linux supported 'stacked' or union file systems...) 5. Next, /linuxrc uses the pivot_root call to change the root file system to the tmpfs, with the CD mounted as /usr. 6. It then invokes /sbin/init in the tmpfs and the boot proceeds normally. Building your own version of the XenDemoCD ========================================== The 'live ISO' version of RedHat is based heavily on Peter Anvin's SuperRescue CD version 2.1.2 and J. McDaniel's Plan-B: http://www.kernel.org/pub/dist/superrescue/v2/ http://projectplanb.org/ Since Xen uses a "multiboot" image format, it was necessary to change the bootloader from isolinux to Grub0.93 with Leonid Lisovskiy's grub.0.93-iso9660.patch The Xen Demo CD contains all of the build scripts that were used to create it, so it is possible to 'unpack' the current iso, modifiy it, then build a new iso. The procedure for doing so is as follows: First, mount either the CD, or the iso image of the CD: mount /dev/cdrom /mnt/cdrom or: mount -o loop xendemo-1.0.iso /mnt/cdrom cd to the directory you want to 'unpack' the iso into then run the unpack script: cd /local/xendemocd /mnt/cdrom/XenDemoCD/unpack-iso.sh The result is a 'build' directory containing the file system tree under the 'root' directory. e.g. /local/xendemocd/build/root To add or remove rpms, its possible to use 'rpm' with the --root option to set the path. For more complex changes, it easiest to boot a machine using using the tree via NFS root. Before doing this, you'll need to edit fstab to comment out the seperate mount of /usr. One thing to watch out for: as part of the CD build process, the contents of the 'rootpatch' tree gets copied over the existing 'root' tree replacing various files. The intention of the rootpatch tree is to contain the files that have been modified from the original RH distribution (e.g. various /etc files). This was done to make it easier to upgrade to newer RH versions in the future. The downside of this is that if you edit an existing file in the root tree you should check that you don't also need to propagate the change to the rootpatch tree to avoid it being overwritten. Once you've made the changes and want to build a new iso, here's the procedure: cd /local/xendemocd/build echo '' > Builder ./make.sh put_your_version_id_here >../buildlog 2>&1 This process can take 30 mins even on a fast machine, but you should eventually end up with an iso image in the build directory. Notes: root - the root of the file system heirarchy as presented to the running system rootpatch - contains files that have been modified from the standard RH, and copied over the root tree as part of the build procedure. irtree - the file system tree that will go into the initrd (initial ram disk) work - a working directory used in the build process usr - this should really be in 'work' as its created as part of the build process. It contains the 'immutable' files that will be served from the CD rather than the tmpfs containing the contents of root.tar.gz. Some files that are normally in /etc or /var that are large and actually unlikely to need changing have been moved into /usr/root and replaced with links. Ian Pratt 9 Sep 2003