path: root/docs/Xen-HOWTO.txt

###########################################
Xen HOWTO

University of Cambridge Computer Laboratory

http://www.cl.cam.ac.uk/netos/xen
#############################


Get Xen Source Code
=============================

The public master BK repository for the 1.2 release lives at:
'bk://xen.bkbits.net/xeno-1.2.bk'

To fetch a local copy, first download the BitKeeper tools at:
http://www.bitmover.com/download with username 'bitkeeper' and
password 'get bitkeeper'.

Then install the tools and then run:
# bk clone bk://xen.bkbits.net/xeno-1.2.bk

Under your current directory, a new directory named 'xeno-1.2.bk' has
been created, which contains all the necessary source codes for the
Xen hypervisor and Linux guest OSes.

To get newest changes to the repository, run
# cd xeno-1.2.bk
# bk pull


Build Xen
=============================

Hint: To see how to build Xen and all the control tools, inspect the
tools/misc/xen-clone script in the BK repository. This script can be
used to clone the repository and perform a full build.

To build Xen manually:

# cd xeno-1.2.bk/xen
# make clean
# make

This will (should) produce a file called 'xen' in the current
directory.  This is the ELF 32-bit LSB executable file of Xen.  You
can also find a gzip version, named 'xen.gz'.

To install the built files on your server under /usr, type 'make
install' at the root of the BK repository. You will need to be root to
do this!

Hint: There is also a 'make dist' rule which copies built files to an
install directory just outside the BK repo; if this suits your setup,
go for it.


Build Linux as a Xen guest OS
==============================

This is a little more involved since the repository only contains a
"sparse" tree -- this is essentially an 'overlay' on a standard linux
kernel source tree. It contains only those files currently 'in play'
which are either modified versions of files in the vanilla linux tree,
or brand new files specific to the Xen port.

So, first you need a vanilla linux-2.4.24 tree, which is located at:
http://www.kernel.org/pub/linux/kernel/v2.4

Then:
  # mv linux-2.4.24.tar.gz /xeno-1.2.bk
  # cd /xeno-1.2.bk
  # tar -zxvf linux-2.4.24.tar.gz

You'll find a new directory 'linux-2.4.24' which contains all
the vanilla Linux 2.4.24 kernel source codes.

Hint: You should choose the vanilla linux kernel tree that has the
same version as the "sparse" tree.

Next, you need to 'overlay' this sparse tree on the full vanilla Linux
kernel tree:

  # cd /xeno-1.2.bk/xenolinux-2.4.24-sparse
  # ./mkbuildtree ../linux-2.4.24

Finally, rename the buildtree since it is now a 'xenolinux' buildtree. 

  # cd /xeno-1.2.bk
  # mv linux-2.4.24 xenolinux-2.4.24

Now that the buildtree is there, you can build the xenolinux kernel.
The default configuration should work fine for most people (use 'make
oldconfig') but you can customise using one of the other config tools
if you want.

  # cd /xeno-1.2.bk/xenolinux-2.4.24
  # ARCH=xen make oldconfig   { or menuconfig, or xconfig, or config }  
  # ARCH=xen make dep
  # ARCH=xen make bzImage

Assuming the build works, you'll end up with
/xeno-1.2.bk/xenolinux-2.4.24/arch/xen/boot/xenolinux.gz. This is the
gzip version of XenoLinux kernel image.


Build the Domain Control Tools
==============================

Under '/xeno-1.2.bk/tools', there are three sub-directories:
'balloon', 'xc' and 'misc', each containing
a group of tools. You can enter any of the four sub-directories
and type 'make' to compile the corresponding group of tools.
Or you can type 'make' under '/xeno-1.2.bk/tools' to compile
all the tools.

In order to compile the control-interface library in 'xc' you must
have zlib and development headers installed. Also you will need at
least Python v2.2. 

'make install' in the tools directory will place executables and
libraries in /usr/bin and /usr/lib. You will need to be root to do this!

As noted earlier, 'make dist' installs files to a local 'install'
directory just outside the BK repository. These files will then need
to be installed manually onto the server.

The Example Scripts
===================

The scripts in tools/examples/ are generally useful for
administering a Xen-based system.  You can install them by running
'make install' in that directory.

The python scripts (*.py) are the main tools for controlling
Xen domains.

'defaults' and 'democd' are example configuration files for starting
new domains.

'xendomains' is a Sys-V style init script for starting and stopping
Xen domains when the system boots / shuts down.

These will be discussed below in more detail.


Installation
==============================

First:
# cp /xen-1.2.bk/xen/xen.gz /boot/xen.gz
# cp /xen-1.2.bk/xenolinux-2.4.24/arch/xen/boot/xenolinux.gz /boot/xenolinux.gz

Second, you must have 'GNU Grub' installed. Then you need to edit
the Grub configuration file '/boot/grub/menu.lst'.

A typical Grub menu option might look like:

title Xen 1.2 / XenoLinux 2.4.24
        kernel /boot/xen.gz dom0_mem=131072 com1=115200,8n1 noht
        module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0

The first line specifies which Xen image to use, and what command line
arguments to pass to Xen. In this case we set the maximum amount of
memory to allocate to domain0, and enable serial I/O at 115200 baud.
We could also disable smp support (nosmp) or disable hyper-threading
support (noht). If you have multiple network interface you can use
ifname=ethXX to select which one to use. If your network card is
unsupported, use ifname=dummy

The second line specifies which XenoLinux image to use, and the
standard linux command line arguments to pass to the kernel. In this
case, we're configuring the root partition and stating that it should
(initially) be mounted read-only (normal practice). 

The following is a list of command line arguments to pass to Xen:

 ignorebiostables Disable parsing of BIOS-supplied tables. This may
                  help with some chipsets that aren't fully supported
                  by Xen. If you specify this option then ACPI tables are
                  also ignored, and SMP support is disabled.

 noreboot         Don't reboot the machine automatically on errors.
                  This is useful to catch debug output if you aren't
                  catching console messages via the serial line.

 nosmp            Disable SMP support.
                  This option is implied by 'ignorebiostables'.

 noacpi           Disable ACPI tables, which confuse Xen on some chipsets.
                  This option is implied by 'ignorebiostables'.

 watchdog         Enable NMI watchdog which can report certain failures.

 noht             Disable Hyperthreading.

 ifname=ethXX     Select which Ethernet interface to use.

 ifname=dummy     Don't use any network interface.

 com1=<baud>,DPS[,<io_base>,<irq>]
 com2=<baud>,DPS[,<io_base>,<irq>]
                  Xen supports up to two 16550-compatible serial ports.
                  For example: 'com1=9600,8n1,0x408,5' maps COM1 to a
                  9600-baud port, 8 data bits, no parity, 1 stop bit,
                  I/O port base 0x408, IRQ 5.
                  If the I/O base and IRQ are standard (com1:0x3f8,4;
                  com2:0x2f8,3) then they need not be specified.

 console=<specifier list>
                  Specify the destination for Xen console I/O.
                  This is a comma-separated list of, for example:
                   vga:  use VGA console and allow keyboard input
                   com1: use serial port com1
                   com2H: use serial port com2. Transmitted chars will
                          have the MSB set. Received chars must have
                          MSB set.
                   com2L: use serial port com2. Transmitted chars will
                          have the MSB cleared. Received chars must
                          have MSB cleared.
                  The latter two examples allow a single port to be
                  shared by two subsystems (eg. console and
                  debugger). Sharing is controlled by MSB of each
                  transmitted/received character.
 [NB. Default for this option is 'com1,tty']

 dom0_mem=xxx     Set the maximum amount of memory for domain0.

 tbuf_size=xxx    Set the size of the per-cpu trace buffers, in pages
                  (default 1).  Note that the trace buffers are only
                  enabled in debug builds.  Most users can ignore
                  this feature completely.

 sched=xxx        Select the CPU scheduler Xen should use.  The current
                  possibilities are 'bvt', 'atropos' and 'rrobin'.  The
                  default is 'bvt'.  For more information see
                  Sched-HOWTO.txt.

Boot into Domain 0
==============================

Reboot your computer; After selecting the kernel to boot, stand back
and watch Xen boot, closely followed by "domain 0" running the
XenoLinux kernel.  Depending on which root partition you have assigned
to XenoLinux kernel in Grub configuration file, you can use the
corresponding username / password to log in.

Once logged in, it should look just like any regular linux box. All
the usual tools and commands should work as per usual.


Start New Domains
==============================

You must be 'root' to start new domains.

Make sure you have successfully configured at least one
physical network interface. Then:

# xen_nat_enable

The xc_dom_create.py program is useful for starting Xen domains.
You can specify configuration files using the -f switch on the command
line.  The default configuration is in /etc/xc/defaults.  You can
create custom versions of this to suit your local configuration.

You can override the settings in a configuration file using command
line arguments to xc_dom_create.py.  However, you may find it simplest
to create a separate configuration file for each domain you start.

xc_dom_create.py will print the local TCP port to which you should
connect to perform console I/O. A suitable console client is provided
by the Python module xenctl.console_client: running this module from
the command line with <host> and <port> parameters will start a
terminal session. This module is also installed as /usr/bin/xencons,
from a copy in tools/misc/xencons.  An alternative to manually running
a terminal client is to specify '-c' to xc_dom_create.py, or add
'auto_console=True' to the defaults file. This will cause
xc_dom_create.py to automatically become the console terminal after
starting the domain.

Boot-time output will be directed to this console by default, because
the console name is tty0. It is also possible to log in via the
virtual console --- once again, your normal startup scripts will work
as normal (e.g., by running mingetty on tty1-7).  The device node to
which the virtual console is attached can be configured by specifying
'xencons=' on the OS command line: 
 'xencons=off' --> disable virtual console
 'xencons=tty' --> attach console to /dev/tty1 (tty0 at boot-time)
 'xencons=ttyS' --> attach console to /dev/ttyS0


Manage Running Domains
==============================

You can see a list of existing domains with:
# xc_dom_control.py list

In order to stop a domain, you use:
# xc_dom_control.py stop <domain_id>

To shutdown a domain cleanly use:
# xc_dom_control.py shutdown <domain_id>

To destroy a domain immediately:
# xc_dom_control.py destroy <domain_id>

There are other more advanced options, including pinning domains to
specific CPUs and saving / resuming domains to / from disk files.  To
get more information, run the tool without any arguments:
# xc_dom_control.py

There is more information available in the Xen README files, the
VBD-HOWTO and the contributed FAQ / HOWTO documents on the web.


Other Control Tasks using Python
================================

A Python module 'Xc' is installed as part of the tools-install
process. This can be imported, and an 'xc object' instantiated, to
provide access to privileged command operations:

# import Xc
# xc = Xc.new()
# dir(xc)
# help(xc.domain_create)

In this way you can see that the class 'xc' contains useful
documentation for you to consult.

A further package of useful routines (xenctl) is also installed:

# import xenctl.utils
# help(xenctl.utils)

You can use these modules to write your own custom scripts or you can
customise the scripts supplied in the Xen distribution.


Automatically start / stop domains at boot / shutdown
=====================================================

A Sys-V style init script for RedHat systems is provided in
tools/examples/xendomains.  When you run 'make install' in that
directory, it should be automatically copied to /etc/init.d/.  You can
then enable it using the chkconfig command, e.g.:

# chkconfig --add xendomains

By default, this will start the boot-time domains in runlevels 3, 4
and 5.  To specify a domain is to start at boot-time, place its
configuration file (or a link to it) under /etc/xc/auto/.

The script will also stop ALL domains when the system is shut down,
even domains that it did not start originally.

You can also use the "service" command (part of the RedHat standard
distribution) to run this script manually, e.g:

# service xendomains start

Starts all the domains with config files under /etc/xc/auto/.

# service xendomains stop

Shuts down ALL running Xen domains.