Friday 9 August 2013

Nested hypervisors

I wanted to try out some different hypervisors and rather than shelling out for additional hardware I thought I would see how well my current system copes with nested hypervisors.

Initial test with VMWare under KVM failed, it did not seem to like the CPU I had provided to the guest and no matter what options I chose it did not want to run, therefore that is on hold for now.

However, Citrix Xenserver appeared to install fine.

I want to add a guest to Xenserver, just to prove this is working, but when I created my original disk I gave it only the bare minimum for Xenserver which was 16Gb. So I have created a new raw disk image using qemu-img and attached this to the Xenserver VM.
This image is attached as a new SCSI disk (so it will be /dev/sdb).

I now need to create a new storage repository (SR) which Xenserver will use to store my guests.

First off I need to fine the id of my new disk (/dev/sdb) this can be done by looking at the symlinks in /dev/disk/by-id and finding the one pointing to /dev/sdb.

For my Xenserver vm
ls -l /dev/disk/by-id















I also need the uuid of the Xenserver host, to get this use
xe host-list

Now I can create the new SR using
xe sr-create content-type=user device-config:device=<disk id> host-uuid=<host-uuid> name-label=”Second Disk” shared=false type=lvm


Creating new SR
















Tip:- When entering the console from the xsconsole application, the font was pretty unreadable, I just ran setfont  with no options to load the default and this resolved the issue. To set the font permanently modify /etc/sysconfig/i18n.

Now I should be able to create a new VM on this new SR, but first I need to get the uid of this new SR using
xe sr-list

To create the new VM using an available template

xe vm-install template=<template-name> new-name-label=<vm-name> sr-uuid=<sr-uuid>
To list the available templates use
xe template-list


Tip:- With the xe command you can use tab completion to finish commands and uuids.

The new VM now shows up in the list of VMs
xe vm-list

I have a basic VM (with nothing in it), which I now need to start up and boot from some install media. As I want to boot from the install media I need to set the disk for my VM to not be bootable as I cannot have two boot devices.
First off I need to find out the uuid for virtual block device (VBD) which will contain my VM, this will be the device xvda in my OS and is therefore the first userdevice (0).


xe vbd-list vm-uuid=<vm_uuid> userdevice=0 params=uuid --minimal

Set this to not be bootable
xe vbd-param-set uuid=<root_disk_uuid> bootable=false
I have attached an iso of the install media for a linux distro as a cdrom/dvd drive to my Xenserver VM, I now need to attach the cdrom/dvd drive to the new VM which I will boot up on Xenserver.
To find out what cdrom/dvd devices are available use
xe cd-list
In my case this is labelled "SCSI 2:0:0:0", and to add this device as a virtual cdrom/dvd drive to my VM is use the following
xe vm-cd-add vm=<vm_name> cd-name="SCSI 2:0:0:0" device=3
This will attach the cdrom/dvd drive of the Xenserver host (in my case the iso I have attached to the cdrom/dvd drive of the Xenserver VM) as device 3 which in Linux will be xvdd.
I need to make the VM boot from the cdrom/dvd drive so I need to find the VBD and set it to be bootable
xe vbd-list vm-uuid=<vm_uuid> device=xvdd params=uuid --minimal
xe vbd-param-set uuid=<cddrive_uuid> bootable=true
Finally, before I start the VM I need to set the install-repository to be the cdrom
xe vm-param-set uuid=<vm_uuid> other-config:install-repository=cdrom
Once all the above has been completed successfully it is time to start the VM
xe vm-start uuid=<vm_uuid>
This should start the VM, to check this look at the power-state line in the output of
xe vm-list uuid=<vm_uuid>
I want to be able to interact with the installation program to install my OS onto my VM, normally if Xenserver was on physical server I could connect to it using Xencenter. However, it is on a virtual machine running under KVM, also that would require me to run Windows.
I could create another VM under KVM and connect to it from there but as this is runnign on my laptop it sounds like overkill.
Xenserver exports the display of the VM via VNC, again not very useful as I do not have a VNC client on the Xenserver host and I would have to forward ssh connections to it from outside the Xenserver VM. So I need a console connection to the console of my newly created VM, there are two commands on Xenserver that allow you to connect to the console of a VM, xenconsole and xl console.
For some reason the location of xenconsole is not by default in the PATH for root, it can be found in /usr/lib/xen/bin.
To use either of these tools I need to know the domain-id of the running VM I want to connect to, all running domains can be gotten using the command list_domains.
list_domains output

My VM is listed as having a domain-id of 4.
The domain-id is also available in the dom-id parameter for the VM and can bo obtained using
xe vm-list uuid=<vm_uuid> params=dom-id
I tried both of these tools and I was not able to get a usable connection to the console for my new VM.

**Update**
I later discovered that this could be due to xapi setting up vncterm which takes the text console and draws them as graphics to make them available to vnc clients.
There is an option which can be set before the VM is start which will then allow me to use xenconsole or xl console, this can be done with the following
xe vm-param-set uuid=<vm_uuid> other-config:disable_pv_vnc=true
This may impact tools that connect via vnc
To remove this option use
xe vm-param-remove uuid=<vm_uuid> param-name=other-config param-key=disable_pv_vnc
*****

I found out that the text consoles are also made available on certain ports. To find out which port I need to look up the console entry for that domain-id (as mentioned above) in xenstore.
xenstore-ls /local/domain/<domain-id>/console

console entry in xenstore

For some guests (depending on whether they HVM or PV) this entry might be /local/domain/<domain-id>/serial.
If I telnet to the "tc-port" I get access to my text console.
A few notes on this, I changed the tty settings using "stty sane" running telnet. I also set character mode within telnet. e.g.
stty sane
telnet localhost <tc-port>
ctrl-] mode character
Once the install has finished the cdrom/dvd should be ejected from the VM
xe vm-cd-eject vm=<name or uuid>