KVM and multipathing

I was curious how to add multipathing to a KVM guest, turns out it is really very simple.

Create two (or more) virtual disks, then depending on whether you run virt-manager or the command line it is just a matter of specifying the same serial number.

virt-manager serial number


qemu-kvm

qemu-kvm ... -device ... -drive if=none,id=sda,file=disk1.img,serial=0001 ... -device -drive if=none,id=sdb,file=disk1.img,serial=0001 ...

Result for libvirt's xml code for KVM guest

 ...
  <devices>
   ...
   <disk type='block' device='disk'>
    ...
    <serial>0001</serial>
    ...
   </disk>
   <disk type='block' device='disk'>
    ...
    <serial>0001</serial>
    ...
   </disk>

OpenIndiana

I have not used Solaris in some time so I thought it would be interesting to get OpenSolaris installed under kvm to test out. However, this has stopped being produced, but there is a fork called OpenIndiana.

I downloaded the Live dvd for desktops and booted that up under kvm, I have given the vm a qcow2 format disk to install to.

Once the OS boots up there is an option to install to disk, this will take you through a few install questions and then install to disk.

Haiku

I am not about to break into poetry so don't panic.

A number of years back when I first started looking at Unix/Linux I came across several other Operating Systems which I would like to try out, one of these was BeOs (time permitting I may explore some more).

However, BeOs has all but died and was a proprietary OS anyway, but I did come across some people developing an opensource implementation called Haiku

I have downloaded it and installed it in a virtual image under KVM.
I created a 3Gb raw image and have booted from the ISO image using the following command to bring up a 32bit vm and given it 512Mb

qemu-system-i386 -m 512 -boot d -cdrom <image file> -hda haiku.img

I just passed it a raw image, so using the disk utility in the installer I need to create a partition and a location to create as Be File system. Follow the instructions in the installer and it is installed.
Next time when booting up the following command is used

qemu-system-i386 -m 512 -boot c -hda haiku.img

Nested hypervisors

I wanted to try out some different hypervisors and rather than shelling out for additional hardware I thought I would see how well my current system copes with nested hypervisors.

Initial test with VMWare under KVM failed, it did not seem to like the CPU I had provided to the guest and no matter what options I chose it did not want to run, therefore that is on hold for now.

However, Citrix Xenserver appeared to install fine.

I want to add a guest to Xenserver, just to prove this is working, but when I created my original disk I gave it only the bare minimum for Xenserver which was 16Gb. So I have created a new raw disk image using qemu-img and attached this to the Xenserver VM.
This image is attached as a new SCSI disk (so it will be /dev/sdb).

I now need to create a new storage repository (SR) which Xenserver will use to store my guests.

First off I need to fine the id of my new disk (/dev/sdb) this can be done by looking at the symlinks in /dev/disk/by-id and finding the one pointing to /dev/sdb.

For my Xenserver vm

ls -l /dev/disk/by-id

I also need the uuid of the Xenserver host, to get this use
xe host-list

Now I can create the new SR using
xe sr-create content-type=user device-config:device=<disk id> host-uuid=<host-uuid> name-label=”Second Disk” shared=false type=lvm

Creating new SR

Tip:- When entering the console from the xsconsole application, the font was pretty unreadable, I just ran setfont  with no options to load the default and this resolved the issue. To set the font permanently modify /etc/sysconfig/i18n.

Now I should be able to create a new VM on this new SR, but first I need to get the uid of this new SR using
xe sr-list

To create the new VM using an available template

xe vm-install template=<template-name> new-name-label=<vm-name> sr-uuid=<sr-uuid>

To list the available templates use

xe template-list

Tip:- With the xe command you can use tab completion to finish commands and uuids.

The new VM now shows up in the list of VMs
xe vm-list

I have a basic VM (with nothing in it), which I now need to start up and boot from some install media. As I want to boot from the install media I need to set the disk for my VM to not be bootable as I cannot have two boot devices.
First off I need to find out the uuid for virtual block device (VBD) which will contain my VM, this will be the device xvda in my OS and is therefore the first userdevice (0).

xe vbd-list vm-uuid=<vm_uuid> userdevice=0 params=uuid --minimal

Set this to not be bootable

xe vbd-param-set uuid=<root_disk_uuid> bootable=false

I have attached an iso of the install media for a linux distro as a cdrom/dvd drive to my Xenserver VM, I now need to attach the cdrom/dvd drive to the new VM which I will boot up on Xenserver.

To find out what cdrom/dvd devices are available use

xe cd-list

In my case this is labelled "SCSI 2:0:0:0", and to add this device as a virtual cdrom/dvd drive to my VM is use the following

xe vm-cd-add vm=<vm_name> cd-name="SCSI 2:0:0:0" device=3

This will attach the cdrom/dvd drive of the Xenserver host (in my case the iso I have attached to the cdrom/dvd drive of the Xenserver VM) as device 3 which in Linux will be xvdd.

I need to make the VM boot from the cdrom/dvd drive so I need to find the VBD and set it to be bootable

xe vbd-list vm-uuid=<vm_uuid> device=xvdd params=uuid --minimal

xe vbd-param-set uuid=<cddrive_uuid> bootable=true

Finally, before I start the VM I need to set the install-repository to be the cdrom

xe vm-param-set uuid=<vm_uuid> other-config:install-repository=cdrom

Once all the above has been completed successfully it is time to start the VM

xe vm-start uuid=<vm_uuid>

This should start the VM, to check this look at the power-state line in the output of

xe vm-list uuid=<vm_uuid>

I want to be able to interact with the installation program to install my OS onto my VM, normally if Xenserver was on physical server I could connect to it using Xencenter. However, it is on a virtual machine running under KVM, also that would require me to run Windows.

I could create another VM under KVM and connect to it from there but as this is runnign on my laptop it sounds like overkill.

Xenserver exports the display of the VM via VNC, again not very useful as I do not have a VNC client on the Xenserver host and I would have to forward ssh connections to it from outside the Xenserver VM. So I need a console connection to the console of my newly created VM, there are two commands on Xenserver that allow you to connect to the console of a VM, xenconsole and xl console.

For some reason the location of xenconsole is not by default in the PATH for root, it can be found in /usr/lib/xen/bin.

To use either of these tools I need to know the domain-id of the running VM I want to connect to, all running domains can be gotten using the command list_domains.

list_domains output

My VM is listed as having a domain-id of 4.

The domain-id is also available in the dom-id parameter for the VM and can bo obtained using

xe vm-list uuid=<vm_uuid> params=dom-id

I tried both of these tools and I was not able to get a usable connection to the console for my new VM.

**Update**

I later discovered that this could be due to xapi setting up vncterm which takes the text console and draws them as graphics to make them available to vnc clients.

There is an option which can be set before the VM is start which will then allow me to use xenconsole or xl console, this can be done with the following

xe vm-param-set uuid=<vm_uuid> other-config:disable_pv_vnc=true

This may impact tools that connect via vnc. 

To remove this option use

xe vm-param-remove uuid=<vm_uuid> param-name=other-config param-key=disable_pv_vnc

*****

I found out that the text consoles are also made available on certain ports. To find out which port I need to look up the console entry for that domain-id (as mentioned above) in xenstore.

xenstore-ls /local/domain/<domain-id>/console

console entry in xenstore

For some guests (depending on whether they HVM or PV) this entry might be /local/domain/<domain-id>/serial.

If I telnet to the "tc-port" I get access to my text console.

A few notes on this, I changed the tty settings using "stty sane" running telnet. I also set character mode within telnet. e.g.

stty sane

telnet localhost <tc-port>

ctrl-] mode character

Once the install has finished the cdrom/dvd should be ejected from the VM

xe vm-cd-eject vm=<name or uuid>

Running Android in KVM... because I can

I have been toying with the idea of playing around with android and took a quick look at downloading the sourcecode. I decided to hold off as it seemed like a lot of time would need to be spent on this and I had a few other projects I wanted to look at first.

However I found the android x86 port and so I decided to get this running in KVM.
Why?
Because I can.

I downloaded the iso for version 4.2 (I had a few problems with 4.3, however this might be solved in the same way I solved booting 4.2).

I created a new qcow2 qemu image (the original instructions I was looking at suggested using this just to store data and emulate an sd card to save data as the iso is also a live image), however I decided to install it to this disk image.

Using qemu-kvm I booted the iso image and attached the qemu disk I had created. The live image did not seem to boot up at all. If I selected the debug option, this just dropped me into a command prompt. Therefore I decided to install the os to the qemu disk image using the option on the boot menu of the live image.

I also used virt-install to import this disk image which would make it easier to work with.
It still failed to boot correctly, so I checked the VM settings and checked what hardware had been assigned to the VM. I gave it a 32bit cpu as it had defaulted to the same cpu as my hypervisor and because it seemed to be complaining about some graphic functions I switched the video card type from Cirrus to VMVGA (VMWare). With these settings it booted up fine and I got the familiar android home screen.
Sound also seemed not to work so I switched over to using the SPICE protocol instead of VNC, which gave me sound but was very choppy (probably due to running in an emulated environment on my laptop which is not too powerful).

I installed a few apps to prove that this was functioning and challenged my wife to a game of Word Fued.

grub boot screen

lock screen

home screen

I have installed version 4.3-test over the top and it seems to work, although Google Play Services periodically crashed during setup.

As this is not on a phone/tablet rotation can be a bit of a headache (I noticed this with WordFued it wanted to start in landscape), therefore I downloaded a rotation locker app to keep the display in potrait and I also turned of auto rotation in android.

LFS - Creating Disk Image

Since I came across it a number of years ago I have always wanted to have a go at this but never found the time.

Now is the time, I am going to install LFS onto a virtual disk and run it under KVM. This post and others will be used to document the process and any issues encountered.

First off I need a virtual disk image to install onto, this can be created with qenu-img, e.g.

qemu-img create -f <fmt> <image filename> <size of disk>

I have gone with qcow2 format and a size of 10GB to start with. There are some other interesting options which I will look at later including encryption.

I now have a disk image (/tmp/LFS.img), but to make use of it I really need to partition it. So how do I partition my image file, I need to manipulate it so that I can run fdisk or parted on it. Fortunately I have found an application called qemu-nbd which exports the qemu disk image using NBD protocol, therefore I should be able to access the block device it creates.

To make use of network block devices I need to make sure the nbd module is loaded and if not insert it using modprobe -v nbd, this creates multiple nbd devices (/dev/nbd0-15). Using one of these devices we connect to the exported image using qmeu-nbd.

qemu-nbd -c /dev/nbd0 /tmp/LFS.img

This now means we can use our partitioning tools on /dev/nbd0 to manipulate our disk image.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I have created a 2GB swap partition and 8GB partition for the OS.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x03db5ccc

     Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1            2048    16779263     8388608   83  Linux
/dev/nbd0p2        16779264    20971519     2096128   82  Linux swap / Solaris

To be able to use these partitions kernel needs to know about them, therefore I have used kpartx to add these to devicemapper and I now have the devices /dev/mapper/nbd0p1 & /dev/mapper/nbd0p2.
**update**
While this works and gives me access to the partitions I created on my disk image, I ran into a problem installing grub. Looking into this there is an easier way to make the partitions available. When loading the nbd module, if I pass the argument max_part=16 (this tells will create a maximum of 16 partitions for this block device, the default is 0, hence I did not see my partitions) device entries are created for my partitions so I can access them via /dev/nbd0p1 & /dev/nbd0p2 once I have run qemu-nbd command as above.

I have created a swap area on nbd0p2 using mkswap and created an ext3 filesystem on nbd0p1 using mke2fs -j .

There is a note on LFS documentation about making sure the version of e2fsprogs on the host system does not add any custom features which could cause problems with the final system.

# debugfs -R feature /dev/mapper/nbd0p1
debugfs 1.42.3 (14-May-2012)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file

This filesystem should be fine. If not e2fsprogs would need to be built from source and used instead.

Note for future reference, to disconnect this disk image, first we need to make sure that nothing is using it, then remove the devicemapper entries created by kpartx, then to disconnect the NBD using qemu-nbd. If the nbd module was loaded manually then remove it using modprobe -rv nbd.

To create my workspace I now need to mount the filesystem and activate the swap partition (swapon).

Now on to installing the OS.