LFS - Importing disk image into KVM and booting

I thought would probably be the least challenging part of getting my LFS setup working under KVM as once everything is installed on the filesystem it should just be a matter of booting it.

However, when I tried the steps below I realised that grub had not been installed correctly and it dropped my into the grub rescue environment from which I did not seem to be able to fix the problem. Details on how I did fix it below.

First of all, details on how to use the image with KVM. The following steps can be done in graphical management utilities but it is just as easy from the commandline using virtinstall.

Run virtinstall and tell it to import rather than to run an installer, I gave the VM a basic set of requirements namely just 2GB of RAM.

virt-install --name LFS --ram 2048 --disk <location of image file> --import

This created my VM configuration and started the VM (if it had been working correctly the OS would have started) and launched virt-viewer so I could see the console.


As noted earlier this failed to boot first time and dropped me into the grub rescue prompt, therefore I had to find a different way to get grub installed onto the MBR of my disk image.

Lots of posts on forums talk about mounting the disk using losetup, this is fine for raw images but I was using qcow2 format and had made it available via NBD. Other posts talk about installing the grub version from the host system onto the disk image, this would have worked, however why go to the effort of building a custom system if you are going to just copy files from the host system.

Whatever I tried inside a chroot onto my filesystem failed to work (even though grub-install returned success), so I looked for a way to run the grub utilities from the filesystem in my disk image by directly from my host system.

Fortunately it seems that grub-install is setup for running in just this manner with a number of optional arguments to override directory paths and locations of utilities. With my diskimage made available with qemu-nbd and mounted under /media/LFS I used the following command to install grub to the MBR of my disk image.

/media/LFS/usr/sbin/grub-install --boot-directory=/media/LFS/boot/ --directory=/media/LFS/usr/lib/grub/i386-pc --modules="ext2 part_msdos" --grub-mkimage=/media/LFS/usr/bin/grub-mkimage --grub-probe=/media/LFS/usr/sbin/grub-probe --grub-mkrelpath=/media/LFS/usr/bin/grub-mkrelpath /dev/nbd0 --debug

A quick explanation of the options (for more details check grub documentation)
--boot-directory  location of "/boot", where grub files will be installed
--directory  location to copy boot files from
--modules  which modules to load
--grub-*  locations of various grub utilities
--debug  undocumented (according to grub-install script) option so I can see exactly what commands it is running to make sure it does not damage grub on my host system

Now when I startup this VM under KVM it boots fine.

Grub screen

Login prompt - before compiling kernel to include nic driver

LFS - Kernel & grub install

I have a filesystem and a set of packages installed, now I need to get the kernel installed so I can boot this system.

The LFS instructions note that due to recent changes in udev the following kernel options need to be enabled
Device Drivers --->
    Generic Driver Options --->
         Maintain a devtmpfs filesystem to mount at /dev

**update**
Remember to compile in (or as module) your nic driver, doh!
 

As I am doing this in a virtual image I need to be careful I am specifying the right devices when it comes to installing and configuring grub.

Once the kernel has been compiled and the built kernel moved to /boot, grub needs to be installed in the MBR for this disk image. This is done with the command
grub-install <device>

As the disk image is mounted via nbd, the device is /dev/nbd0.

The first  time I tried to do this it failed, grub-install is a script which calls grub-probe and this claimed that the location where /boot/grub is, would not be accessible to grub.
I realised that I had been mounting the filesystem wrong and if I pass the parameter max_part=16 when I load the nbd module, devices are created for my partitions and I can mount them without having to use kpartx or partx.

When I did this, grub-install ran successfully within a chroot from the filesystem on my disk image.

**update**
This did not produce a bootable image, which I did not discover until I tried to boot it in KVM.

During one of my failed attempts to resolve this problem, I had created a custom devicemap entry for grub and when trying to do grub-install I got an error about blocklists similar to

Attempting to install GRUB to a partition disk or to a partition.  This is a BAD idea.
Embedding is not possible.  GRUB can only be installed in this setup by using blocklists. 
However, blocklists are UNRELIABLE and their use is discouraged.

There was a suggestion in the grub manual that I could safely use the option --force in this case, however this was not needed once I remove the custom devicemap.

LFS - Creating Disk Image

Since I came across it a number of years ago I have always wanted to have a go at this but never found the time.

Now is the time, I am going to install LFS onto a virtual disk and run it under KVM. This post and others will be used to document the process and any issues encountered.

First off I need a virtual disk image to install onto, this can be created with qenu-img, e.g.

qemu-img create -f <fmt> <image filename> <size of disk>

I have gone with qcow2 format and a size of 10GB to start with. There are some other interesting options which I will look at later including encryption.

I now have a disk image (/tmp/LFS.img), but to make use of it I really need to partition it. So how do I partition my image file, I need to manipulate it so that I can run fdisk or parted on it. Fortunately I have found an application called qemu-nbd which exports the qemu disk image using NBD protocol, therefore I should be able to access the block device it creates.

To make use of network block devices I need to make sure the nbd module is loaded and if not insert it using modprobe -v nbd, this creates multiple nbd devices (/dev/nbd0-15). Using one of these devices we connect to the exported image using qmeu-nbd.

qemu-nbd -c /dev/nbd0 /tmp/LFS.img

This now means we can use our partitioning tools on /dev/nbd0 to manipulate our disk image.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I have created a 2GB swap partition and 8GB partition for the OS.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x03db5ccc

     Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1            2048    16779263     8388608   83  Linux
/dev/nbd0p2        16779264    20971519     2096128   82  Linux swap / Solaris

To be able to use these partitions kernel needs to know about them, therefore I have used kpartx to add these to devicemapper and I now have the devices /dev/mapper/nbd0p1 & /dev/mapper/nbd0p2.
**update**
While this works and gives me access to the partitions I created on my disk image, I ran into a problem installing grub. Looking into this there is an easier way to make the partitions available. When loading the nbd module, if I pass the argument max_part=16 (this tells will create a maximum of 16 partitions for this block device, the default is 0, hence I did not see my partitions) device entries are created for my partitions so I can access them via /dev/nbd0p1 & /dev/nbd0p2 once I have run qemu-nbd command as above.

I have created a swap area on nbd0p2 using mkswap and created an ext3 filesystem on nbd0p1 using mke2fs -j .

There is a note on LFS documentation about making sure the version of e2fsprogs on the host system does not add any custom features which could cause problems with the final system.

# debugfs -R feature /dev/mapper/nbd0p1
debugfs 1.42.3 (14-May-2012)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file

This filesystem should be fine. If not e2fsprogs would need to be built from source and used instead.

Note for future reference, to disconnect this disk image, first we need to make sure that nothing is using it, then remove the devicemapper entries created by kpartx, then to disconnect the NBD using qemu-nbd. If the nbd module was loaded manually then remove it using modprobe -rv nbd.

To create my workspace I now need to mount the filesystem and activate the swap partition (swapon).

Now on to installing the OS.