LFS - Importing disk image into KVM and booting

I thought would probably be the least challenging part of getting my LFS setup working under KVM as once everything is installed on the filesystem it should just be a matter of booting it.

However, when I tried the steps below I realised that grub had not been installed correctly and it dropped my into the grub rescue environment from which I did not seem to be able to fix the problem. Details on how I did fix it below.

First of all, details on how to use the image with KVM. The following steps can be done in graphical management utilities but it is just as easy from the commandline using virtinstall.

Run virtinstall and tell it to import rather than to run an installer, I gave the VM a basic set of requirements namely just 2GB of RAM.

virt-install --name LFS --ram 2048 --disk <location of image file> --import

This created my VM configuration and started the VM (if it had been working correctly the OS would have started) and launched virt-viewer so I could see the console.


As noted earlier this failed to boot first time and dropped me into the grub rescue prompt, therefore I had to find a different way to get grub installed onto the MBR of my disk image.

Lots of posts on forums talk about mounting the disk using losetup, this is fine for raw images but I was using qcow2 format and had made it available via NBD. Other posts talk about installing the grub version from the host system onto the disk image, this would have worked, however why go to the effort of building a custom system if you are going to just copy files from the host system.

Whatever I tried inside a chroot onto my filesystem failed to work (even though grub-install returned success), so I looked for a way to run the grub utilities from the filesystem in my disk image by directly from my host system.

Fortunately it seems that grub-install is setup for running in just this manner with a number of optional arguments to override directory paths and locations of utilities. With my diskimage made available with qemu-nbd and mounted under /media/LFS I used the following command to install grub to the MBR of my disk image.

/media/LFS/usr/sbin/grub-install --boot-directory=/media/LFS/boot/ --directory=/media/LFS/usr/lib/grub/i386-pc --modules="ext2 part_msdos" --grub-mkimage=/media/LFS/usr/bin/grub-mkimage --grub-probe=/media/LFS/usr/sbin/grub-probe --grub-mkrelpath=/media/LFS/usr/bin/grub-mkrelpath /dev/nbd0 --debug

A quick explanation of the options (for more details check grub documentation)
--boot-directory  location of "/boot", where grub files will be installed
--directory  location to copy boot files from
--modules  which modules to load
--grub-*  locations of various grub utilities
--debug  undocumented (according to grub-install script) option so I can see exactly what commands it is running to make sure it does not damage grub on my host system

Now when I startup this VM under KVM it boots fine.

Grub screen

Login prompt - before compiling kernel to include nic driver

LFS - Kernel & grub install

I have a filesystem and a set of packages installed, now I need to get the kernel installed so I can boot this system.

The LFS instructions note that due to recent changes in udev the following kernel options need to be enabled
Device Drivers --->
    Generic Driver Options --->
         Maintain a devtmpfs filesystem to mount at /dev

**update**
Remember to compile in (or as module) your nic driver, doh!
 

As I am doing this in a virtual image I need to be careful I am specifying the right devices when it comes to installing and configuring grub.

Once the kernel has been compiled and the built kernel moved to /boot, grub needs to be installed in the MBR for this disk image. This is done with the command
grub-install <device>

As the disk image is mounted via nbd, the device is /dev/nbd0.

The first  time I tried to do this it failed, grub-install is a script which calls grub-probe and this claimed that the location where /boot/grub is, would not be accessible to grub.
I realised that I had been mounting the filesystem wrong and if I pass the parameter max_part=16 when I load the nbd module, devices are created for my partitions and I can mount them without having to use kpartx or partx.

When I did this, grub-install ran successfully within a chroot from the filesystem on my disk image.

**update**
This did not produce a bootable image, which I did not discover until I tried to boot it in KVM.

During one of my failed attempts to resolve this problem, I had created a custom devicemap entry for grub and when trying to do grub-install I got an error about blocklists similar to

Attempting to install GRUB to a partition disk or to a partition.  This is a BAD idea.
Embedding is not possible.  GRUB can only be installed in this setup by using blocklists. 
However, blocklists are UNRELIABLE and their use is discouraged.

There was a suggestion in the grub manual that I could safely use the option --force in this case, however this was not needed once I remove the custom devicemap.

LFS - Installing software

I have a file system (http://paulsrandomcontent.blogspot.co.uk/2013/07/lfs-creating-disk-image.html) and I have a set of tools (http://paulsrandomcontent.blogspot.co.uk/2013/07/lfs-configuring-tool-chain.html) to build my system, so now is time time to start the build of the new system.

First off I need to create directories for the kernel's virtual filesystems (/proc, /dev, /sys) on my filesystem and also some initial device nodes (/dev/console and /dev/null). To find out the major/minor numbers for certain devices, check Documentation/devices.txt in the kernel source tree.

Entries in /dev need to be created so we can mount the virtual filesystems required, this can be achieved by creating a bind mount from my host system. Once we have this, the virtual filesystems /dev/pts, /dev/shm, /proc, and /sys can be mounted.

The install of the system now needs to happen within a chroot environment on the filesystem I have created which contains all the tools that were installed in a previous entry. A directory structure based on Filesystem Hierachy Standard (FHS) is created into which software will be installed.

As there are some hardwired paths for certain basic utilities in some programs, symlinks in /bin are created to point to our pre-built toolchain.

A basic /etc/passwd and /etc/group is created.

Some programs require log files to have already been created before they try to write to them (such as agetty, login), therefore we need we need to create log files such as /var/log/lastlog, /var/log/btmp, /var/log/wtmp.

The kernel headers are extracted for use by userspace programs and in particular glibc when it is built.

The following packages are installed (watch out for programs shipped as part of multiple packages)
man pages (not sure why first, check??)
glibc (configure timezone and setup /etc/ld.so.conf, gcc was also modified to ensure that no references to /tools exists)
zlib
file
binutils
gmp
mpfr
mpc
gcc (rebuilt so it now refers to installed glibc)
sed
bzip2
pkg-config
ncurses
util-linux
ps-misc
procps-ng
e2fsprogs
shadow (I installed cracklib and enabled support in shadow)
coreutils
iana-etc
m4
bison
grep
readline
bash
libtool
gdbm
inetutils
perl
autoconf
automake
diffutils
gawk
findutils
flex
gettext
groff
xz
grub
less
gzip
iproute2
kbd
kmod
libpipeline
make
man-DB
patch
sysklogd
sysvinit
tar
texinfo
systemd(udev)
vim

Some of these packages took quite some time to compile on my host system.


I ran into a problem when compiling glibc, it seemed to cause gnome-terminal to segfault (once this build is complete I will have to see if I can reproduce the problem and find out what is going on).




LFS - Configuring tool chain

Continuing on building my LFS virtual image.

I am using version 7.3 of the LFS documentation, they have helpfully provided a file to use with wget to download all required packages and a separate md5sum list to verify once they have downloaded.

Files have all been downloaded to directory sources on my new filesystem (created in http://paulsrandomcontent.blogspot.co.uk/2013/07/lfs-creating-disk-image.html).

A new directory (tools) for tools has also been created and a symlink /tools has been created pointing to this location, e.g.

ln -sv /mnt/LFS/tools /

Instructions suggest the creation of a new non-root user for building packages so as not to damage the host system, which is a very good idea. I also want to keep this separate from my normal user on my system just in case.

I have created the user lfs to work with.

The following is also suggested to remove stray settings from our build host and create a simple bash_profile
cat > ~/.bash_profile << "EOF"
exec env -i HOME=$HOME TERM=$TERM PS1='\u:\w\$ ' /bin/bash
EOF


and the following to create a simple bashrc (note the mount location of the new filesystem for $LFS)
cat > ~/.bashrc << "EOF"
set +h
umask 022
LFS=/mnt/lfs
LC_ALL=POSIX
LFS_TGT=$(uname -m)-lfs-linux-gnu
PATH=/tools/bin:/bin:/usr/bin
export LFS LC_ALL LFS_TGT PATH
EOF

First step is to build our host independent toolchain which will be used for building tools needed to build our new system, these will be installed under $LFS/tools on our new filesystem.

While building binutils for a 64bit system it is recommended to create a 64bit lib location (which will be used by other tools as well) using the following

case $(uname -m) in
x86_64) mkdir -v /tools/lib && ln -sv lib /tools/lib64 ;;
esac

The list below marks the order in which the software packages were installed, however it should be noted that certain critical packages are install a couple of times to build a known good set of tools on our filesystem, which we use to build other software.

Binutils is required by both gcc and glibc configuration so must be built first. Once gcc is built it is used to build glibc. This new glibc can be used when rebuilding binutils and gcc, these tools will then be referencing only our new filesystem and not the host system. Effectively we are bootstrapping our compiler.


Tools built in the following order
binutils
gcc (only C language support)
Linux API headers (from kernel source, so glibc can interface with features provided by Linux kernel)
glibc

tcl (for binutils, gcc and other test suites)
expect
dejagnu
check
ncurses
bash
bzip2
coreutils
diffutils
file
findutils
gawk
msg-fmt binary (from gettext package)
grep
gzip
m4
make
patch
perl5 (only some tools and libraries for now)
sed
tar
texinfo
xz

LFS - Creating Disk Image

Since I came across it a number of years ago I have always wanted to have a go at this but never found the time.

Now is the time, I am going to install LFS onto a virtual disk and run it under KVM. This post and others will be used to document the process and any issues encountered.

First off I need a virtual disk image to install onto, this can be created with qenu-img, e.g.

qemu-img create -f <fmt> <image filename> <size of disk>

I have gone with qcow2 format and a size of 10GB to start with. There are some other interesting options which I will look at later including encryption.

I now have a disk image (/tmp/LFS.img), but to make use of it I really need to partition it. So how do I partition my image file, I need to manipulate it so that I can run fdisk or parted on it. Fortunately I have found an application called qemu-nbd which exports the qemu disk image using NBD protocol, therefore I should be able to access the block device it creates.

To make use of network block devices I need to make sure the nbd module is loaded and if not insert it using modprobe -v nbd, this creates multiple nbd devices (/dev/nbd0-15). Using one of these devices we connect to the exported image using qmeu-nbd.

qemu-nbd -c /dev/nbd0 /tmp/LFS.img

This now means we can use our partitioning tools on /dev/nbd0 to manipulate our disk image.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I have created a 2GB swap partition and 8GB partition for the OS.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x03db5ccc

     Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1            2048    16779263     8388608   83  Linux
/dev/nbd0p2        16779264    20971519     2096128   82  Linux swap / Solaris

To be able to use these partitions kernel needs to know about them, therefore I have used kpartx to add these to devicemapper and I now have the devices /dev/mapper/nbd0p1 & /dev/mapper/nbd0p2.
**update**
While this works and gives me access to the partitions I created on my disk image, I ran into a problem installing grub. Looking into this there is an easier way to make the partitions available. When loading the nbd module, if I pass the argument max_part=16 (this tells will create a maximum of 16 partitions for this block device, the default is 0, hence I did not see my partitions) device entries are created for my partitions so I can access them via /dev/nbd0p1 & /dev/nbd0p2 once I have run qemu-nbd command as above.

I have created a swap area on nbd0p2 using mkswap and created an ext3 filesystem on nbd0p1 using mke2fs -j .

There is a note on LFS documentation about making sure the version of e2fsprogs on the host system does not add any custom features which could cause problems with the final system.

# debugfs -R feature /dev/mapper/nbd0p1
debugfs 1.42.3 (14-May-2012)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file

This filesystem should be fine. If not e2fsprogs would need to be built from source and used instead.

Note for future reference, to disconnect this disk image, first we need to make sure that nothing is using it, then remove the devicemapper entries created by kpartx, then to disconnect the NBD using qemu-nbd. If the nbd module was loaded manually then remove it using modprobe -rv nbd.

To create my workspace I now need to mount the filesystem and activate the swap partition (swapon).

Now on to installing the OS.