LFS - Configuring tool chain

Continuing on building my LFS virtual image.

I am using version 7.3 of the LFS documentation, they have helpfully provided a file to use with wget to download all required packages and a separate md5sum list to verify once they have downloaded.

Files have all been downloaded to directory sources on my new filesystem (created in http://paulsrandomcontent.blogspot.co.uk/2013/07/lfs-creating-disk-image.html).

A new directory (tools) for tools has also been created and a symlink /tools has been created pointing to this location, e.g.

ln -sv /mnt/LFS/tools /

Instructions suggest the creation of a new non-root user for building packages so as not to damage the host system, which is a very good idea. I also want to keep this separate from my normal user on my system just in case.

I have created the user lfs to work with.

The following is also suggested to remove stray settings from our build host and create a simple bash_profile
cat > ~/.bash_profile << "EOF"
exec env -i HOME=$HOME TERM=$TERM PS1='\u:\w\$ ' /bin/bash
EOF


and the following to create a simple bashrc (note the mount location of the new filesystem for $LFS)
cat > ~/.bashrc << "EOF"
set +h
umask 022
LFS=/mnt/lfs
LC_ALL=POSIX
LFS_TGT=$(uname -m)-lfs-linux-gnu
PATH=/tools/bin:/bin:/usr/bin
export LFS LC_ALL LFS_TGT PATH
EOF

First step is to build our host independent toolchain which will be used for building tools needed to build our new system, these will be installed under $LFS/tools on our new filesystem.

While building binutils for a 64bit system it is recommended to create a 64bit lib location (which will be used by other tools as well) using the following

case $(uname -m) in
x86_64) mkdir -v /tools/lib && ln -sv lib /tools/lib64 ;;
esac

The list below marks the order in which the software packages were installed, however it should be noted that certain critical packages are install a couple of times to build a known good set of tools on our filesystem, which we use to build other software.

Binutils is required by both gcc and glibc configuration so must be built first. Once gcc is built it is used to build glibc. This new glibc can be used when rebuilding binutils and gcc, these tools will then be referencing only our new filesystem and not the host system. Effectively we are bootstrapping our compiler.


Tools built in the following order
binutils
gcc (only C language support)
Linux API headers (from kernel source, so glibc can interface with features provided by Linux kernel)
glibc

tcl (for binutils, gcc and other test suites)
expect
dejagnu
check
ncurses
bash
bzip2
coreutils
diffutils
file
findutils
gawk
msg-fmt binary (from gettext package)
grep
gzip
m4
make
patch
perl5 (only some tools and libraries for now)
sed
tar
texinfo
xz

firefox keyword trick

I was wondering if you could add a proxy server as a new custom search to the address bar of firefox and while it is possible for me to write a few lines in a plugin to do this I found another easier way to do this with bookmarks and keywords.

Create a bookmark giving the full url of the proxy server and where you wish to pass the url place %s, give the bookmark a keyword, such as proxy.

Bookmark properties

To use this proxy bookmark, in the address bar type

proxy <url you want to view>

LFS - Creating Disk Image

Since I came across it a number of years ago I have always wanted to have a go at this but never found the time.

Now is the time, I am going to install LFS onto a virtual disk and run it under KVM. This post and others will be used to document the process and any issues encountered.

First off I need a virtual disk image to install onto, this can be created with qenu-img, e.g.

qemu-img create -f <fmt> <image filename> <size of disk>

I have gone with qcow2 format and a size of 10GB to start with. There are some other interesting options which I will look at later including encryption.

I now have a disk image (/tmp/LFS.img), but to make use of it I really need to partition it. So how do I partition my image file, I need to manipulate it so that I can run fdisk or parted on it. Fortunately I have found an application called qemu-nbd which exports the qemu disk image using NBD protocol, therefore I should be able to access the block device it creates.

To make use of network block devices I need to make sure the nbd module is loaded and if not insert it using modprobe -v nbd, this creates multiple nbd devices (/dev/nbd0-15). Using one of these devices we connect to the exported image using qmeu-nbd.

qemu-nbd -c /dev/nbd0 /tmp/LFS.img

This now means we can use our partitioning tools on /dev/nbd0 to manipulate our disk image.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I have created a 2GB swap partition and 8GB partition for the OS.

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x03db5ccc

     Device Boot      Start         End      Blocks   Id  System
/dev/nbd0p1            2048    16779263     8388608   83  Linux
/dev/nbd0p2        16779264    20971519     2096128   82  Linux swap / Solaris

To be able to use these partitions kernel needs to know about them, therefore I have used kpartx to add these to devicemapper and I now have the devices /dev/mapper/nbd0p1 & /dev/mapper/nbd0p2.
**update**
While this works and gives me access to the partitions I created on my disk image, I ran into a problem installing grub. Looking into this there is an easier way to make the partitions available. When loading the nbd module, if I pass the argument max_part=16 (this tells will create a maximum of 16 partitions for this block device, the default is 0, hence I did not see my partitions) device entries are created for my partitions so I can access them via /dev/nbd0p1 & /dev/nbd0p2 once I have run qemu-nbd command as above.

I have created a swap area on nbd0p2 using mkswap and created an ext3 filesystem on nbd0p1 using mke2fs -j .

There is a note on LFS documentation about making sure the version of e2fsprogs on the host system does not add any custom features which could cause problems with the final system.

# debugfs -R feature /dev/mapper/nbd0p1
debugfs 1.42.3 (14-May-2012)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file

This filesystem should be fine. If not e2fsprogs would need to be built from source and used instead.

Note for future reference, to disconnect this disk image, first we need to make sure that nothing is using it, then remove the devicemapper entries created by kpartx, then to disconnect the NBD using qemu-nbd. If the nbd module was loaded manually then remove it using modprobe -rv nbd.

To create my workspace I now need to mount the filesystem and activate the swap partition (swapon).

Now on to installing the OS.

Watching X events

I was watching a presentation on Weyland and I wondered about how to see X events as they happen, I seem to recall doing this many years ago supporting customers but have not had a reason to do it recently.

Looking into this, it should be possible with xev.

xev allows you to monitor an existing window, however I wanted to watch events when a program starts up, e.g. possibly diagnose a slow starting application which you suspect to be caused by X.

I believe this can be done by getting xev to monitor you root window (putting it in the background) and then launching the application that you wish to monitor, e.g.

xev -r &
<graphical application>

C and Python

Not really sure why I would ever want to do this but here is how to run python code from within C Code. Here is a test program that prints a string using python being called from C.

/*
 * Test program to run python code from C
 * Compile with gcc -Wall -I /usr/include/python2.7/ -lpython2.7 -o runpython runpython.c
 *
 */

#include <Python.h>

int main() {
  Py_Initialize();
  PyRun_SimpleString("print('print using python')");
  return 0;
}

Python Easter Eggs

There are some amusing Easter Eggs in Python.

There is the "Zen of Python" obtained by
import this

This prints out Python programming principles that have been rot13 encoded in the module.

While checking out pypy, I came across this one
import antigravity

This opens a web browser and opens http://xkcd.com/353/

Compiling Python code

I know that compiling python code to bytecode speeds loading of code, however I never really thought about how to do this until now.
There appears to be multiple ways to do this according to the documentation.

The first method runs the code and generate an .pyc file.
In the python interpreter just import your .py file, e.g.
>>> import myfile.py


The next method I have come across is to use the py_compile module, e.g
import py_compile

py_compile.compile("myfile.py")


The next method is to use the compiler package, note this is marked as deprecated since version 2.6 and has been removed from version 3, e.g.
import compiler

compiler.compileFile(myfile.py")

With multiple files this could become time consuming, therefore there is another module compileall, which will compile all files in a directory, e.g.

import compileall 

compileall.compile_dir("mylib")

In the above example it will not recompile exisiting .pyc files if the corresponding .py timestamp has not been updated (timestamp is stored in .pyc file). To force compilation add argument force=1 when calling compile_dir().

See documentation for more useful options.