gnuplot changing font

I was following some instructions on using gnuplot, however the instructions for changing the font used in the graphs was wrong. Therefore I had to figure out how to achieve this.
I was looking to output some graphs to png images, so I added the following to the top of my command file used to draw the graphs

set terminal png enhanced font "<full path to font>"
set output "<filename>"

There is also an environment variable GDFONTPATH which can be set to directories containing font files, if this is used the above lines can be changed to

set terminal png enhanced font "<font name>"
set output "<filename>"

ioctl decoding

I was interested to know how to decode an ioctl hex string, I worked through the example described in the kernel documentation at /ioctl/ioctl-decoding.txt
Then I tested it out on an ioctl code listed in an error message in a server log, here is my working (this is for an x86_64 server, other architectures may vary)

ioctl code in hex is
cc770002

Which in binary is
11001100011101110000000000000010

First two bits gives us the macro used
11
this according to the document is _IOWR (Read/Write)

The next 14 bits give the size of the arguments
00110001110111
which in decimal is 3191

The next 8 bits are an ascii character, which in my example gives NULL (could well be an issue and why the message appears in the logs for that application)
00000000

The final 8 bits gives the function number, in my example this is
00000010
which in decimal is 2

If I had the sourcecode of the application I could go and look up this function using the character and the function number. In the example above, looks like a bug in the application is generating the wrong character and the ioctl call will fail.

bc setting defaults

I have never really thought much about this, but for years whenever I start bc I always set the number of decimal points (scale) by issuing

scale=<number>

To automatically set this (and other settings) place them into a file (I called mine .bcrc) and set the environment variable BC_ENV_ARGS to point to this file.

BC_ENV_ARGS=$HOME/.bcrc

To set this variable up for every shell add to your shells start up files.

xchat username registration

I am sure I did this many years ago, but failed to remember exactly the steps I took.

I had the need to register my nick with a nick registration service on a particular IRC channel. I have tended to use xchat as it is available and simple to use.
To save me some time and so I don't have to remember to do it every time I thought I would automate this login.

In xchat's network list select the irc server you are connecting to and click edit, which will bring up a screen similar to the following

Edit Network dialog box

In the "Connect command" box I added the command

load -e <filename>

where filename contained the commands I needed to login with the nick registration service (minus the leading "/" is would normally use with interactive commands). The commands were similar to

msg <registration service> login <nick> <password>

Wireless Bridging

I have just started a new job and have discovered that while I have an office, I cannot put everything on wireless, especially the new office IP phone I have been given. Therefore, I decided to combine to projects together, flashing of an old Linksys router with dd-wrt and setting up additional networking in my office so I can connect my office phone.
This seemed the simplest option rather than stringing cables all round the place, another good alternative could have been ethernet over powerline adapters, but no technical tweaking involved.

I followed the dd-wrt instructions for installation on my old linksys WRT54G router, making sure to follow all instructions to the letter. When it came to using tftp to install the new firmware on the system after 99 tries it failed, so I went back to an earlier step where in the Management Mode window a custom image created at the beginning of the process is uploaded. This must not have taken first time as the tftp worked first time.
Once dd-wrt was up and installed I had to connect it to my existing wireless AP, this was straight forward in the wireless settings I select client bridge and give it the details of my existing wireless network (ensure things match exactly). I also made sure that SIP firewall was off and no dhcp server was running on the router.
Once changes had been made and rooter rebooted (as necessary), I connected via ethernet cable a laptop and it successfully got a dhcp lease from the main access point and I was able to browse the web.

The Dark Mod

Having seen the announcement The Dark Mod version 2.0 is completely standalone and was inspired by the Thief series of games which I enjoyed playing (and I am looking forward to the new one that is coming out soon), I decided to give it a try.

There were a few issues to getting it up and running.

Following the instructions I downloaded the updater and set it running, which downloaded all the needed files. However, when I tried to run it, it kept crashing out.
I had hoped this was just due to it being a 32bit binary and I was running it on 64bit. I installed additional 32bit libraries (and their dependencies) that were missing libboost-filesystem, DevIL, mesa-dri-drivers.

This got me a little further, I got a blank screen and could hear music and sound effects. When looking at a copy of the startup messages (passed to tee and dumped to a file), I saw messages relating to not being able to load various font and image files (including ones related to the menu), hence the black screen. I double checked that there were no addition files by re-running the updater, everything was downloaded.
Checking the startup messages and checking the forum, quickly lead me to this page which appeared to have exactly the same symptoms and was suggesting that this was related to S3TC (S3 Texture Compression) support.
There was a suggestion that disabling this in the config might work, however this did not work for me.
Was I missing the 32bit library that contained this support? I was, however as I was testing this on Fedora, Fedora does not ship this particular library as it is patented by S3. I could have gone and downloaded this from a 3rd party repo, but I am not happy at installing lots of extra packages from random locations.
Luckily I came across another solution when using mesa drivers >  version 9 and this is to set the following variable before launching thedarkmod

force_s3tc_enable=true

This worked fine and allowed me to launch the game and start playing through the training mission.

zenity

I have been messing round with zenity to pop up dialog boxes.

One particular script was to remind me to take a break every so often, when it popped up a dialog box it has a random fortune in it. This was fairly easy to get going but there were a couple of strange behaviours in zenity that I had to figure out.

First one was random sizes of dialog box. When using the --info selection of dialog box often the windows it created were off the screen, even with small amounts of text.
This seems to be due to zenity wrapping the text with a fixed width as mentioned in a number of bug postings, so to solve this I used the  --no-wrap option.

Second one is a little more random. I had occasionally noticed my dialog boxes containing the text "All updates are complete", originally I thought that maybe this was a random fortune, however none of the fortune files contains this string.
Doing some further digging, I found out that this is the default message (mentioned here) and that zenity is trying to handle Pango markup and that I should make sure that any of the following characters "&\<>" are escaped prior to passing in.
In later versions of zenity (including the version I am using) there is an undocumented option --no-markup. This now works however I cannot pass "\n" to be interpreted as a new line. Therefore I can either, use this option and explicitly put a line break in the output I want, or pass the output of fortune through sed first and not worry about the no markup option.
The invocation of sed is

sed -e 's/\\/\\\\/g' -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g

There was another issue I came across.
When zenity generates a dialog box it will try to place it above the window where it was launched from, it does this by getting the environment variable $WINDOWID, to prevent this unset this variable in the script before calling zenity.

Sending key presses to specific windows in X

For a project I wanted to send key presses to a specific X Window, so I dug out my code from "Moving mouse and pressing keys in X using python" and made some modifications.
I also included the code I wrote in "screenshots using Python under Linux" for finding a particular window id from its name.


As an example the following code takes a window name as an argument to the program when run, finds the window id and then sends the key press of F11 (the window I wanted to interact with was a browser window and therefore it will go fullscreen).

#!/usr/bin/python
"""
Issue a keypress event in X to a specific window
"""
from Xlib import XK, display, ext, X, protocol
import sys
import time

def get_window(display, name):
    root_win = display.screen().root
    window_list = [root_win]

    while len(window_list) != 0:
        win = window_list.pop(0)
        if win.get_wm_name() == name:
           return win.id
        children = win.query_tree().children
        if children != None:
            window_list += children

    print 'Unable to find window matching - %s\n' % name
    sys.exit(1)


d=display.Display()

win=get_window(d,sys.argv[1])

keysym=XK.string_to_keysym("F11")
keycode=d.keysym_to_keycode(keysym)

event = protocol.event.KeyPress(
   time = int(time.time()),
   root = d.screen().root,
   window = win,
   same_screen = 0, child = X.NONE,
   root_x = 0, root_y = 0, event_x = 0, event_y = 0,
   state = 0,
   detail = keycode
)
d.send_event(win, event, propagate=True)
d.sync()
event = protocol.event.KeyRelease(
   time = int(time.time()),
   root = d.screen().root,
   window = win,
   same_screen = 0, child = X.NONE,
   root_x = 0, root_y = 0, event_x = 0, event_y = 0,
   state = 0,
   detail = keycode
)
d.send_event(win, event, propagate=True)
d.sync()

While this seemed to work (my Firefox browser window went into fullscreen mode), I did start to see some strange behaviour, such as none of the menus would display either from the toolbar or right clicking in the window. I did notice that clicking on a menu did very briefly flash up what looked like an outline, so I wondered if this could have anything to do with focus and modified my code like so.

#!/usr/bin/python
"""
Issue a keypress event in X to a specific window
"""
from Xlib import XK, display, ext, X, protocol
import sys
import time

def get_window(display, name):
root_win = display.screen().root
window_list = [root_win]

while len(window_list) != 0:
win = window_list.pop(0)
#print win.get_wm_name()
if win.get_wm_name() == name:
return win.id
children = win.query_tree().children
if children != None:
window_list += children

print 'Unable to find window matching - %s\n' % name
sys.exit(1)

d=display.Display()

win=get_window(d,sys.argv[1])

keysym=XK.string_to_keysym("F11")
keycode=d.keysym_to_keycode(keysym)

#store current input focus
currentfocus=d.get_input_focus()

#set input focus to selected window
d.set_input_focus(win, X.RevertToParent,X.CurrentTime)

#send keypress and keyrelease
ext.xtest.fake_input(d, X.KeyPress, keycode)
ext.xtest.fake_input(d, X.KeyRelease, keycode)

#revert focus to original window
d.set_input_focus(currentfocus.focus,X.RevertToParent,X.CurrentTime)

d.sync()
d.close()

This seems to work much better and Firefox still works correctly.

vim substitution tricks

I wanted to figure out a way to add an incrementing number before a certain character on each line for a program I was writing.
In vim this is fairly simple to achieve.

Here is a sample of some lines in my file

number=1234
number=4543
number=7978
number=3321

Now I want to add an incrementing number before the "=" so I have some unique variables. I can either select the lines I want in vim using the visual selection (shift-v) or I can put an address specification.
For this example I used the visual selection so I get and address specified as '<,'>

Now I supply the ex command

:let i=1 | '<,'>s/=/\=i."="/ | let i+=1

This will now give me

number1=1234
number2=4543
number3=7978
number4=3321

This seemed to work the same as the following command but less typing

:let i=1 | '<,>'g/=/s//\=i."="/ | let i+=1

However if I wish to change the increment I have to use the long command, e.g. to go up in steps of two, change the ex command to

:let i=1 | '<,>'g/=/s//\=i."="/ | let i+=2

Which now gives me

number1=1234
number3=4543
number5=7978
number7=3321

The string to replace in the search and replace pattern above starts with "\=" indicating that this is the start of an expression, subsequent strings are concatenated with ".".

python and numbers 08 and 09

I had noticed that the syntax highlighting in vim did not seem to pick up the numbers 08 and 09 in some code I was writing. However, it was only when running some code containing these numbers that I appreciated what was happening.

If numbers have been represented with a leading 0 then these are interpreted as octal numbers and therefore each digit after can only go up to 7, therefore 08 and 09 are invalid.

This can be seen if you try to run the following code

number = 08
print number

You get an error similar to

  File "number", line 4
    number=08
            ^
SyntaxError: invalid token

Random Programming challenges

I have found some interesting websites that set interesting problems to challenge your programming skills and as I have been messing round more with python I have decided to have a go at some of them using python.

A particularly interesting one is Project Euler where I have just achieved the award Decathlete for solving the first ten problems. I suspect the rest will be much harder.

Another challenging set of problems is at http://escape.alf.nu/ this guides you through some common pitfalls in javascript programming. The aim is to get the sample code to call alert(1).

Rookie mistake python loops and variable scope

While writing some simple python code I stumbled upon a rookie mistake which I appear to still be making so writing it down here to remind me.

Scope of variables is important in loops especially when using them to increment.

I was writing some code to work out if the product of two 3 digit numbers is a palindrome (not very elegant algorithm just brute forces it). Here is my original algorithm

a=100
b=100

numbers=[]

while a < 1000:
  while b < 1000:
    product=a*b
    print a, b, product
    if (product == int(str(product)[::-1])):
        print "palindrome"
        numbers.append(product)
    b+=1
  a+=1

From the output I noticed something strange the outer loop only went round once even though I thought I had incremented a after the inner loop.

Then it struck me I had defined b outside my while loop so it would not get reset upon next iteration of the outer loop and therefore the inner loop would not run again (and print something out). Therefore it looked like I was only looping once on the outer loop.
This could be proven by adding "print a" before incrementing a on the last line.

Moving "b=100" inside the outer loop solves the problem.

Working code looks like so

a=100
numbers=[]
while a < 1000:
  b=100
  while b < 1000:
    product=a*b
    print a, b, product
    if (product == int(str(product)[::-1])):
        print "palindrome"
        numbers.append(product)
    b+=1
  a+=1

Another alternative is to change the while loops to for loops.

Easter egg in gnome-shell

I have just found and easter egg in gnome-shell.

In gnome, move the cursor up to the top left corner as if you were going to open an application, then in the search box type "free the fish"

Wanda

Clicking on the icon or hitting return will bring up a dialog box containing a random fortune.

Drag and Drop in a webpage

I saw a cool feature on a website allowing you to drag and drop a file onto the page and it will be uploaded.

I wondered how this was done and after some experimentation I believe I understand how to do this. I should note that I was working with firefox on Linux and I have not tested this code in other browsers on other platforms.

I created a simple blank html page with a line explaining to drag and drop a file onto the page.

Then a little javascript to handle the dragging and dropping a file.

In my script I need to get the whole document (or the body element) which can be done with
document.documentElement

Then I need to add some event listeners to this to listen for the drag and drop events (in particular the "drop" event). I created some functions to be called when these events are triggered.
To prove a file has been dropped all I did was popup an alert with the name of the file dragged and dropped.
My test code only handled the first file dropped, but it should not be too difficult to add a loop to process the rest.

So my basic javascript would look like this

function noop(e) {
    e.stopPropagation();
    e.preventDefault();
    return false;
}

var drop = document.documentElement;

drop.addEventListener("dragover", noop, false);
drop.addEventListener("dragexit", noop, false);
drop.addEventListener("drop", function(e) {
     e.stopPropagation();
     e.preventDefault();
     //only get the first file, add loop for multiple file handling
     var files = e.dataTransfer.files;
     var file = files[0];
     alert("file '"+file.name+"' dragged and dropped");
     return false;
},false);

This was not successful and kept giving me an error in firefox, this was probably due to trying to do things before the document was ready. Therefore I used jQuery to ensure the document was ready, now my code was wrapped like so

$(document).ready(function() {
...
});

However, this still did not seem to be working. I found some old bugs which suggested that firefox had problems with drag and drop from anything other than nautilus, but I was using nautilus so it should work.
Then it struck me I was dragging items from the recent files window in nautilus, could these be references to the files? I opened the actual directory where the test files were and drag and drop worked perfectly.

I did not do anything with the file dropped, but is should not be difficult to add upload code to pass to a server side script for processing.

Reversing Shellcode

I was looking at some shellcode and there are lots of articles on how to compose shellcode but not that many on how to get it back to assembly for analysis.

The easiest way appears to be to dump the shellcode to a file,  this can be done with a perl one liner e.g.

 perl -e 'print "\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e"' > shellcode

 Then using ndisasm from nasm to convert this shellcode back to assembly mnemonics. The shellcode I was looking at was for 32bit architecture therefore I can use the following

ndisasm -b 32 shellcode

Which gives me
00000000  682F2F7368        push dword 0x68732f2f
00000005  682F62696E        push dword 0x6e69622f

It should be noted that this is a nonsense example as the shellcode only pushes "/bin//sh" onto the stack and does not do anything with it, however it shows how to use the tools.

bash history tricks

I came across an interesting blog entry about maintaining a persistent history of commands across multiple bash sessions using some features of bash I was unaware of.

First there was a variable PROMPT_COMMAND which will run the command stored in the variable before presenting the new prompt.

Then there was the BASH_REMATCH variables, which store substring patterns when used with [[ ... ]] e.g.

test_string="Random string of numbers : 1234"
[[
   $test_string =~ (.*)\:\ +([0-9]+)
]]

echo ${BASH_REMATCH[0]}
echo ${BASH_REMATCH[1]}
echo ${BASH_REMATCH[2]}

This snippet will output the following
Random string of numbers : 1234
Random string of numbers
1234

I have simplified the example from Eli's blog as I am not using the same format for history output , so my code in .bashrc looks like

log_bash_history()
{
  [[
    $(history 1) =~ ^\ *[0-9]+\ +(.*)$
  ]]
  local command="${BASH_REMATCH[1]}"
  if [ "$command" != "$HISTORY_LAST" ]
  then
    echo "$command" >> ~/.persistent_history
    export HISTORY_LAST="$command"
  fi
}
export PROMPT_COMMAND=log_bash_history

gdb not stepping into break point set on strcpy

I was messing around with gdb and some test programs and I set a breakpoint on a call to strcpy, when I ran the program it stepped right over this breakpoint.

Initially I had assumed that glibc had been stripped but this was not the case. Further investigation lead me onto this discussion, where it was suggested to compile with the gcc option -fno-builtin.

This appeared to solve my issue and now when running this code in gdb it does step into the breakpoint set for strcpy.

launching applications upon login to Gnome3

I wanted to run a simple program after logging into a graphical session (Gnome3).

There is a simple graphical utility startup applications (gnome-session-properties), this can be run directly but does not by default show up in search.
To have it show up in all applications or search edit the file /usr/share/applications/gnome-session-properties.desktop and change the line NoDisplay to false e.g.

NoDisplay=false

Adding a new startup application will create a new desktop file in ~/.config/autostart. Therefore to create a new startup application a new file could be created in this directory, for format of the file just copy an existing file and modify as necessary.
A simple entry is below

[Desktop Entry]
Type=Application
Exec=<command>
Hidden=false
X-GNOME-Autostart-enabled=true
Name[en_US]=<name>
Name=<name>
Comment[en_US]=<comment>
Comment=<comment>

python module is in which file on the filesystem

I wanted a quick way to find out which file contains certain python modules, this can by done by looking at __file__ variable for a module.

e.g. in the interpreter
>>> import pygame
>>> print pygame.__file__
/usr/lib64/python2.7/site-packages/pygame/__init__.pyc

 

screenshots using Python under Linux

I have been working on porting some code to Linux and required a method to grab a screenshot of a window using python. I could just execute a command line application such as scrot, but I wanted to see if there was an easy way in native python.
Turns out it is fairly straight forward.

First I looked at taking a basic image of the whole screen, then tried to modify it to allow me to grab particular windows.

My first attempt seemed to work but the output image was skewed, this was using pygtk (gtk.gdk).

#!/usr/bin/python
'''
Modified from example at http://ubuntuforums.org/showthread.php?t=448160&p=2681009#post2681009
'''

import gtk.gdk
import os
import time
import Image

def screenGrab():
   root_win = gtk.gdk.get_default_root_window()
   size = root_win.get_size()
   pixbuf = gtk.gdk.Pixbuf(gtk.gdk.COLORSPACE_RGB,False,8,size[0],size[1])
   pixbuf = pixbuf.get_from_drawable(root_win,root_win.get_colormap(),0,0,0,0,size[0],size[1])
   if (pixbuf == None):
     return False
   else:
     width,height=size[0],size[1]
     return Image.fromstring("RGB",(width,height),pixbuf.get_pixels())if __name__ == '__main__':

   screen=screenGrab()
   #screen.show()
   screen.save(os.getcwd()+'/'+str(int(time.time()))+'.png', 'PNG')

This appeared to be using gtk2, was there anyway to use gtk3?

While searching I found a tutorial on PyCairo and this had an example of taking a screenshot using Gdk and using a cairo Image Surface. This seemed to do what I needed.

#!/usr/bin/python

'''
Modified from
ZetCode PyCairo tutorial

This code example takes a screenshot.

Original author: Jan Bodnar
website: zetcode.com
'''

from gi.repository import Gdk
import cairo
import os
import time


def main():
   
    root_win = Gdk.get_default_root_window()

    width = root_win.get_width()
    height = root_win.get_height()   
   
    image_surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height)               
    pixbuf = Gdk.pixbuf_get_from_window(root_win, 0, 0, width, height)
       
    cr = cairo.Context(image_surface)   
    Gdk.cairo_set_source_pixbuf(cr, pixbuf, 0, 0)    
    cr.paint()

    image_surface.write_to_png("screenshot-"+str(int(time.time()))+".png")
       
       
if __name__ == "__main__":   
    main()


Now to modify this to capture a particular window. I can get information on any window using xwininfo, included in this is the xid (window id).
If I have the id of the window I want to capture can use the function GdkX11.X11Window.foreign_new_for_display() to switch to this window instead of the root window before creating the pixbuf and image surface.

#!/usr/bin/python

'''
Modified from
ZetCode PyCairo tutorial

original author: Jan Bodnar
website: zetcode.com

This code example takes a screenshot of particular window, id in hex
supplied as first argument

'''

from gi.repository import Gdk
from gi.repository import GdkX11
import cairo
import os
import time
import sys

def main():
    winid = int(sys.argv[1],16)
    root_win = GdkX11.X11Display.get_default()
    win = GdkX11.X11Window.foreign_new_for_display(root_win,winid)

    width = win.get_width()
    height = win.get_height()   
   
    image_surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height)               
    pixbuf = Gdk.pixbuf_get_from_window(win, 0, 0, width, height)
       
    cr = cairo.Context(image_surface)   
    Gdk.cairo_set_source_pixbuf(cr, pixbuf, 0, 0)    
    cr.paint()

    image_surface.write_to_png("screenshot-"+str(int(time.time()))+".png")
       
       
if __name__ == "__main__":   
    main()


This works well, however I do not want to have to rely on an external application to get the id. Can I supply a window name and do the same thing?

GdkX11.X11Window.foreign_new_for_display() as well as taking a window id as an argument also takes a window object, so I just need to find a way to identify which window. This was not successful, however if I can identify the window by the name I can just pass the id of that window to GdkX11.X11Window.foreign_new_for_display().
This worked and now I have a script that will take a window name and dump an image of it to a png file.

#!/usr/bin/python

'''
Expanded upon code from
ZetCode PyCairo tutorial

original author: Jan Bodnar
website: zetcode.com

This code example takes a screenshot of a particular window from name
passed as argument.

'''

from gi.repository import Gdk
from gi.repository import GdkX11
import Xlib
import Xlib.display
import cairo
import os
import time
import sys

def get_window(name):
    mydisplay = Xlib.display.Display()
    root_win = mydisplay.screen().root
    window_list = [root_win]

    while len(window_list) != 0:
        win = window_list.pop(0)
        #print win.get_wm_name()
        if win.get_wm_name() == name:
           return win.id
        children = win.query_tree().children
        if children != None:
            window_list += children

    print 'Unable to find window matching - %s\n' % name
    sys.exit(1)
    return None

def main():
   
    root_win = GdkX11.X11Display.get_default()
    browser_win = get_window(sys.argv[1])
    win = GdkX11.X11Window.foreign_new_for_display(root_win,browser_win)

    width = win.get_width()
    height = win.get_height()   
   
    image_surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height)               
    pixbuf = Gdk.pixbuf_get_from_window(win, 0, 0, width, height)
       
    cr = cairo.Context(image_surface)   
    Gdk.cairo_set_source_pixbuf(cr, pixbuf, 0, 0)    
    cr.paint()

    image_surface.write_to_png("screenshot-"+str(int(time.time()))+".png")
       
       
if __name__ == "__main__":   
    main()

Moving mouse and pressing keys in X using python

I have just seen a cool piece of code in python which simulates key presses and mouse clicks to automatically play a browser based game. This was on Windows but it got me thinking of how similar keyboard and mouse events could be simulated in Linux using pure python (I could wrap programs such as xvkbd or xdotool, or even write some extensions in C and hook into xlib).

Fortunately someone has already written a python X library (Xlib) which I can utilise. Here are a couple of simple programs to get an idea of how to simulate mouse and keypress events.

First up how to move the mouse. This snippet moves the moves to a set location and clicks the first mouse button (left click for my setup)


#!/usr/bin/python
from Xlib import X, display, ext

d = display.Display()
s = d.screen()
root = s.root
#move pointer to set location
root.warp_pointer(300,300)
d.sync()
#press button 1, for middle mouse button use 2, for opposite button use 3
ext.xtest.fake_input(d, X.ButtonPress,1)
d.sync()
#we want a click so we need to also relese the same button
ext.xtest.fake_input(d, X.ButtonRelease,1)
d.sync()


Now, how to simulate keypresses.

#!/usr/bin/python
from Xlib import XK, display, ext, X

d=display.Display()
#send F1 as in gnome this will bring up help screen, proves it works
keysym=XK.string_to_keysym("F1")
keycode=d.keysym_to_keycode(keysym)
#press key
ext.xtest.fake_input(d, X.KeyPress, keycode)
d.sync()
#remember to release it, otherwise help screen will continue to appear
ext.xtest.fake_input(d, X.KeyRelease, keycode)
d.sync()

OpenIndiana

I have not used Solaris in some time so I thought it would be interesting to get OpenSolaris installed under kvm to test out. However, this has stopped being produced, but there is a fork called OpenIndiana.

I downloaded the Live dvd for desktops and booted that up under kvm, I have given the vm a qcow2 format disk to install to.

Once the OS boots up there is an option to install to disk, this will take you through a few install questions and then install to disk.

Qemu monitor

If your virtual system is running under qmeu then you can access the qemu monitor to interact with that os, various things can be achieved such as sending keystrokes, sending nmi, changing state of the vm, snapshotting, add/remove devices etc.

To gain access to the monitor
ctrl-alt-2

There is a good description of the various options in QEMU-Buch / QEMU-Book, the book is in German but there is an English translation.

Running Raspbian image via qemu on x86_64

While testing some virtualisation configurations I came across mention that it is possible for qemu to run code compiled for different architectures. I was particularly interested in arm, as I could then run some quick testing of Raspberry Pi stuff from my laptop.

So here is how to get Raspbian OS running under qemu.

I needed to install the package containing qemu-system-arm to allow me to run software compiled for arm. I also needed a suitable kernel compiled for arm as well, there are some good instructions located at http://xecdesign.com/compiling-a-kernel/ .

The particular arm cpu I am going to use is arm1176, first double check that this is provided by qemu, it should show up in the list of cpus given by the command
qemu-system-arm -cpu ?

The particular arm machine I am going to use is versatilepb, for a full list of machines run
qemu-system-arm -M ?

Next I need a copy of Raspbian OS image from http://www.raspberrypi.org/downloads, extract from the archive.

To boot the OS I use the following command
qemu-system-arm -kernel <location of kernel for arm> -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1" -hda 2013-07-26-wheezy-raspbian.img

This did not work upon first attempt, this is mentioned on the Raspberry Pi forums. I changed my command to include an additional kernel boot parameter init=/bin/sh, and once booted up I remounted the root filesystem rw and commented out the line in /etc/ld.so.preload.

I restarted the OS using the unmodified command above and it booted.

Monkey Patching

This was mentioned in passing in a python book I read and I ignored it until I saw it being used to discover the internals of frozen/obfuscated python code.

It is a great term, basically you can overload defined functions with your own code, this is particularly useful for code testing. For example I have seen someone discuss testing an emailing function and rather than send out lots of emails they monkey patch parts of smtplib to add the messages to a dictionary.

A simple example
class orig:
    def method1(self):
         print "Original method1()"

    def method2(self):
        print "Original method2()"

if __name__ == "__main__":

    obj = orig()

    def monkeypatch():
        print "Monkey code!"

    print "method1()"
    obj.method1()
    obj.method1 = monkeypatch
    print "method1()"
    obj.method1()
    print "method2()"
    obj.method2()

Developing Android apps

I thought I would have a quick run through some of the tutorial info on getting a basic Android application built and then run it on the Android x86 port I have running in a VM.

I downloaded the adt bundle for linux which includes the sdk, a version of eclipse and the adt plugin. First time through I forgot that I did not have a jdk environment installed so had to grab that as well.
I also added the platform-tools and tools directories to my PATH.

Following the instructions I tested out a basic layout on the emulator that comes as part of the bundle and then exported my project as an Android application.

Here is where I believed the tricky part would be, to get the apk file onto my android instance running in a VM. I could have placed it on a webserver and then downloaded it (but where is the fun in that). If this was runnign on a real device/handset I could connect it up via usb and use adb.

Actually I can still use adb, as android is running in a VM and has networkig configured I just need to configure adb to connect to the VM.
I do this by the command

adb connect <ip address of VM>:5555

Then I can load my custom application using
adb install <apk file>

When I did this I needed to dismiss a warning that popped up on android, but the application installed fine. 

Once the application has been installed, disconnect adb using
adb disconnect

Both before connecting adb and after it is worth checking that an existing adb server instance is not already running, it can be killed with
adb kill-server

Now to figure out a useful app to develop.

Haiku

I am not about to break into poetry so don't panic.

A number of years back when I first started looking at Unix/Linux I came across several other Operating Systems which I would like to try out, one of these was BeOs (time permitting I may explore some more).

However, BeOs has all but died and was a proprietary OS anyway, but I did come across some people developing an opensource implementation called Haiku

I have downloaded it and installed it in a virtual image under KVM.
I created a 3Gb raw image and have booted from the ISO image using the following command to bring up a 32bit vm and given it 512Mb

qemu-system-i386 -m 512 -boot d -cdrom <image file> -hda haiku.img

I just passed it a raw image, so using the disk utility in the installer I need to create a partition and a location to create as Be File system. Follow the instructions in the installer and it is installed.
Next time when booting up the following command is used

qemu-system-i386 -m 512 -boot c -hda haiku.img

emscripten

There was a cool demo using the Unreal Engine that had been ported to javascript using emscripten I saw a while ago.
I had some time I thought I would test this out and see what it can do, I will come back to it later if I have any nifty code that I want to reproduce on the web.

Following the instructions, I downloaded and compiled up llvm (& clang) version 3.2, as per the instructions 3.3 (which was provided in my distribution) does not work.
I needed to grab node (version 10.15 was the one I tested) as well.

Once I had all these I ran the emcc utility which generated a default config file in my home directory (.emscripten), the paths in this needed modifying as I compiled/installed the utilities in my own directory and not in directories for system wide use.

I ran emcc again and it ran with out errors, just the fact that I had not provided any input files, so looks good.

Following the brief tutorial on the wiki I was able to run some of the test programs and some very simple C code that I had written (including some basic openGL using simple lines).

When I have some good examples I will attach them to this page.

Simple http server

In need of a quick simple http server on your local system to do some testing?

I have just come across this useful tip.

In a directory that you want to make available via http to do some testing on your local machine run the following command where <port> is a port to listen on

python -m SimpleHTTPServer <port>

The -m option means run a module as a script.

Compiling old versions of gcc

I wanted to compile up an old version of gcc (2.95.3) for another project which I might blog about in the future, I assumed this would not be too difficult, however it turned out more troublesome than I had thought.

I have on my Linux system gcc4 compiler and to use this to compile the source of version 2.95.3 a few tweaks have to be made. I should also mention as a remindeer to read and understand the output/errors they can save some time. I would have got to the solution sooner if I had matched up the error with some information I already had.

I unpacked gcc 2.95.3 source to a directory and then created a separate directory to build into (as recommended), I have also created a separate directory to install finished binaries into so as not to mess up my current system (also doing this as a non-root user).

First attempt to compile (from within my build directory) using the following options
../gcc-2.95.3/configure --prefix=<new bin directory> --enable-languages=c,c++ --enable-threads=posix --enable-shared --disable-nls --host=i386-pc-linux-gnu --target=i386-pc-linux-gnu

I only want support for c and c++, posix threads are enabled, I have disable NLS as I do not need it and I am specifying I want to create a compiler for 32bit.

This attempt fails to compile (due to two issues, but originally I concentrated on one and wasted a lot of time).
The first issue I noted was the errors from the assembler such as
Error: invalid instruction suffix for `push'
Error: invalid instruction suffix for `pop'

This suggested I had an architecture issue, so I spent a lot of time looking for the solution to this and ignored the following problem (which later on I realsied needed fixing too).

/usr/include/bits/pthreadtypes.h:99: warning: unnamed struct/union that defines no instances

This version of gcc does not support the unnamed union definition that appears in one of the header files on my system, more about how to solve this one later.

While trying to track down solutions for the architecture problem I believed I was facing above, I came across a blog entry here, which seemed to document a similar issue about invalid instructions.
The fix given was to add "--32" to the gcc assembler spec options in the header file gcc/config/svr4.h 

Second attempt to build having added the above entry, again compiling in my build directory with the following options

../gcc-2.95.3/configure --prefix=<new bin directory> --enable-languages=c,c++ --enable-threads=posix --enable-shared --disable-nls --host=i386-pc-linux-gnu --target=i386-pc-linux-gnu

This also failed to compile and returned

In file included from /usr/include/sys/types.h:271,
                 from /usr/include/stdlib.h:320,
                 from ../../gcc-2.95.3/gcc/frame.c:42:
/usr/include/bits/pthreadtypes.h:99: warning: unnamed struct/union that defines no instances
../../gcc-2.95.3/gcc/frame.c:55: extra brace group at end of initializer
../../gcc-2.95.3/gcc/frame.c:55: (near initialization for `object_mutex.__data')
../../gcc-2.95.3/gcc/frame.c:55: warning: excess elements in struct initializer
../../gcc-2.95.3/gcc/frame.c:55: warning: (near initialization for `object_mutex.__data')

This is what I noted earlier and spent ages on other possible problems ignoring this one and again this I had already found on the blog entry here, and is due to an unnamed union definition. The solution to this is just to make a copy of the header file, add a name to the unnamed union and then get gcc to include it.
The makefile that needs modifying is gcc/Makefile.in, the directory containing the modified header file is included before any others.

This now successfully compiled on the next attempt and I was able to use this to compile 32bit binaries.

In between each attempt I issued "make distclean" to ensure a clean directory in which to do the builds.

Nested hypervisors

I wanted to try out some different hypervisors and rather than shelling out for additional hardware I thought I would see how well my current system copes with nested hypervisors.

Initial test with VMWare under KVM failed, it did not seem to like the CPU I had provided to the guest and no matter what options I chose it did not want to run, therefore that is on hold for now.

However, Citrix Xenserver appeared to install fine.

I want to add a guest to Xenserver, just to prove this is working, but when I created my original disk I gave it only the bare minimum for Xenserver which was 16Gb. So I have created a new raw disk image using qemu-img and attached this to the Xenserver VM.
This image is attached as a new SCSI disk (so it will be /dev/sdb).

I now need to create a new storage repository (SR) which Xenserver will use to store my guests.

First off I need to fine the id of my new disk (/dev/sdb) this can be done by looking at the symlinks in /dev/disk/by-id and finding the one pointing to /dev/sdb.

For my Xenserver vm

ls -l /dev/disk/by-id

I also need the uuid of the Xenserver host, to get this use
xe host-list

Now I can create the new SR using
xe sr-create content-type=user device-config:device=<disk id> host-uuid=<host-uuid> name-label=”Second Disk” shared=false type=lvm

Creating new SR

Tip:- When entering the console from the xsconsole application, the font was pretty unreadable, I just ran setfont  with no options to load the default and this resolved the issue. To set the font permanently modify /etc/sysconfig/i18n.

Now I should be able to create a new VM on this new SR, but first I need to get the uid of this new SR using
xe sr-list

To create the new VM using an available template

xe vm-install template=<template-name> new-name-label=<vm-name> sr-uuid=<sr-uuid>

To list the available templates use

xe template-list

Tip:- With the xe command you can use tab completion to finish commands and uuids.

The new VM now shows up in the list of VMs
xe vm-list

I have a basic VM (with nothing in it), which I now need to start up and boot from some install media. As I want to boot from the install media I need to set the disk for my VM to not be bootable as I cannot have two boot devices.
First off I need to find out the uuid for virtual block device (VBD) which will contain my VM, this will be the device xvda in my OS and is therefore the first userdevice (0).

xe vbd-list vm-uuid=<vm_uuid> userdevice=0 params=uuid --minimal

Set this to not be bootable

xe vbd-param-set uuid=<root_disk_uuid> bootable=false

I have attached an iso of the install media for a linux distro as a cdrom/dvd drive to my Xenserver VM, I now need to attach the cdrom/dvd drive to the new VM which I will boot up on Xenserver.

To find out what cdrom/dvd devices are available use

xe cd-list

In my case this is labelled "SCSI 2:0:0:0", and to add this device as a virtual cdrom/dvd drive to my VM is use the following

xe vm-cd-add vm=<vm_name> cd-name="SCSI 2:0:0:0" device=3

This will attach the cdrom/dvd drive of the Xenserver host (in my case the iso I have attached to the cdrom/dvd drive of the Xenserver VM) as device 3 which in Linux will be xvdd.

I need to make the VM boot from the cdrom/dvd drive so I need to find the VBD and set it to be bootable

xe vbd-list vm-uuid=<vm_uuid> device=xvdd params=uuid --minimal

xe vbd-param-set uuid=<cddrive_uuid> bootable=true

Finally, before I start the VM I need to set the install-repository to be the cdrom

xe vm-param-set uuid=<vm_uuid> other-config:install-repository=cdrom

Once all the above has been completed successfully it is time to start the VM

xe vm-start uuid=<vm_uuid>

This should start the VM, to check this look at the power-state line in the output of

xe vm-list uuid=<vm_uuid>

I want to be able to interact with the installation program to install my OS onto my VM, normally if Xenserver was on physical server I could connect to it using Xencenter. However, it is on a virtual machine running under KVM, also that would require me to run Windows.

I could create another VM under KVM and connect to it from there but as this is runnign on my laptop it sounds like overkill.

Xenserver exports the display of the VM via VNC, again not very useful as I do not have a VNC client on the Xenserver host and I would have to forward ssh connections to it from outside the Xenserver VM. So I need a console connection to the console of my newly created VM, there are two commands on Xenserver that allow you to connect to the console of a VM, xenconsole and xl console.

For some reason the location of xenconsole is not by default in the PATH for root, it can be found in /usr/lib/xen/bin.

To use either of these tools I need to know the domain-id of the running VM I want to connect to, all running domains can be gotten using the command list_domains.

list_domains output

My VM is listed as having a domain-id of 4.

The domain-id is also available in the dom-id parameter for the VM and can bo obtained using

xe vm-list uuid=<vm_uuid> params=dom-id

I tried both of these tools and I was not able to get a usable connection to the console for my new VM.

**Update**

I later discovered that this could be due to xapi setting up vncterm which takes the text console and draws them as graphics to make them available to vnc clients.

There is an option which can be set before the VM is start which will then allow me to use xenconsole or xl console, this can be done with the following

xe vm-param-set uuid=<vm_uuid> other-config:disable_pv_vnc=true

This may impact tools that connect via vnc. 

To remove this option use

xe vm-param-remove uuid=<vm_uuid> param-name=other-config param-key=disable_pv_vnc

*****

I found out that the text consoles are also made available on certain ports. To find out which port I need to look up the console entry for that domain-id (as mentioned above) in xenstore.

xenstore-ls /local/domain/<domain-id>/console

console entry in xenstore

For some guests (depending on whether they HVM or PV) this entry might be /local/domain/<domain-id>/serial.

If I telnet to the "tc-port" I get access to my text console.

A few notes on this, I changed the tty settings using "stty sane" running telnet. I also set character mode within telnet. e.g.

stty sane

telnet localhost <tc-port>

ctrl-] mode character

Once the install has finished the cdrom/dvd should be ejected from the VM

xe vm-cd-eject vm=<name or uuid>

Internals of Android applications .apk files

Ever since I saw an article on creating a basic android application I have been curious about how some android applications have been created and since I have a working android installation it is time to take a look at how they work.

My android install is on a qcow2 disk image so I can mount it on my system via ndb, just like I did when building my LFS system.

I copied off an apk file to test.

The apk file is effectively just a zip file, so I could just unzip it to get the files, however when I do this most of the files are in a binary format (serialized objects?) and unreadable. Since the discussion a few months back on android bug 8219321 I was aware of the apktool utility to unpack an apk file.

So used apktool to unpack the apk file, like so

apktool d <apkfile> <dir to unpack to>

I now have manifest xml files which are readable and code. I was expecting to get java sourcecode files from this process, but all the files were smali (Dalvik byte code). This does not look too hard to understand but I wanted something I was a little more familiar with, i.e. java.

Therefore I needed to extract files from within the classes.dex file (Dalvik executables) and convert them to java classes.
So I just extracted the classes.dex file from the original apk file using unzip (jar would work as well if a jdk was installed).

I now need to convert the Dalvik executables to java class files, to do this I used the utility dex2jar e.g.
d2j-dex2jar.sh classes.dex

This has converted the dex file into a jar file, again I could use the jar utility to extract the class files but as it is basically a zip file I will use unzip.

I now have a bunch of java class files which I need to disassemble and look at. In the past I have not had much luck with java dissassemblers, however, I have seen one recommended called jd-gui.

This is a 32bit application and as I am running a 64bit OS I needed to install the 32bit libXxf86vm runtime library.

OCR

I was wondering if there were any easy to use OCR tools for Linux, turns out there is tesseract, originally developed by HP and now maintained by Google (I guess they are using it for their digitisation projects).

To use it simply pass it an image file and the file you want the text to be written to e.g.
tesseract <image file> <output>

I first tried with a png image but that did not produce anything, looks like tiff seems to work so I converted the png image using gimp.

This could be useful to script up something with some scanning software (like scanimage) to automaticlly produce a text file of a scanned document.

I will post more on this once I have played around with it some more.

Dynamic favicon

I saw a post the other day, that Google had changed the favicon to display a play symbol when audio is playing in that tab/window, therefore I decided to see if I could knock up a page that dynamically changes the favicon.

My idea was to create icons that would count to 10 just so it would be easy to tell when it is working correctly. Therefore I created 16x16 pixel png images containing my numbers 1-10 called favicon1.png - favicon10.png. I also created a start image called favicon0.png this is the same image as posted on the banner of my blog (one of the fractal images I created).

To add a favicon is easy just add the following to your html in the head section

<link id="favicon" rel="shortcut icon" type="image/png" href="location of your image">

To change this dynamically it should be easy create a loop where the loop counter goes from 0 to 10 and modify the href in this line.
First problem, how to modify this line. I gave the link an id so I should be able to reference this via the DOM. This can be referenced using the following piece of javascript

document.getElementById('favicon');

Now I just need to create a loop in javascript and modify the href, but wait there is a problem with this. To display each individual favicon image I need my script to pause for a short amount of time. In other languages we would just pause by calling some kind of sleep or wait function, however as this will be running in a browser we do not want it too sleep as this would impact the loading of the page. It also turns out that there is no sleep function in javascript for precisely this reason.
So how to achieve a pause to display the images. Looking around, there are some Timing Events that can be used in javascript, the one that looked to fit the bill is setInterval.
To use this I had to modify the way I was writing the script, rather than having a loop and pausing within the loop, I created a function that I would call at a particular interval.

I also had to remove the old favicon link before adding a new one.

The following sample code is what I came up with and seems to do what I was after, note I was testing on my local machine so the href entries all point to local files.


function changefav(i) {
   //only have 10 numbers to display
   if (i==11) {
         //remove Timeout
        clearTimeout(changeInterval);
        return;
   }
   var filename = "file:///tmp/favicon" + i +".png";
   myfav=document.getElementById('favicon');
   header.removeChild(myfav);
   lnk.href=filename;
   header.appendChild(lnk);
}

header=document.getElementsByTagName("head")[0];
//create new link element, only href will change
var lnk=document.createElement('link');
lnk.id="favicon";
lnk.type="image/png";
lnk.rel="shortcut icon";

var i=0;

//call changefav function every 100 milliseconds
changeInterval=setInterval('changefav(i++)', 100);



I placed this into a simple html document and added a favicon link as mentioned above.

This worked well in firefox.

Instead of creating separate images I could have taken a base image and then modified it using a canvas element dynamically in javascript.

While researching favicons I found Defender of the Favicon, a Defender clone implemented as dynamic favicons.

Uploading images to Blogger

I have been struggling for a while to get images uploaded to blogger successfully every time.

My usual browser is Firefox, less and less frequently am I able to upload images using the Insert Image button in the composer without it throwing an error. I have seen other blogger's comments that say it works much better in html mode than in the wysiwyg mode, however most of the time this failed for me in the same way.

There is also some suggestion by other bloggers that this works much better in Chrome, however, I also see a number of comments from people that this is unpredictable as well.

I came across another web browser, which seems to work every time (fingers crossed), this is a lightweight webkit based browser called Arora (http://code.google.com/p/arora/).

While this continues to work, I will be using this to update my blog.

**Update Sept 2013**
An update to firefox worked for a while, however I am back with Arora as now I am getting a different issue.
Every time I login I get a message telling me I have logged out from somewhere else and do I want to login again. I see this was reported as a known issue a year ago, however it seems to have reappeared.

Running Android in KVM... because I can

I have been toying with the idea of playing around with android and took a quick look at downloading the sourcecode. I decided to hold off as it seemed like a lot of time would need to be spent on this and I had a few other projects I wanted to look at first.

However I found the android x86 port and so I decided to get this running in KVM.
Why?
Because I can.

I downloaded the iso for version 4.2 (I had a few problems with 4.3, however this might be solved in the same way I solved booting 4.2).

I created a new qcow2 qemu image (the original instructions I was looking at suggested using this just to store data and emulate an sd card to save data as the iso is also a live image), however I decided to install it to this disk image.

Using qemu-kvm I booted the iso image and attached the qemu disk I had created. The live image did not seem to boot up at all. If I selected the debug option, this just dropped me into a command prompt. Therefore I decided to install the os to the qemu disk image using the option on the boot menu of the live image.

I also used virt-install to import this disk image which would make it easier to work with.
It still failed to boot correctly, so I checked the VM settings and checked what hardware had been assigned to the VM. I gave it a 32bit cpu as it had defaulted to the same cpu as my hypervisor and because it seemed to be complaining about some graphic functions I switched the video card type from Cirrus to VMVGA (VMWare). With these settings it booted up fine and I got the familiar android home screen.
Sound also seemed not to work so I switched over to using the SPICE protocol instead of VNC, which gave me sound but was very choppy (probably due to running in an emulated environment on my laptop which is not too powerful).

I installed a few apps to prove that this was functioning and challenged my wife to a game of Word Fued.

grub boot screen

lock screen

home screen

I have installed version 4.3-test over the top and it seems to work, although Google Play Services periodically crashed during setup.

As this is not on a phone/tablet rotation can be a bit of a headache (I noticed this with WordFued it wanted to start in landscape), therefore I downloaded a rotation locker app to keep the display in potrait and I also turned of auto rotation in android.

Inkscape/Sozi presentation

Someone a while back showed me a presentation in a web browser which was designed and put together using only opensource tools. Therefore I decided to investigate doing this using Inkscape and Sozi.

The principle of this method is to layout your presentation like a poster and the get Sozi to animate the transitions between portions of this design which will become your "slides". So it all comes down to your flair for design.

I needed something to do a presentation on, so I settled on some of my interests and some of the technologies I have been using and blogging about here.

As soon as I can find a way to link an svg file to the blog I will upload a sample.

libc version

While looking at different ways to find out what version of libc is installed on a system (especially useful on a system that is compiled from source such as LFS), I discovered that you can actually run libc and it will print out useful version info.

For example

# /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.7.2.
Compiled on a Linux 3.8.1 system on 2013-07-18.
Available extensions:
    crypt add-on version 2.1 by Michael Glad and others
    GNU Libidn by Simon Josefsson
    Native POSIX Threads Library by Ulrich Drepper et al
    BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

LFS - Importing disk image into KVM and booting

I thought would probably be the least challenging part of getting my LFS setup working under KVM as once everything is installed on the filesystem it should just be a matter of booting it.

However, when I tried the steps below I realised that grub had not been installed correctly and it dropped my into the grub rescue environment from which I did not seem to be able to fix the problem. Details on how I did fix it below.

First of all, details on how to use the image with KVM. The following steps can be done in graphical management utilities but it is just as easy from the commandline using virtinstall.

Run virtinstall and tell it to import rather than to run an installer, I gave the VM a basic set of requirements namely just 2GB of RAM.

virt-install --name LFS --ram 2048 --disk <location of image file> --import

This created my VM configuration and started the VM (if it had been working correctly the OS would have started) and launched virt-viewer so I could see the console.


As noted earlier this failed to boot first time and dropped me into the grub rescue prompt, therefore I had to find a different way to get grub installed onto the MBR of my disk image.

Lots of posts on forums talk about mounting the disk using losetup, this is fine for raw images but I was using qcow2 format and had made it available via NBD. Other posts talk about installing the grub version from the host system onto the disk image, this would have worked, however why go to the effort of building a custom system if you are going to just copy files from the host system.

Whatever I tried inside a chroot onto my filesystem failed to work (even though grub-install returned success), so I looked for a way to run the grub utilities from the filesystem in my disk image by directly from my host system.

Fortunately it seems that grub-install is setup for running in just this manner with a number of optional arguments to override directory paths and locations of utilities. With my diskimage made available with qemu-nbd and mounted under /media/LFS I used the following command to install grub to the MBR of my disk image.

/media/LFS/usr/sbin/grub-install --boot-directory=/media/LFS/boot/ --directory=/media/LFS/usr/lib/grub/i386-pc --modules="ext2 part_msdos" --grub-mkimage=/media/LFS/usr/bin/grub-mkimage --grub-probe=/media/LFS/usr/sbin/grub-probe --grub-mkrelpath=/media/LFS/usr/bin/grub-mkrelpath /dev/nbd0 --debug

A quick explanation of the options (for more details check grub documentation)
--boot-directory  location of "/boot", where grub files will be installed
--directory  location to copy boot files from
--modules  which modules to load
--grub-*  locations of various grub utilities
--debug  undocumented (according to grub-install script) option so I can see exactly what commands it is running to make sure it does not damage grub on my host system

Now when I startup this VM under KVM it boots fine.

Grub screen

Login prompt - before compiling kernel to include nic driver

LFS - Kernel & grub install

I have a filesystem and a set of packages installed, now I need to get the kernel installed so I can boot this system.

The LFS instructions note that due to recent changes in udev the following kernel options need to be enabled
Device Drivers --->
    Generic Driver Options --->
         Maintain a devtmpfs filesystem to mount at /dev

**update**
Remember to compile in (or as module) your nic driver, doh!
 

As I am doing this in a virtual image I need to be careful I am specifying the right devices when it comes to installing and configuring grub.

Once the kernel has been compiled and the built kernel moved to /boot, grub needs to be installed in the MBR for this disk image. This is done with the command
grub-install <device>

As the disk image is mounted via nbd, the device is /dev/nbd0.

The first  time I tried to do this it failed, grub-install is a script which calls grub-probe and this claimed that the location where /boot/grub is, would not be accessible to grub.
I realised that I had been mounting the filesystem wrong and if I pass the parameter max_part=16 when I load the nbd module, devices are created for my partitions and I can mount them without having to use kpartx or partx.

When I did this, grub-install ran successfully within a chroot from the filesystem on my disk image.

**update**
This did not produce a bootable image, which I did not discover until I tried to boot it in KVM.

During one of my failed attempts to resolve this problem, I had created a custom devicemap entry for grub and when trying to do grub-install I got an error about blocklists similar to

Attempting to install GRUB to a partition disk or to a partition.  This is a BAD idea.
Embedding is not possible.  GRUB can only be installed in this setup by using blocklists. 
However, blocklists are UNRELIABLE and their use is discouraged.

There was a suggestion in the grub manual that I could safely use the option --force in this case, however this was not needed once I remove the custom devicemap.

LFS - Installing software

I have a file system (http://paulsrandomcontent.blogspot.co.uk/2013/07/lfs-creating-disk-image.html) and I have a set of tools (http://paulsrandomcontent.blogspot.co.uk/2013/07/lfs-configuring-tool-chain.html) to build my system, so now is time time to start the build of the new system.

First off I need to create directories for the kernel's virtual filesystems (/proc, /dev, /sys) on my filesystem and also some initial device nodes (/dev/console and /dev/null). To find out the major/minor numbers for certain devices, check Documentation/devices.txt in the kernel source tree.

Entries in /dev need to be created so we can mount the virtual filesystems required, this can be achieved by creating a bind mount from my host system. Once we have this, the virtual filesystems /dev/pts, /dev/shm, /proc, and /sys can be mounted.

The install of the system now needs to happen within a chroot environment on the filesystem I have created which contains all the tools that were installed in a previous entry. A directory structure based on Filesystem Hierachy Standard (FHS) is created into which software will be installed.

As there are some hardwired paths for certain basic utilities in some programs, symlinks in /bin are created to point to our pre-built toolchain.

A basic /etc/passwd and /etc/group is created.

Some programs require log files to have already been created before they try to write to them (such as agetty, login), therefore we need we need to create log files such as /var/log/lastlog, /var/log/btmp, /var/log/wtmp.

The kernel headers are extracted for use by userspace programs and in particular glibc when it is built.

The following packages are installed (watch out for programs shipped as part of multiple packages)
man pages (not sure why first, check??)
glibc (configure timezone and setup /etc/ld.so.conf, gcc was also modified to ensure that no references to /tools exists)
zlib
file
binutils
gmp
mpfr
mpc
gcc (rebuilt so it now refers to installed glibc)
sed
bzip2
pkg-config
ncurses
util-linux
ps-misc
procps-ng
e2fsprogs
shadow (I installed cracklib and enabled support in shadow)
coreutils
iana-etc
m4
bison
grep
readline
bash
libtool
gdbm
inetutils
perl
autoconf
automake
diffutils
gawk
findutils
flex
gettext
groff
xz
grub
less
gzip
iproute2
kbd
kmod
libpipeline
make
man-DB
patch
sysklogd
sysvinit
tar
texinfo
systemd(udev)
vim

Some of these packages took quite some time to compile on my host system.


I ran into a problem when compiling glibc, it seemed to cause gnome-terminal to segfault (once this build is complete I will have to see if I can reproduce the problem and find out what is going on).