Image Manipulation with ImageMagick

Whilst sorting some photos I remembered I had taken a sequence of photographs at different stages of the build of various lego sets (including this one - Great Pyramid of Giza). This was an opportunity to put them together into some kind of animation showing the build progressing.

[note: I did not take the pictures with this aim in mind, if I had then I would not have run into some of the issues I had, better subject positioning, better backgrounds, consistent camera angle etc.]

I knew I could use ImageMagick to bring together my "static" images into an animation but never done this for a significant number of photos.
[note: I could also have done something with ffmpeg - perhaps a future project]

My original requirement was to extract the data and time the picture was taken from the exif data and display that on the image. As I started to look at this I noticed a common positioning and colour might not work due to the various backgrounds and positioning of the subject (again better planning when taking the images would have helped).

I also saw mention of a "polaroid" option in ImageMagick which would add a frame similar in style to a polaroid picture and would also put the image on a jaunty angle. These "polaroids" could then be stacked to make an interesting animation.

Below is my testing on one set of photos (Lego Pyramid), I am hoping to then take a minimal set of steps and apply it to other groups of photos.

Adding a Polaroid effect and adding the Date and Time the photo was taken from the exif data:

magick -caption %[exif:DateTimeOriginal] image.jpg -gravity center -background black -pointsize 20 +polaroid test_timestamp.jpg

Image looked quite good with the Polaroid frame but the text was way too small to be readable (more on why later).
I also wanted the manipulation to be as automatic as possible so I could apply the same to other project photos, so I run the commands through a loop in bash.

Bash loop (increased font size - still not big enough)

for img in *jpg ; do magick -caption %[exif:DateTimeOriginal] "$img" -gravity center -background black -pointsize 25 +polaroid "test_""$img"; done

[side note - separate article on image orientation]

It was obvious when looking at the resulting test images that the images were both too large (did not do any scaling of the originals) and that the orientation was wrong. This is not always obvious when looking at the photos in modern tools that do auto-orientation. As the orientation of the original images was what I wanted when viewed in the image viewer I was using, I decided to manually adjust the few images that were not as I wanted.

magick <image.jpg> -auto-orient <image_fix.jpg

I now had a directory of original images and some additional "fixed" images, for my processing I would only want the "fixed" image (if available) and the originals if not. For following examples/tests I also decided to write the images to new directories so it would be easy to role back to a previous step/set of transformations.

Another quick bash loop, and some bash variable manipulation:

for img in *jpg ; do fix=${img%.jpg}_fix.jpg ; \
if [ -e "$fix" ] ; then echo "fix exists!" ; img=$fix ; echo "using $img" ;fi; \
magick -caption %[exif:DateTimeOriginal] "$img" -gravity center -background black -pointsize 25 +polaroid polaroids/"polaroid""$img"; \
done

Orientation of images is now how I want but still too large and caption font size too small.
As I could get roughly the effect I wanted for each of the images I decided to go back and scale down the images to something more managable and not the files directly from the camera (yes I should have done this first but initially I wanted as little manipulation as possible and to see if I could get something to work).

for img in *jpg ; do magick $img -resize 15% scale/half_$img ; done

Doing a few tests I went with 15% of the original in the end, which is about 600x450 pixels for the images that did not need rotation.

Added the polaroid effect to the scaled images and because the images I am using are much smaller then the pointsize used gives a reasonable size for the caption without needing to increase it. Win!

Lets test creating the animation with following command:

magick -dispose none -size 500x500 xc:Black -delay 5x1 -dispose previous *jpg -loop 2 animation.gif

Explanation of options used, start with a Black background 500x500 pixels (for those keeping count that is too small for the 15% scaled images I mentioned earlier, this test was done with images at 10% scaled images) and "do nothing" as the disposal for this first transition (I added this as I wanted to test different options for each frame as I add the images). Overlay each image every 5 seconds (5, 1 second ticks) and set the disposal to previous (return canvas back to how it looked like before applying the image).
There were some problems with this, I wanted a stacking look so previous disposal option is probably not what I want, and the polaroid effect was adding some extra colouring round the edge (probably want this to be transparent). The images that were originally rotated were bigger than the canvas size I was using and so were partly out of view and caused the images to overlay in a way that made the stacking effect look odd.
I also noticed that even if I got a stacking effect when viewing the resulting gif animation (which should not have happened with the dispoal setting I was orginally testing above) this was being caused by overflow of the buffer the image viewer was using to display the images and therefore was not clearing the previous image as requested.
As mentioned at the top if I was planning to animate the images from the beginning I would have set the Photos up better, subject in the middle of the frame and all take with the same orientation etc.

At this point I decided the simplest solution is to crop the rotated images, I don't need the uncropped rotation so I just overwrote them. I also decided to go with a slightly large image so only scaled to 15% instead of 10% that I had originally been testing.

The majority of the images were 600x400 pixels and the rotated ones (which were causing me the issue were 450x600 pixels) so I decided to crop them using:

for img in *jpg ; do magick identify $img | grep "450x600"; RESULT=$?; if [ $RESULT -eq 0 ]; then echo "crop "$img ; magick $img -crop 600x500+0+0 +repage $img ; fi ; done

Applied the polaroid effect again to these cropped images.

Nearly there, but something looked odd, there were extra artifacts which looked like background bleeding through from a previous image
At this point I thought it would be best to have a transparent background to make sure this was not happening, to achieve this I probably need to work with pngs instead of jpegs so I converted them all using:

for img in *jpg ; do fix=${img%.jpg}_fix.jpg ; if [ -e "$fix" ] ; then echo "fix exists!" ; img=$fix ; echo "using $img" ;fi; magick -caption %[exif:DateTimeOriginal] "$img" -gravity center -background white -pointsize 25 +polaroid -transparent white polaroids/"polaroid""${img%.jpg}.png"; done

Interesting note:
I was not aware there was such a thing as animated png (APNG), so I tried this to generate one from my images using:

magick -dispose none -size 500x500 xc:Black -delay 5x1 -dispose previous *png -loop 2 APNG:animation.png

This did not work very welli, some issues with backgrounds and transitions, also support did not appear to be there in the image viewers I tried and it was only displaying the first image. I might come back and look at this in the future.

Animated gif generated using the following command:

magick -dispose none -size 750x700 xc:Black -delay 5x1 -dispose previous *png -loop 2 animation.gif

Not quite what I wanted as I had forgotten to change the disposal setting as it originally seemed to work in the image viewer as mentioned above, I actually wanted "-dispose none" so command should be:

magick -dispose none -size 750x700 xc:Black -delay 5x1 -dispose none *png -loop 2 animation.gif

After all this here is the resulting animated gif for my Lego Pyramid build

Animation gets a little mangled when uploaded but gives you an idea.

First proper use of CAD to create a 3D print

Most of my 3D printing so far has been using existing models and making small tweaks / remixing them for my own needs / interest. Finally I decided I should learn some basic CAD to design my own prints. My first project turned out well (if not slightly larger than I was expecting) ...

This was exactly as I had envisaged, a name tag that I could attach to a present which could be easily detached to be kept if wanted. I used FreeCAD to design and export as .stl file which I could import into Cura for slicing.

Now for more of my own designs...

(wow it has been a long time since I posted anything here)

First Adventures in 3D printing

I have purchased a 3D printer and have started my adventures in 3D printing.

There is a steep learning curve and have so far just scratched the surface but a few good lessons learnt.

Obligatory Owls

Obligatory Owls (headless as I had not levelled the bed properly)

Also a #3DBenchy


First attempt where I forgot to deselect the print supports option.


So far printing other people's designs.

KVM and multipathing

I was curious how to add multipathing to a KVM guest, turns out it is really very simple.

Create two (or more) virtual disks, then depending on whether you run virt-manager or the command line it is just a matter of specifying the same serial number.

virt-manager serial number


qemu-kvm

qemu-kvm ... -device ... -drive if=none,id=sda,file=disk1.img,serial=0001 ... -device -drive if=none,id=sdb,file=disk1.img,serial=0001 ...

Result for libvirt's xml code for KVM guest

 ...
  <devices>
   ...
   <disk type='block' device='disk'>
    ...
    <serial>0001</serial>
    ...
   </disk>
   <disk type='block' device='disk'>
    ...
    <serial>0001</serial>
    ...
   </disk>

gdb and python

Wow no posts since 2015...

I have been messing around in gdb running some python scripts in the embedded python interpreter.

One very useful tip I found was enabling the full python stack trace enabling me to debug the scripts I was running.

To do this use

set python print-stack full

First Post of 2015

Wow heading rapidly to halfway through 2015 without any posts.

Must do something about that.

Small items not enough for full posts

• Fedora 20-21 upgrade finally got around to doing, sar did not run "systemctl disable sysstat" and "systemctl enable sysstat" fixed it, sysstat-collect.timer was not enabled.
• old dangling symlinks in /etc/rc.d/rc* directories probably from a previous upgrade switching to systemd

Recording desktop session in Gnome

I came across this by accident some time ago, but some reminded me of this again today.

Within gnome-shell you can record you desktop session by simply pressing ctrl-alt-shift-r, this will place a red dot in gnome-shell and to stop again press ctrl-alt-shift-r. A file will be created in your home directory called Screencast from .webm.

openwrt on Actiontec DSL modem

Sorting out some old hardware recently I found a number of old DSL modems and routers and I was curious to know which ones I could re-flash.

One of these was an Actiontec GT701-WG DSL modem, which I was surprised to find would actually take openwrt (after a small hiccup) rather easily and has also got me interested in this piece of hardware.

This particular modem uses AR7-SoC and has an ADAM2 bootloader which can be used to flash the device. This bootloader will listen for ftp connections on 192.168.0.1 for a short window immediately after boot, once connected we can use this FTP session to upload and reflash.

The basic steps I found on the following blog entry and consist of

1. power on DSL modem
2. ftp to 192.168.0.1
3. login using adam2/adam2 for username/password
4. set ftp client to use binary and passive
5. set environment variables for mac_port to 0 (to specificy first ethernet port, this modem only has one ethernet port) and creating a new partition which spans the location of the old kernel and filesystem (this detail is listed in openwrt wiki)
6. set MEDIA to FLSH
7. upload new firmware to new partition we created in step 5
8. Reboot modem

The above instructions looked fairly straightforward so I set off to do this, but I forgot one minor detail, I needed to make sure I was allowing connection through firewall on my machine. After spotting this minor detail, things seemed to be going well until upload to modem ground to a halt. I assumed there might be an issue with the firmware on this device so I tried to flash original firmware back onto it using openwrt instructions. This also failed.

At this point the modem failed to boot, however I was still able to get to ADAM2 FTP.
Doing some searching on related models I came across the following setting when connecting to ftp from linux (from openwrt gt704 page)

# echo 0 > /proc/sys/net/ipv4/tcp_frto

I did not recognise this option from the top of my head, this disables F-RTO, an enhanced recovery algorithm for TCP retrans‐mission timeouts.
Once I had disabled this, the steps above worked perfectly.

This modem now has openwrt installed on it :-)

Next steps (for a future blog once I have a few minutes)
1. update version of openwrt
2. run my own custom application/binary on the modem

Pretty print JSON

I really need to spend some time looking at things like JSON, however I came across a quick way to pretty print JSON, using json.tool python module.

example from top of script (or from help once imported json.tool module)

$ echo '{"json":"obj"}' | python -m json.tool
{
"json":"obj"
}

psql - expanded mode

I was looking at some postgresql tables trying to diagnose a problem, the amount of data in the table was making it difficult to see the separate records and not being familiar with what data I needed I could not just select the columns I was interested in.

I had see someone produce some output from a similar table where it was clearer laying out the records and so I was curious how this could be achieved a quick search through the docs mentioned "expanded mode". This can be enabled using

\x on

This gives the output to queries in the following format

select * from random_table;
-[ Record 1 ]------+------------------------------------------
column_1   | some data
column_2   | some other data
-[ Record 2 ]------+------------------------------------------
column_1   | some data 2
column_2   | some other data 2
--------------------------------------------------------------

This was much clearer for me to see what was going on on the terminal.

Launching Javascript from new tab in firefox

I had a piece of javascript that would take a case id as an argument and open a page on a ticketing tool and to make use of it I created a bookmark in firefox and assigned it to a keyword.

This means I could use the keyword with the case id in the url bar and open up the case directly, similar to setting up proxy bookmark.

However in the versions of firefox I was using, when a new tab was opened it loads a new tab which did not allow me to run javascript.

One solution to this is to change what loads in the new tab, preferably you do not want to load a webpage every time a tab is opened.
To set what to open in the new tab, open about:config, click on the "I'll be careful, I promise!" button and then search for "tab" in the search box.
There should be a setting browser.newtab.url, right click on this and select Modify and enter the url you want to have in each new tab.
As mentioned you don't want this to load a webpage every time, so I use about:blank.

First post of 2014

What no posts in 2014 yet.

I have been really busy with work and therefore have not managed to play with much fun stuff outside of work.

I aim to add some posts as often as I can.

gnuplot changing font

I was following some instructions on using gnuplot, however the instructions for changing the font used in the graphs was wrong. Therefore I had to figure out how to achieve this.
I was looking to output some graphs to png images, so I added the following to the top of my command file used to draw the graphs

set terminal png enhanced font "<full path to font>"
set output "<filename>"

There is also an environment variable GDFONTPATH which can be set to directories containing font files, if this is used the above lines can be changed to

set terminal png enhanced font "<font name>"
set output "<filename>"

ioctl decoding

I was interested to know how to decode an ioctl hex string, I worked through the example described in the kernel documentation at /ioctl/ioctl-decoding.txt
Then I tested it out on an ioctl code listed in an error message in a server log, here is my working (this is for an x86_64 server, other architectures may vary)

ioctl code in hex is
cc770002

Which in binary is
11001100011101110000000000000010

First two bits gives us the macro used
11
this according to the document is _IOWR (Read/Write)

The next 14 bits give the size of the arguments
00110001110111
which in decimal is 3191

The next 8 bits are an ascii character, which in my example gives NULL (could well be an issue and why the message appears in the logs for that application)
00000000

The final 8 bits gives the function number, in my example this is
00000010
which in decimal is 2

If I had the sourcecode of the application I could go and look up this function using the character and the function number. In the example above, looks like a bug in the application is generating the wrong character and the ioctl call will fail.

bc setting defaults

I have never really thought much about this, but for years whenever I start bc I always set the number of decimal points (scale) by issuing

scale=<number>

To automatically set this (and other settings) place them into a file (I called mine .bcrc) and set the environment variable BC_ENV_ARGS to point to this file.

BC_ENV_ARGS=$HOME/.bcrc

To set this variable up for every shell add to your shells start up files.

xchat username registration

I am sure I did this many years ago, but failed to remember exactly the steps I took.

I had the need to register my nick with a nick registration service on a particular IRC channel. I have tended to use xchat as it is available and simple to use.
To save me some time and so I don't have to remember to do it every time I thought I would automate this login.

In xchat's network list select the irc server you are connecting to and click edit, which will bring up a screen similar to the following

Edit Network dialog box

In the "Connect command" box I added the command

load -e <filename>

where filename contained the commands I needed to login with the nick registration service (minus the leading "/" is would normally use with interactive commands). The commands were similar to

msg <registration service> login <nick> <password>

Wireless Bridging

I have just started a new job and have discovered that while I have an office, I cannot put everything on wireless, especially the new office IP phone I have been given. Therefore, I decided to combine to projects together, flashing of an old Linksys router with dd-wrt and setting up additional networking in my office so I can connect my office phone.
This seemed the simplest option rather than stringing cables all round the place, another good alternative could have been ethernet over powerline adapters, but no technical tweaking involved.

I followed the dd-wrt instructions for installation on my old linksys WRT54G router, making sure to follow all instructions to the letter. When it came to using tftp to install the new firmware on the system after 99 tries it failed, so I went back to an earlier step where in the Management Mode window a custom image created at the beginning of the process is uploaded. This must not have taken first time as the tftp worked first time.
Once dd-wrt was up and installed I had to connect it to my existing wireless AP, this was straight forward in the wireless settings I select client bridge and give it the details of my existing wireless network (ensure things match exactly). I also made sure that SIP firewall was off and no dhcp server was running on the router.
Once changes had been made and rooter rebooted (as necessary), I connected via ethernet cable a laptop and it successfully got a dhcp lease from the main access point and I was able to browse the web.

The Dark Mod

Having seen the announcement The Dark Mod version 2.0 is completely standalone and was inspired by the Thief series of games which I enjoyed playing (and I am looking forward to the new one that is coming out soon), I decided to give it a try.

There were a few issues to getting it up and running.

Following the instructions I downloaded the updater and set it running, which downloaded all the needed files. However, when I tried to run it, it kept crashing out.
I had hoped this was just due to it being a 32bit binary and I was running it on 64bit. I installed additional 32bit libraries (and their dependencies) that were missing libboost-filesystem, DevIL, mesa-dri-drivers.

This got me a little further, I got a blank screen and could hear music and sound effects. When looking at a copy of the startup messages (passed to tee and dumped to a file), I saw messages relating to not being able to load various font and image files (including ones related to the menu), hence the black screen. I double checked that there were no addition files by re-running the updater, everything was downloaded.
Checking the startup messages and checking the forum, quickly lead me to this page which appeared to have exactly the same symptoms and was suggesting that this was related to S3TC (S3 Texture Compression) support.
There was a suggestion that disabling this in the config might work, however this did not work for me.
Was I missing the 32bit library that contained this support? I was, however as I was testing this on Fedora, Fedora does not ship this particular library as it is patented by S3. I could have gone and downloaded this from a 3rd party repo, but I am not happy at installing lots of extra packages from random locations.
Luckily I came across another solution when using mesa drivers >  version 9 and this is to set the following variable before launching thedarkmod

force_s3tc_enable=true

This worked fine and allowed me to launch the game and start playing through the training mission.

zenity

I have been messing round with zenity to pop up dialog boxes.

One particular script was to remind me to take a break every so often, when it popped up a dialog box it has a random fortune in it. This was fairly easy to get going but there were a couple of strange behaviours in zenity that I had to figure out.

First one was random sizes of dialog box. When using the --info selection of dialog box often the windows it created were off the screen, even with small amounts of text.
This seems to be due to zenity wrapping the text with a fixed width as mentioned in a number of bug postings, so to solve this I used the  --no-wrap option.

Second one is a little more random. I had occasionally noticed my dialog boxes containing the text "All updates are complete", originally I thought that maybe this was a random fortune, however none of the fortune files contains this string.
Doing some further digging, I found out that this is the default message (mentioned here) and that zenity is trying to handle Pango markup and that I should make sure that any of the following characters "&\<>" are escaped prior to passing in.
In later versions of zenity (including the version I am using) there is an undocumented option --no-markup. This now works however I cannot pass "\n" to be interpreted as a new line. Therefore I can either, use this option and explicitly put a line break in the output I want, or pass the output of fortune through sed first and not worry about the no markup option.
The invocation of sed is

sed -e 's/\\/\\\\/g' -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g

There was another issue I came across.
When zenity generates a dialog box it will try to place it above the window where it was launched from, it does this by getting the environment variable $WINDOWID, to prevent this unset this variable in the script before calling zenity.

Sending key presses to specific windows in X

For a project I wanted to send key presses to a specific X Window, so I dug out my code from "Moving mouse and pressing keys in X using python" and made some modifications.
I also included the code I wrote in "screenshots using Python under Linux" for finding a particular window id from its name.


As an example the following code takes a window name as an argument to the program when run, finds the window id and then sends the key press of F11 (the window I wanted to interact with was a browser window and therefore it will go fullscreen).

#!/usr/bin/python
"""
Issue a keypress event in X to a specific window
"""
from Xlib import XK, display, ext, X, protocol
import sys
import time

def get_window(display, name):
    root_win = display.screen().root
    window_list = [root_win]

    while len(window_list) != 0:
        win = window_list.pop(0)
        if win.get_wm_name() == name:
           return win.id
        children = win.query_tree().children
        if children != None:
            window_list += children

    print 'Unable to find window matching - %s\n' % name
    sys.exit(1)


d=display.Display()

win=get_window(d,sys.argv[1])

keysym=XK.string_to_keysym("F11")
keycode=d.keysym_to_keycode(keysym)

event = protocol.event.KeyPress(
   time = int(time.time()),
   root = d.screen().root,
   window = win,
   same_screen = 0, child = X.NONE,
   root_x = 0, root_y = 0, event_x = 0, event_y = 0,
   state = 0,
   detail = keycode
)
d.send_event(win, event, propagate=True)
d.sync()
event = protocol.event.KeyRelease(
   time = int(time.time()),
   root = d.screen().root,
   window = win,
   same_screen = 0, child = X.NONE,
   root_x = 0, root_y = 0, event_x = 0, event_y = 0,
   state = 0,
   detail = keycode
)
d.send_event(win, event, propagate=True)
d.sync()

While this seemed to work (my Firefox browser window went into fullscreen mode), I did start to see some strange behaviour, such as none of the menus would display either from the toolbar or right clicking in the window. I did notice that clicking on a menu did very briefly flash up what looked like an outline, so I wondered if this could have anything to do with focus and modified my code like so.

#!/usr/bin/python
"""
Issue a keypress event in X to a specific window
"""
from Xlib import XK, display, ext, X, protocol
import sys
import time

def get_window(display, name):
root_win = display.screen().root
window_list = [root_win]

while len(window_list) != 0:
win = window_list.pop(0)
#print win.get_wm_name()
if win.get_wm_name() == name:
return win.id
children = win.query_tree().children
if children != None:
window_list += children

print 'Unable to find window matching - %s\n' % name
sys.exit(1)

d=display.Display()

win=get_window(d,sys.argv[1])

keysym=XK.string_to_keysym("F11")
keycode=d.keysym_to_keycode(keysym)

#store current input focus
currentfocus=d.get_input_focus()

#set input focus to selected window
d.set_input_focus(win, X.RevertToParent,X.CurrentTime)

#send keypress and keyrelease
ext.xtest.fake_input(d, X.KeyPress, keycode)
ext.xtest.fake_input(d, X.KeyRelease, keycode)

#revert focus to original window
d.set_input_focus(currentfocus.focus,X.RevertToParent,X.CurrentTime)

d.sync()
d.close()

This seems to work much better and Firefox still works correctly.