tag:blogger.com,1999:blog-20617596900506196082023-06-20T21:33:59.873-07:00ironic cogMy blog about softwarepjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.comBlogger38125tag:blogger.com,1999:blog-2061759690050619608.post-43003201319787871182015-09-06T09:59:00.000-07:002015-09-06T09:59:27.029-07:00Upgrading Fedora Linux 20 (Heisenbug) to 21 & 22<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_4929">
Although I've recently switched to Ubuntu for my work, I'm still using Fedora (dual-boot with Windows 7) on my home workstation. I've been running Fedora 20 ("Heisenbug", horrible name) since mid-2014 so my installation was pretty aged - and with the release of Fedora 22 in May <a href="https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle">has been end-of-life'd since June</a> this year. So it was definitely time for an update.<br />
<br />
When I moved previously to F20 I did a fresh install (although I was able to keep my <span style="font-family: "Courier New",Courier,monospace;">/home</span> from the previous F16 installation). However this time I decided to go with an upgrade using <a href="https://apps.fedoraproject.org/packages/fedup">FedUp</a> (FEDora UPgrader), especially attractive since it promised to also update the installed packages - meaning that I wouldn't need to start over again with installing and configuring all the applications and tools that I use.</div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_4910">
<br /></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_4910">
The upgrade procedure is covered pretty comprehensively in the FedUp documentation at <a href="https://fedoraproject.org/wiki/FedUp">https://fedoraproject.org/wiki/FedUp</a>. As I was two releases adrift I decided to perform the upgrade twice (i.e. going from F20->F21->F22) as recommended. This seemed the safest route, especially given two of the significant changes introduced with each of these realeases, specifically:<br />
<ul>
<li><b>Fedora 21 </b>introduced the idea of "products", which can be thought of as sub-releases specifically targeted for particular user needs. When upgrading from F20, you need to specify which product sub-release you're moving to: the available options are "workstation" (which looks like the best choice for desktop use), "server" or "cloud".</li>
<li><b>Fedora 22</b> switched from the old <span style="font-family: "Courier New",Courier,monospace;">yum</span> package manager to a newer version called <span style="font-family: "Courier New",Courier,monospace;">dnf</span> ("dandified yum") - see <a href="https://fedoraproject.org/wiki/Changes/ReplaceYumWithDNF">https://fedoraproject.org/wiki/Changes/ReplaceYumWithDNF</a></li>
</ul>
</div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5132">
(As an aside, another change is that from F21 the releases will no longer have names - so F21 is just "Twenty One", and F22 is "Twenty Two".)<br />
<br />
The upgrade process for each revision was then: </div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_4952">
<br /></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_4953">
1. Install the <span style="font-family: "Courier New",Courier,monospace;">fedup</span> package:</div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5017">
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo yum install fedup</span></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5175">
<br /></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5018">
2. Perform all updates & backup your system<span id="yiv6622315900yui_3_16_0_1_1440077096763_5299"> (my preference is still for <a href="http://www.clonezilla.org/clonezilla-live.php">CloneZilla Live</a>)</span><br />
<br />
<span id="yiv6622315900yui_3_16_0_1_1440077096763_5299">3. Perform a network-based upgrade (the recommended method over e.g. upgrading from an ISO image):</span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo yum update fedup fedora-release</span></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5298">
<div id="yiv6622315900yui_3_16_0_1_1441021385497_2700">
<span style="font-family: "Courier New",Courier,monospace;">sudo fedup --network 21 --product=workstation</span></div>
</div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5374">
<br />
(As noted earlier, f<span id="yiv6622315900yui_3_16_0_1_1440077096763_5299">or F20-to-F21, you need to specify which product line you are moving to - hence the <span style="font-family: "Courier New",Courier,monospace;">--product</span> option above. </span>For F21-to-F22 the second line can be reduced to simply <span style="font-family: "Courier New",Courier,monospace;">sudo fedup --network 22</span>)</div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5403">
<br />
4. Reboot and select "System Upgrade (fedup)" from the GRUB menu, which will boot into an upgrader environment and then churn through all the packages. (Note that this can take some time: for F20-to-F21 this has a textual display and for me the screen would go black periodically during this process, however moving the mouse seemed to restore the display. For F21-to-F22 there is a nice graphical progress bar; using <span style="font-family: "Courier New",Courier,monospace;">F1</span> toggled between this and the textual display.)<br />
<br />
5. If the previous step completes
successfully then the system will reboot automatically and there should be an option for the new Fedora version in the boot menu. Select this to boot into the new version and check that it looks as you expect.</div>
<br />
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1440077096763_5607">
<div id="yiv6622315900yui_3_16_0_1_1441028363385_2532">
At this point the documentation outlines some post-upgrade clean-up actions that need to be performed manually:</div>
<div id="yiv6622315900yui_3_16_0_1_1441028363385_2533">
<br /></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1441028363385_2665">
1. Rebuild the RPM database:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo rpm --rebuilddb</span><br />
<span style="font-family: "Courier New",Courier,monospace;">sudo yum distro-sync --setopt=deltarpm=0</span></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1441028363385_2667">
<br />
2. Install and run <span style="font-family: "Courier New",Courier,monospace;">rpmconf</span> to identify packages that have more recent versions, and interactively decide what to do for each:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo yum install rpmconf</span></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1441028363385_2668">
<span style="font-family: "Courier New",Courier,monospace;">sudo rpmconf -a</span></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1441028363385_2735">
<br />
(After upgrading to F22 replace the first line with<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo dnf install rpmconf</span><br />
However for me when running the subsequent step, <span style="font-family: "Courier New",Courier,monospace;">rpmconf</span> crashed out with an error about a missing file, so I wasn't able to complete this step.)</div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1441028363385_2735">
<br /></div>
<div dir="ltr" id="yiv6622315900yui_3_16_0_1_1441028363385_2735">
Overall the process was pretty painless, and after a week or so I haven't noticed any issues (other than my continued inability to get a working version of Google Chrome - but that problem predates this upgrade), with everything appearing to work at least as well as before.<br />
<br />
Also while Fedora 22 looks very much like Fedora 20, there have been some visual tweaks which I feel make it a bit nicer to look at than before (however this is perhaps a personal thing - see Chris Duckett's <a href="http://www.zdnet.com/article/a-month-with-fedora-22-leaves-me-hungry-for-23/">A month with Fedora 22 leaves me hungry for 23</a> for a counter-view), and for now I'm enjoying my "refreshed" Fedora experience.</div>
</div>
pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-85648284631830827962015-08-26T13:33:00.001-07:002015-08-26T13:33:27.974-07:00Checking the RAID level with MegaCLI on Ubuntu 14.04I recently <a href="http://ironiccog.blogspot.co.uk/2015/08/experiences-installing-ubuntu-1404-lts.html">installed Ubuntu 14.04 on a new PC</a> (Dell Precision Tower 7910) and wanted to check the RAID configuration that the system had been set up to use. Unfortunately none of the BIOS or system configuration menus seemed to give me access to this information.<br />
<br />
Luckily a colleague pointed me towards MegaCLI, a utility for obtaining information about (and troubleshooting) LSI RAID controllers. Instructions on how to install and use MegaCLI are given in this <i>Nerdy Notes</i> blog post: <a href="http://blog.nold.ca/2012/10/installing-megacli.html" id="yui_3_16_0_1_1439639883730_6709" target="_blank">http://blog.nold.ca/2012/10/installing-megacli.html</a>. However as this dates from 2012, doesn't cover use on Ubuntu, and doesn't include information on how to interpret the outputs from the utility, I've written up what I did below.<br />
<br />
<b>0. Check you're using a MegaRAID controller</b><br />
<br />
Directly from <i>Nerdy Notes</i>: do<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">lspci -nn | grep RAID</span><br />
<br />
and check that the output for <span style="font-family: "Courier New",Courier,monospace;">MegaRAID</span>, e.g.<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">02:00.0 <span style="color: red;"><b>RAID</b></span> bus controller [0104]: LSI Logic / Symbios Logic Mega<span style="color: red;"><b>RAID</b></span> SAS-3 3008 [Fury] [1000:005f] (rev 02)</span><br />
<br />
<b>1. Obtain and install MegaCLI</b><br />
<br />
The first problem is actually obtaining a copy of MegaCLI. According to the <i>Nerdy Notes</i> post, it should be available for download by searching on the LSI website, however when I tried this the download link was non-functional. Subsequently I've discovered that it can be located by searching on the Avago site (go to <a href="http://www.avagotech.com/">http://www.avagotech.com/</a> and enter "megacli" into the search box).<br />
<br />
Opening the downloaded zip file reveals a set of subdirectories with versions of the utility for different OSes, with the <span style="font-family: "Courier New",Courier,monospace;">Linux</span> directory only holding a "noarch" RPM file. This should be straightforward to install using <span style="font-family: Courier New, Courier, monospace;">yum</span> on Redhat-based systems (such as Fedora), but for Ubuntu it's necessary to extract the MegaCLI executables using the <span style="font-family: "Courier New",Courier,monospace;">rpm2cpio</span> utility:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">rpm2cpio MegaCli-8.07.14-1.noarch.rpm | cpio -dimv</span><br />
<br />
(The <span style="font-family: "Courier New",Courier,monospace;">rpm2cpio</span> utility can be installed via the <span style="font-family: "Courier New",Courier,monospace;">rpm2cpio</span> package using Synaptic or <span style="font-family: "Courier New",Courier,monospace;">apt-get</span>.)<br />
<br />
This should pull out the <span style="font-family: "Courier New",Courier,monospace;">MegaCli64</span> executable. Note that it doesn't need to be installed in any special location, you can run it from wherever you extracted it to using the above command, however <b>you need to run with superuser privileges otherwise the output is blank</b>.<br />
<br />
<b>2. Run MegaCli6 to probe the RAID information</b><br />
<br />
To get information on the adapter and see which RAID levels the system supports, do:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo MegaCli64 -AdpAllInfo -aAll</span><br />
<br />
and <span style="font-family: "Courier New",Courier,monospace;">grep</span> the output for "RAID"; this should give you output of the following form:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">RAID Level Supported : RAID0, RAID1, RAID5, RAID00, RAID10, RAID50, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span</span><br />
<br />
Then to get information on which RAID level is actually being used:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo MegaCli64 -LDInfo -Lall -aAll</span><br />
<br />
and again <span style="font-family: "Courier New",Courier,monospace;">grep</span> for "RAID": <br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3</span><br />
<br />
This doesn't explicitly name a "standard" RAID level, but I found this ServerFault post which gives details on how to interpret the output:<a href="http://serverfault.com/questions/385796/how-to-interpret-this-output-from-megacli" id="yui_3_16_0_1_1439639883730_6710" target="_blank"> http://serverfault.com/questions/385796/how-to-interpret-this-output-from-megacli</a><br />
<br />
In this case "Primary-5, Secondary-0, RAID Level Qualifier-3" turns out to be equivalent to RAID 5 <br />
(<a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5" id="yui_3_16_0_1_1439639883730_6711" target="_blank">https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5</a>), which is suitable for my needs - so I was able to rest a bit easier.<br />
<br />
<i>I'd like to acknowledge my colleague Ian Donaldson for tipping me off about MegaCLI.</i>pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-61779364597095828812015-08-24T12:42:00.003-07:002015-08-24T12:42:56.915-07:00Enabling correct monitor drivers on Ubuntu 14.04 LTSHaving <a href="http://ironiccog.blogspot.co.uk/2015/08/experiences-installing-ubuntu-1404-lts.html">recently installed Ubuntu 14.04 LTS on a new PC</a>, I had some troubles configuring the display to operate correctly with a Dell 24" monitor: specifically, the monitor aspect ratio of 1920x1200 (16:10) didn't seem to be supported by the default install from the Live CD. In fact it was more than a little unstable (to say the least: for example, attempting to change the display aspect ratio rendered the display unusable several times, necessitating a complete reinstall to recover).<br />
<br />
This appeared to be a missing driver issue, but it was difficult to find out what drivers were needed. Searching on the web for the specific problem was rather fruitless, and the many suggestions to look at <span style="font-family: "Courier New",Courier,monospace;">xrand</span> (for example <a href="https://wiki.ubuntu.com/X/Config/Resolution">https://wiki.ubuntu.com/X/Config/Resolution</a>) didn't work for me either. In the end I stumbled across a post which suggested using the command:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">ubuntu-drivers devices</span><br />
<br />
from a terminal, which shows all devices which need drivers, and which packages apply to them. For example for my system the output looks like:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">== /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0 ==</span><br />
<span style="font-family: "Courier New",Courier,monospace;">vendor : NVIDIA Corporation</span><br />
<span style="font-family: "Courier New",Courier,monospace;">modalias : pci:v000010DEd000013BAsv000010DEsd00001097bc03sc00i00</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-346-updates - distro non-free</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-346 - third-party free</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-340 - third-party free</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-352 - third-party free</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : xserver-xorg-video-nouveau - distro free builtin</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-349 - third-party non-free</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-355 - third-party free recommended</span><br />
<span style="font-family: "Courier New",Courier,monospace;">driver : nvidia-340-updates - distro non-free</span><br />
<br />
When I ran this it marked 'nvidia-349' as the recommended driver (nb the output above is from a subsequent run of <span style="font-family: "Courier New",Courier,monospace;">ubuntu-drivers</span> and doesn't show the mark). This was easily installed using:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">apt-get install nvidia-349</span><br />
<br />
and appeared to solve my issues. Unfortunately I haven't been able to find the post again that originally suggested this, however if you are experiencing similar resolution issues then it might be worth a try before grappling with <span style="font-family: "Courier New",Courier,monospace;">xrand</span>.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-32549261198466149732015-08-23T12:05:00.000-07:002015-08-23T12:09:05.421-07:00Experiences installing Ubuntu 14.04 LTS on UEFI system with Boot RepairI recently set up a new desktop PC (Dell Precision Tower 7910) for my work computer. For this new machine I've decided to move away from Fedora Linux (which I've used for the past four years) and go with the latest <a href="http://www.ubuntu.com/">Ubuntu</a> LTS release (<a href="http://releases.ubuntu.com/14.04/">14.04 "Trusty Tahr"</a>). (While I've enjoyed using Fedora, the relatively high turnover of releases has been a nuisance - so the thought that I'd only need to do one install over the lifespan of this machine was an attractive one.)<br />
<br />
Although this was a new machine with Windows pre-installed, I decided to trash this and install Ubuntu over it as a single boot setup(my preference is to run Windows inside a virtual machine on the Linux box, rather than dual booting). The system was also already set up to use UEFI (Unified Extensible Firmware Interface, a replacement for traditional BIOS firmware), however according to the official Ubuntu documentation for UEFI (<a href="https://help.ubuntu.com/community/UEFI">https://help.ubuntu.com/community/UEFI</a>) for a single boot system either UEFI or "legacy" boot modes can be used.<br />
<br />
So if you don't really care which mode is used then the section on <a href="https://help.ubuntu.com/community/UEFI#Installing_Ubuntu_for_Single_Boot_with_a_Random_Boot_Mode">Installing Ubuntu for Single Boot with a Random Boot Mode</a> suggests that the easiest approach is basically just try and install from the Live media, and see what happens. This was the approach that I took, installing from the Live CD and setting up my own partitioning scheme with <span style="font-family: "Courier New",Courier,monospace;">/home</span> separated from the rest of the system.<br />
<br />
Initially attempting to reboot post-installation failed with <span style="font-family: "Courier New",Courier,monospace;">no bootable devices</span>. However it's possible to repair this quite easily and complete the installation using the <a href="https://help.ubuntu.com/community/Boot-Repair">Boot Repair</a> utility. The basic recipe for obtaining and running it is summarised in the post at <a href="http://askubuntu.com/a/604623">http://askubuntu.com/a/604623</a> - but unfortunately the version from the repo given (<span style="font-family: "Courier New",Courier,monospace;">yannubuntu/boot-repair</span>) didn't seem to work for Ubuntu 14.04 - instead it's necessary to get it from a different location (<span style="font-family: "Courier New",Courier,monospace;">kranich/cubuntu</span>; see <a href="http://ubuntuforums.org/showthread.php?t=2277484">http://ubuntuforums.org/showthread.php?t=2277484</a>).<br />
<br />
Based on the above posts, I restarted from the Live CD and selected "Try Ubuntu" to boot from the CD. Once the system was ready I opened a terminal window and did:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">sudo add-apt-repository ppa:kranich/cubuntu</span><br />
<span style="font-family: "Courier New",Courier,monospace;">sudo apt-get update</span><br />
<span style="font-family: "Courier New",Courier,monospace;">sudo apt-get install -y boot-repair</span><br />
<span style="font-family: "Courier New",Courier,monospace;">sudo boot-repair</span><br />
<br />
which brings up the <i>Boot Repair</i> GUI. I selected the "Recommended Repair" option, and once this had completed I restarted the system again without the Live CD. This time Ubuntu booted okay directly from the hard drive, and the installation was complete.<br />
<br />
<b>Addendum:</b> it looks like the system ended up being installed in UEFI mode, which <a href="https://help.ubuntu.com/community/UEFI#Identifying_if_the_computer_boots_the_HDD_in_UEFI_mode">according to the Ubuntu docs</a> is indicated by the existence of the <span style="font-family: "Courier New",Courier,monospace;">/sys/firmware/efi/</span> directory on the hard drive.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-13888834342935007112015-08-15T05:44:00.003-07:002015-08-15T05:46:41.756-07:00Keeping a GitHub fork up-to-date with the original repo<a href="https://guides.github.com/activities/forking/">Forking</a> is a standard way to make changes to a third party repository on GitHub. Typically a developer makes a fork (essentially a clone on the GitHub server side) of someone else's repo in order to make their own changes to the project. These changes might be for private use, or they could subsequently be submitted back to the original repo for evaluation and possible inclusion via the <a href="http://oss-watch.ac.uk/resources/pullrequest">pull request</a> mechanism.<br />
<br />
However: once the fork is created, any further commits or other changes
made to the original repo will not automatically be reflected in the
fork. This post describes how to keep the fork up-to-date with the original repository (which is normally referred to as the <b>upstream repo</b>) by pulling in the changes manually via
a clone of the fork, using the process described below.<br />
<br />
The update procedure is:<br />
<br />
<b>0. Make a clone of your fork</b> onto your local machine (or use one you've already made), and move into the cloned directory.<br />
<br />
<b>1. Add a new remote repository to the clone</b> which points to the upstream repo. The general syntax to do this is:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPO.git</span><br />
<br />
A <i>remote</i> is simply a version of the project which is hosted on the Internet or elsewhere on the network, and a repo can have multiple remotes (use <span style="font-family: "Courier New",Courier,monospace;">git remote -v</span> to list all the ones that are defined; you should that <span style="font-family: "Courier New",Courier,monospace;">origin</span> is also a remote). Conventionally the remote used for this syncing is given the name <span style="font-family: "Courier New",Courier,monospace;">upstream</span>, however it could be called anything you like.<br />
<br />
For example:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">git remote add upstream https://github.com/fls-bioinformatics-core/genomics</span> <br />
<br />
This step only needs to be done once for any given clone.<br />
<br />
<b>2. Update the local clone</b> by fetching and merging in any changes from the upstream repo, using the procedure:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">git fetch upstream<br />git checkout master<br />git merge upstream/master</span><br />
<br />
If you have made your changes in your own dedicated branches and avoided making commits to the fork's <span style="font-family: "Courier New",Courier,monospace;">master</span> branch then the upstream changes should merge cleanly without any conflicts. (It's considered best practice that changes
made to the fork should always be done in a purpose-made branch<span style="font-family: "Courier New",Courier,monospace;">;</span> working directly in <span style="font-family: "Courier New",Courier,monospace;">master</span> makes it harder to make clean pull requests and can cause conflicts when trying to merge the upstream changes into your fork.)<br />
<br />
<b>3. Push the updated <span style="font-family: "Courier New",Courier,monospace;">master</span> branch back to your fork</b> on GitHub using:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">git push origin master</span><br />
<br />
Once again, if you have avoided committing local changes to <span style="font-family: "Courier New",Courier,monospace;">master</span> then the push should be drama-free.<br />
<br />
Finally, you will need to repeat steps 2 and 3 in order to stay up to date whenever new changes are made to the upstream repo.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-39205794181023204862013-12-09T08:42:00.003-08:002013-12-09T11:36:10.109-08:00Book Review: "Instant Flask Web Development" by Ron DuPlain"Instant Flask Web Development" by Ron DuPlain (Packt Publishing <a href="http://www.packtpub.com/flask-web-development/book">http://www.packtpub.com/flask-web-development/book</a>) is intended to be an
introduction to Flask, a lightweight web application framework written
in Python and based on the Werkzeug WSGI toolkit and Jinja2 template
engine. <br />
<br />
The book takes a tutorial style approach, building
up an example appointment-management web application using Flask and
introducing various features of the framework on the way. As the example
application becomes more complicated, additional Python packages are
covered which are not part of the Flask framework (for example SQLAlchemy for managing interactions with a
database backend, and WTForm for handling form generation and
validation) along with various Flask extensions that can be used for
more complicated tasks (for example managing user logins and sessions). The final section of the book gives an overview of how to deploy the
application in a production environment, using gunicorn (a Python WSGI server) and nginx.<br />
<br />
Given
its length (just short of 70 pages) the book is quite ambitious in the
amount of ground that it attempts to cover, and it's quite impressive
how much the author has managed to pack in whilst maintaining a light
touch with the material. So while inevitably there is a limit to the level of detail that
can be fitted in, there are some excellent and concise overviews of
many of the topics that could act as excellent starting points for more
experienced developers (for me the section on SQLAlchemy is a particular
highlight). Overall the pacing of the book is also quite sprightly and
conveys a sense of how quickly and easily Flask could be used to build a
web application from scratch.<br />
<br />
The flipside of the
book's brevity is that it cannot possibly contain everything that a
developer needs to know (although this is mitigated to some extent by
extensive references to online documentation and resources). In this
regard it is really more a showcase for Flask, and is best viewed as a
good starting point for someone wishing to quickly get up to speed with the framework's potential. I'd also question how suitable this is for
newcomers to either Python, or to web programming in general - I felt that some of the concepts and example code (for example the sudden appearance of a feature implemented using Ajax) might be a bit of a stretch for a novice. Also there are some occasional frustrating glitches in the text and example code which meant
it took a bit of additional reading and debugging in places to get the
example application working in practice.<br />
<br />
In summary
then: I'd recommend this book as a good starting point for developers
who already have some familiarity with web application development, and
who are interested in a quick introduction to the key components of Flask and how they're used - with the
caveat that you will most likely have to refer to other resources to get
the most out of it. <br />
<br />
<span style="font-style: italic;">Disclosure: a free e-copy of this book
was received from the publisher for review purposes; this review has
also been submitted to Amazon. The opinions expressed here are entirely my own.</span>pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-69069786576774493042013-04-19T12:27:00.000-07:002013-04-23T11:25:31.129-07:00Software Carpentry Bootcamp ManchesterI've spent the last two days as a helper at a <a href="http://software-carpentry.org/bootcamps/2013-04-manchester.html">Software Carpentry Bootcamp held at the University of Manchester</a>, and it's been really interesting and fun. <a href="http://software-carpentry.org/">Software Carpentry</a> is a volunteer organisation and runs the bootcamps with the aim of helping postgraduate students and scientists become more productive
by teaching them basic computing skills like program design, version
control, testing, and task automation. Many of the materials are freely available online via the <a href="https://github.com/swcarpentry/boot-camps/blob/2013-04-manchester/README.md">bootcamp's Github page</a>: along with transcipts of some of the tutorials there are some excellent supporting materials including hints and tips on <a href="https://github.com/swcarpentry/boot-camps/blob/2013-04-manchester/HintsAndTips.md">common Bash and editor commands</a> (there's even more on the main Software Carpentry website).<br />
<br />
The bootcamp format consisted of short tutorials alternating with hands-on practical
exercises, and as a helper the main task was to support the instructors by offering assistance to participants if they found themselves stuck for some reason in the exercises. I'll admit I felt some trepidation beforehand about being a helper, as being put on the spot to debug something is very different to doing it from the relaxed privacy of my desk. However it turned out to be a both very enjoyable and very educational experience; even though I consider myself to be quiet a proficient and experienced shell and Python programmer, I learned some new things from helping the participants both with understanding some of the concepts and with getting their
examples to work.<br />
<br />
There were certainly lots of fresh insights and I learned some new things from the taught sessions too, including:<br />
<ul>
<li><b>Bash/shell scripting:</b> using <span style="font-family: "Courier New",Courier,monospace;">$(...)</span> instead of "backtick" notation to execute a command or pipeline within a shell script;</li>
<li><b>Version control: </b>learning that <a href="https://bitbucket.org/">Bitbucket</a> now offers free private repositories (and a reminder that <span style="font-family: "Courier New",Courier,monospace;">git push</span> doesn't automatically push tags to the origin, for that you also need to explicitly use <span style="font-family: "Courier New",Courier,monospace;">git push --tags</span>);</li>
<li><b>Python: </b>a reminder that <a href="http://stackoverflow.com/questions/509211/the-python-slice-notation">slice notation</a> <span style="font-family: "Courier New",Courier,monospace;">[i:j]</span> is inclusive of the first index i but exclusive of the second index j, and independently that string methods often don't play well with Unicode;</li>
<li><b>Testing:</b> a reminder that writing and running tests doesn't have to impose a big overhead - good test functions can be implemented just with assert statements, and by observing a simple naming convention (i.e. put tests in a <span style="font-family: "Courier New",Courier,monospace;">test_<module>.py</span> file, and name test functions <span style="font-family: "Courier New",Courier,monospace;">test_<name></span>), Python <a href="https://nose.readthedocs.org/en/latest/">nose</a> can run them automatically without any additional infrastructure.</li>
<li><b>Make:</b> good to finally have an introduction to the basic mechanics of Makefiles (including targets, dependencies, automatic variables, wildcards and macros), after all these years!</li>
</ul>
As a helper I really enjoyed the bootcamp, and from the very positive comments made by the participants both during and at the end it sounded like everyone got something valuable from the two days - largely due to the efforts of instructors Mike Jackson, David Jones and Aleksandra Pawlik, who worked extremely hard to deliver excellent tutorials and thoroughly deserved the applause they received at the end. (Kudos should also go to Casey Bergman and Carole Goble for acting as local organisers and bringing the bootcamp to the university in the first place.) Ultimately the workshop isn't about turning researchers into
software engineers but rather getting them started with practices and tools that will support their
research efforts, in the same way that good laboratory practices support
experimental research. (This isn't an abstract issue, there can be very
real consequences as demonstrated by cases of <a href="http://en.wikipedia.org/wiki/Geoffrey_Chang">Geoffrey Chang</a>, <a href="http://crookedtimber.org/2004/08/25/mckitrick-mucks-it-up/">McKitrick and Michaels</a>, and the <a href="http://www.around.com/ariane.html">Ariane 5 rocket failure</a> - the latter resulting in a very real "crash".)<br />
<br />
If any of this sounds interesting to you then the Software Carpentry <a href="http://software-carpentry.org/bootcamps/">bootcamp calendar</a> shows future events planned in both Europe and the US, so it's worth a look to see if there's one coming up near your location. Otherwise you could consider hosting or running your own bootcamp. Either way I'd very much recommend taking part to any researchers who want to make a positive impact on their work with software.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-55791036859274287932012-08-19T08:54:00.000-07:002012-10-24T12:45:41.290-07:00Inline images in HTML tags with PythonI recently discovered a neat trick for embedding images within HTML documents, really useful if you've got an application where you would like the HTML files to be portable (in the sense of being moved from one location to another) and not have to rely on also moving a bunch of related image files.<br />
<br />
Essentially the inlining is achieved by base64 encoding the data from the image file into an ASCII string, which can then be copied into the src attribute of an <img> tag with the following general syntax:<br />
<pre style="background-color: beige; overflow-y: auto; overflow: scroll; width: 450px;"><span style="font-size: 85%;">
<img src="data:image/<i><b>image_type</b></i>;base64,<i><b>base64_encoded_string</b></i>" />
</span>
</pre>
For example to embed a PNG image:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
<img src="data:image/png;base64,iVBORw...." /></div>
<br />
i.e. <i>image_type</i> is <span style="font-family: "Courier New",Courier,monospace;">png</span> and <span style="font-family: "Courier New",Courier,monospace;">iVBORw...</span> is the base64 encoded string (truncated here for readability).<br />
<br />
If you're familiar with Python then it's straightforward to encode any file using the base64 module, e.g. (for a PNG image):<br />
<pre style="background-color: beige; overflow-y: auto; overflow: scroll; width: 450px;"><span style="font-size: 85%;">
>>> import base64
>>> pngdata = base64.b64encode(open("cog.png",'rb').read())
>>> print "<img src='data:image/png;base64,%s' />" % pngdata
</span>
</pre>
And here's an example inlined PNG generated using this method:<br />
<br />
<div style="text-align: center;">
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdwnLpRPAAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAABqdJREFUWMPVl3tM1WUYx9/DRcEJC/KCGqSAIIhyv6iIRy6hklqKFy7ihQOK5iW1pamoxUUQjqLC4aKmW7Ny67Iu09VM19amtlZtTsucKSPNSzkjSTQ9fZ7jOfTjAHkp/+jd3u3s/T3v83yf73N7j1L/txX8/KsqdekeFZVd6Rc3p0ovOzq7MvCDs2bLt/98DZ3ymhoyaX13lLtjSD0ZY1DGQ9fVqNwdFYkLG27Jjs/dUYuozjtxhRo+vVSAuAdN3tBj2LQSpVx9Hs1wyNQi5Tv+FYeIzM0hI+dt3zly7rY9ERnlkSPmVOXFG6pr9QvqziS/sMssm9/nOGsYMXdbATIRyJi4sz8yc3NswMRCR3HioVd4RrkKm7EpcnRezemkRTvNeNo6Zn7tFX7fsRm233y7C5iryP6RzJ2EfNMFAKXG5Gx5SOvQhnE1LL04AABnuzJ4vw3gS+iJggml84x5cOPQroZPK+kzat72vXhzuxNP74wtqL/ObmI3sn/pTI6zP8mVt0Onlw4gaZWzd1LXdomVGvjMyw5QlgjqaIy/LtRrFWKomVh/SObnx8wyxkRlVfiyB5KcoXGzt07H2C5CcFF7T34LCHSGozsl6LmNzuyOAMRrhIKh/Aeokzi2apVwfjx6lnFi4KT1rpHZRuVgR+mQyRuUT9JKR+iOwOB+qQ4NE7fQ+bPkBN9HSE54xhW0L7Xe8YudyGKTvddWDz6ivAbFzq4SlpzEm9icLcvJ9Hq+7eVeJRTPRI+XfmEDZbjRjbMyrRO2LeAGp61zlSprB4AQuICsHJpbtBeg/Bj54At9IueFYeOYfNNllFvA2Tb3bqP8K5hM7zt6iWPgxMIeAGzQOsSdm5RoDX3CjQS3CwEGwmeWBUHVBU3Mf4P2Z4m5NJe+XH5PDCdiDGBf0hu2sysxfGTsgjoLcMBdi8yqyFtsOioh9YX2E5p+cRk7oVIV7Za0UGKsMJSNgbY6H03C+U9Y46p6pehgRygVJb+TdIXEvJfZbFbjV72jBqet7SlGMXZJ7iXk1VzAeBzeK0KxysaC9AmcMOBk+7YtCWQwHla0VKM29ggX4J2ijIaivEkAAKTCK2Gps9yxLUAqAUNFLIK1W3IXZxq+vWs5i+DsqiakpqMtZtV2HwHZ3hjapm2v0NyMsVF4oCi5XDEOvU3kQxCAOlSRZWbAioTGysIpEs0LeQ88/loThh+RqSO0wZYOSVLJDgPlr3Y1fwmhIAwLjUWWUjRUH2Y+9Ohs6olHwoLMDauhK8Q7ZFDqqm4AOGTXoCQZk5H9RwBXABBiBVBsBfCZ77jVrvcB0GAFcFUA+E1Y0x0AR7oEQPLIHkDnK0HwlEaoBWDJnCuoypcQkAeNUBrQVQhoUh7Qe9QSgnzT94SgX+iMTb3182tPakJwhnBXksgBhPbeZS6qnNKDMuM3J/+dsWZKbLVkLEqGo/CiJQlnGYvdInId5Y5t+Y5frZbUHRdHcinHm9YE3iOMwGKClLM2CY9pk7CtDA3VUoYZMkBswiTSF0OnFnkqv3QH2NhqScT5tdfpeitoXO4py/ep8MxKRVhcaECZyDdZvb9MT0mQyoJm433LUBYXpHFII/pJ28NjZ28tGLdsr1A8gFw4KCDwqBW2PkdZCXsDYfoYaputI7iZqlpyoMmsaDgxeo0+a2JGSNi7B2V0bMVMtCqU37Sb6edRlBBvqJFXkjeA6ji7pm3D1gfLHeg9iWxOP/0yJ3KlP33lE7tWfBsn3sB7jw6zoEfYXAe8MdoPIyulJ/EqSXnodTSdbrA1EiDrYWEfRt8lV+qJtYH+7p1VfMDSgvn2vg2cVhcheZPKcGkHwDaOUexPSz6Bh02JdkzInAfgurCZZU97RufriHvbXffQOSqn4pC8oJ4gV7IB/I2d562cnZcqkrBI9rv4pLYH4J+2Rvkkv6QDRAxlFgRSk3Yu2F5CKDmNd/V4nQegCexUaM3irJxvx6XG7d+J6NpNSAKJfwKPEad2FdDZkoTEG08BIQnX1QPU+ixvFWCdhU5ijo5d6Ool01b1SVUPvKBahlAwpdX4Lx6lUo5RHcbvgywZUlyOpT98Z31wtDCMGsXrzry1PUIpO3moNifde5afw3hS7EM/y9W9R6pbpEEHE4E0qGqy3kR2B1Cq6cS7dEwn7RW5LIsM/UGoJ7GH9de/qNN2zUcC4p+21jlwUmHP4ClF6qm4HJVT9qm07U1Uyg28vkG975C265e60jaWXbnX7bH8VxSlKYt3S4h8YCVGtjzN3zrxmP6cPu71F6Ga+Xb8q/hVAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDEyLTA4LTE5VDE2OjMyOjM5KzAxOjAwBAPjlQAAACV0RVh0ZGF0ZTptb2RpZnkAMjAxMi0wMi0xMlQxNDoyMDowMyswMDowMGX07p4AAAAASUVORK5CYII=" />
</div>
<br />
In fact this inlining is an example of the general data URI scheme for embedding data directly in HTML documents, and can also be used for example in CSS to set an inline background image - the Wikipedia entry on the <a href="http://en.wikipedia.org/wiki/Data_URI_scheme">"Data URI scheme"</a> is a good place to start for more detailed information.<br />
<br />
There are some caveats, particularly if you're interested in cross-browser compatibility: older versions of Internet Explorer (version 7 and older) don't support data URIs at all, while version 8 only supports encoded strings up to 32KB. More generally the encoded strings can be around 1/3 larger than the original images and are implicitly downloaded each time the document is refreshed; these are definitely considerations if you're concerned about bandwidth. However if these aren't issues for your application then this can be a handy trick to have in your toolbox.<br />
<br />
Update 24/10/2012: another caveat I've discovered since is that at least some command line HTML-to-PDF converters (for example <a href="http://code.google.com/p/wkhtmltopdf/">wkhtmltopdf</a>) aren't able to convert the encoded images, so it's worth bearing this in mind if you plan to use them. (On the other hand the PDF conversion in Firefox - via "Print to file" - works fine but can't be run from the command line AFAIK.)pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com1tag:blogger.com,1999:blog-2061759690050619608.post-34667489528351893452012-04-22T05:30:00.000-07:002012-04-22T08:00:00.843-07:00Installing Fedora 16 (Verne) dual-boot with Windows 7Back in February I finally got around to installing Fedora 16 (Verne) as a dual-boot option on my Windows 7 home machine. I was helped considerably by an impressively detailed tutorial on <a href="http://www.linuxbsdos.com/2011/06/27/how-to-dual-boot-fedora-15-and-windows-7/">How to dual-boot Fedora 15 and Windows 7</a> posted on the <a href="http://www.linuxbsdos.com/">LinuxBSDos.com</a> blog, which was invaluable for the tricky steps of partitioning and boot loader set up.<br />
<br />
In this post I give an overview of what I did for my own dual-boot installation, with some hints and tips that I picked up in the process.<br />
<br />
<b>Preparation: back up the existing system</b><br />
<br />
My first step was to make an image of the existing system on an external hard drive so that I could recover everything if the installation should fail for any reason. To do this I used <a href="http://www.clonezilla.org/clonezilla-live.php">Clonezilla Live</a>, which can be burned onto a CD or other bootable media (I used the 1.2.12-10 AMD64 iso). I found the Clonezilla website quite difficult to navigate but there are very detailed usage instructions at <a href="http://www.clonezilla.org/clonezilla-live-doc.php">http://www.clonezilla.org/clonezilla-live-doc.php</a>, and while Clonezilla Live's menu-driven structure might feel a bit retro to those used to a slicker UI, the program itself turns out to be very easy to use.<br />
<br />
Note that the size of the resulting image file (and hence the amount of space required on the external drive) depends on how much of the disk actually contains data, rather than the size of the disk or partition being imaged. So although the Windows partitions had been allocated the whole of my 1 TB hard drive, in this case only about 70 GB was in use and the corresponding image file was even smaller at around 45 GB.<br />
<br />
<b>Repartitioning: create space for the Fedora installation</b><br />
<br />
Having created the image the next step was to free up some space by shrinking the size allocated to the Windows partitions (the LinuxBSDos dual-booting tutorial assumes there's already some free space available for the Fedora install). To do this I used <a href="http://gparted.sourceforge.net/livecd.php">GParted Live</a> (version 0.11.0-10), a free partition editor that runs from bootable media and works with 64-bit systems. Launching the GParted application from the bootable CD is covered at <a href="http://gparted.sourceforge.net/display-doc.php?name=gparted-live-manual">http://gparted.sourceforge.net/display-doc.php?name=gparted-live-manual</a> and there is comprehensive documentation on how to use it (with screenshots) at <a href="http://gparted.sourceforge.net/display-doc.php?name=help-manual">gparted.sourceforge.net/display-doc.php?name=help-manual</a>. However I think the user interface is pretty intuitive on its own. For my installation I reduced the Windows allocation down to 50% of the disk space, leaving around 500 GB of unallocated space.<br />
<br />
<b>Installing Fedora 16</b><br />
<br />
I was now in a position to install Fedora into the free space. I downloaded and burned the Fedora 16 installable live media (aka Live Desktop CD) from <a href="http://fedoraproject.org/en/get-fedora">http://fedoraproject.org/en/get-fedora</a> (make sure you get the appropriate 32- or 64-bit version for your system), which also allows you to try out Fedora without installing (see my previous post on <a href="http://ironiccog.blogspot.co.uk/2011/07/fedora-15-and-gnome-3-user-basics.html">Fedora 15 and Gnome 3 user basics</a> if you're unfamiliar with the Gnome 3 desktop). <br />
<br />
Detailed information about the installation procedure can be found in the extensive <a href="http://docs.fedoraproject.org/en-US/Fedora/16/html/Installation_Guide/index.html">Fedora 16 Installation Guide</a> but in summary this is what I did:<br />
<ul>
<li>Booted the system from the Fedora Live Desktop CD and started the installer via <b><i>Activies/Applications/Install to Hard Drive</i></b>.</li>
</ul>
<ul>
<li>Clicked through the installer screens and set up keyboard layout, computer name, network setup, computer name, and timezone etc (generally if you're installing a vanilla home desktop system then the default options
should be okay) until I reached the <b><i>Type of installation</i></b> screen. Here I needed to tell the installer where to put Fedora:<br />
<ul>
<li>Choose the <b><i>Use Free Space</i></b> option to use the unallocated disk space, and</li>
<li>Make sure the <i><b>Review and modify partitioning layout</b></i> option is checked. </li>
</ul>
before clicking through to the next screen.</li>
</ul>
<ul>
<li>The next step was to modify the assigned logical volumes to my preferred layout (here the detailed instructions in the LinuxBSDos blog post were invaluable; I strongly recommend consulting them if you're unfamiliar with logical volumes and partitions). For my installation I changed the Fedora defaults to set up the following new partitions:<br />
<ul>
<li><b>lv_root</b> (/): 10 GB</li>
</ul>
<ul>
<li><b>lv_home</b>:</li>
<ul>
<li><span style="font-family: "Courier New",Courier,monospace;">/home</span>: 75 GB</li>
<li><span style="font-family: "Courier New",Courier,monospace;">/boot</span>: 500 MB</li>
</ul>
<li><b>lv_swap</b>: 16 GB</li>
</ul>
Generally it's recommended to only use the space you need for the new installation and leave unused capacity unallocated; keeping <span style="font-family: "Courier New",Courier,monospace;">/home</span> separate from the root partition means that I can share user directories between multiple Linux installations in future; and the recommendation is that swap should be allocated around twice the size of the system RAM. However more elaborate partitioning schemes can be created, with various pros and cons - there's useful information on this in the <a href="http://docs.fedoraproject.org/en-US/Fedora/16/html/Installation_Guide/s2-diskpartrecommend-x86.html">Recommended Partitioning Scheme</a> section of the Fedora documentation.</li>
</ul>
<ul>
<li>Having set up the partition layout, I then needed to change the location of the Fedora boot loader (GRUB) from the default of <b><i>Master Boot Record</i></b> to <b><i>First sector of boot partition</i></b>. The reason for this is that any changes that are made to the Master Boot Record (MBR) (such as installing GRUB) are liable to be overwritten by future Windows updates, so it's better to manage the boot options from within Windows after Fedora has been installed:<br />
<ul>
<li>Click <i><b>Change Device</b></i> and select <i><b>First sector of boot partition</b></i> (I'd recommend verifying that the correct location is shown before clicking <b><i>Next</i></b> to complete the installation, and also making a note of the boot loader location for reference later).</li>
</ul>
</li>
</ul>
At this point I was finished and the installation could complete, which was a relatively quick process (in my case it took around 5 minutes).<br />
<br />
<b>Configuring the Windows boot manager</b><br />
<br />
Once the installation process had completed the computer rebooted directly into Windows, so to be able to access Fedora I needed to update the Windows boot manager: for this I used <a href="http://neosmart.net/EasyBCD/">EasyBCD</a> 2.1.2 (EasyBCD is a Windows bootloader editor freely available from <a href="http://neosmart.net/">NeoSmart Technologies</a>) as recommended in the LinuxBSDos.com tutorial. Once installed in Windows I was able to add a new entry for Fedora 16:<br />
<ul>
<li>Click on the <i><b>Add New Entry</b></i> tab</li>
<li>Select the <b><i>Linux/BSD</i></b> tab and select the partition that the GRUB boot loader was installed to (which I made a note of earlier)</li>
<li>Select <i><b>Edit Boot Menu</b></i> to view the change</li>
</ul>
Then restarting the system then gave me the option of booting either Windows or Fedora 16.<br />
<br />
(Note that using this method, when Fedora 16 is selected you get the Linux GRUB boot loader rather than just booting directly into Linux; however it should be possible to configure GRUB to boot Fedora immediately.)<br />
<br />
<b>Post-installation Fedora set-up</b><br />
<br />
Once Fedora 16 was booted for the first time, there were a few standard post-installation steps to complete: agreeing to the user license, setting up the date and time - usually I select the option to synchronise over the network - and creating a non-root user account. I also ran the <i><b>Software Update</b></i> application to pull in any updates (this also fixed the settings for my monitor, which initially weren't properly configured). It was then possible to start building up the software inventory and configuring the system to my personal preferences.<br />
<br />
The final thing I did was to make donations to the Clonezilla, GParted and EasyBCD projects, as I would have been unable to set up my dual-boot system without them. It's also worth taking a moment to acknowledge the efforts of the Fedora community and the wider Linux community for making a complete operating system available and easy to install at no cost to the end user.<br />
<br />
For my part I hope that this overview will be useful to someone else - and good luck with your own dual-booting installations.<br />
<ul>
</ul>pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-53896282017335793872011-11-30T11:47:00.000-08:002011-11-30T13:11:41.721-08:00Firefox add-ons for the occasional web developerI'm not a hardcore web developer but I do some occasional web-based work, and one of the issues I have is that - because web applications exist in an environment which spans both browser and server (and which often seems to hide the workings of its components) - it can be quite difficult to see under the hood when there are problems.<br /><br />Fortunately there are a number of adds-on for Firefox (my browser of choice) that can help. These are the ones that I like to use:<br /><ul><li><span style="font-weight: bold;">Firebug</span> <a href="http://getfirebug.com/">http://getfirebug.com/</a> is possibly one of the most essential add-ons for web development. I first came across it as a Javascript logger and debugger, but it's far more than that: describing itself as a complete web development tool, its functionality extends to HTML and CSS in additional to script profiling and network analysis capabilities. As an occasional user I've found the Javascript debugging functions invaluable, and the ability to edit CSS in-place and see the results immediately has also been really helpful in debugging style sheets - in fact its biggest downside from my perspective is that it's not immediately obvious how to use many of its functions.</li></ul><ul><li><span style="font-weight: bold;">Live HTTP Headers</span> <a href="http://livehttpheaders.mozdev.org/">http://livehttpheaders.mozdev.org/</a> is a great tool for exposing the interactions between your browser and a server. I found this invaluable when I was debugging some website functionality that I was developing earlier this year, as it enabled me to follow a redirect that I'm sure I couldn't have seen otherwise.</li></ul><ul><li><span style="font-weight: bold;">QuickJava </span><a href="http://quickjavaplugin.blogspot.com/">http://quickjavaplugin.blogspot.com/</a> is a utility that allows support for Java, Javascript, Flash, Silverlight, images and others to be toggled on and off within your browser, enabling you to check how a page behaves when viewed by someone who doesn't have these enabled.<br /></li></ul><ul><li>I really like the <span style="font-weight: bold;">HTML Validator</span> <a href="http://users.skynet.be/mgueury/mozilla/">http://users.skynet.be/mgueury/mozilla/</a> for ensuring that my HTML markup is actually W3C compliant; the main issue with this is that it's only available for Windows platforms. Provided you have the "Add-on Bar" visible in Firefox (toggle via "Options" in the Firefox main menu, or do ctrl+/), this displays a little icon at the bottom of the screen indicating the goodness or otherwise of your markup.</li></ul><p>There are a few other useful add-ons for working with design elements like colours and images:<br /></p><ul><li><span style="font-weight: bold;">Colorzilla</span> <a href="http://www.colorzilla.com/">http://www.colorzilla.com/</a> is a tool that allows you (among other things) to pick colours from the current webpage and get the corresponding hex codes or RGB values.</li></ul> <ul><li><span style="font-weight: bold;">Measureit</span> <a href="http://frayd.us/">http://frayd.us/</a> creates a ruler that lets you measure the size of page elements in pixels - particularly helpful when sizing images for web display.</li></ul> <ul><li>In the past I've found the in-browser screen capture utility <span style="font-weight: bold;">Fireshot</span> <a href="http://screenshot-program.com/fireshot/">http://screenshot-program.com/fireshot/</a> quite handy for taking screenshots of an entire webpage including the off-screen portions. I have to admit I haven't used it for a while though. There's a paid "pro" version which offers a lot of additional functionality.<br /></li></ul>Although I've given URLs, the easiest way to install any of these is via the <span style="font-style: italic;">"Get Add-ons"</span> tab accessed via the <span style="font-style: italic;">"Add-ons"</span> option in Firefox's main menu (I'm using Firefox 8.0 at the time of writing). Once installed the individual add-ons appear in various places, for example by default Firebug's icon can be found at the top-right hand corner. If an add-on's icon doesn't appear automatically (as seems to happen for Measureit) then you might have to add it manually: go to <span style="font-style: italic;">"Options"</span>/<span style="font-style: italic;">"Toolbar layout"</span>, locate the item and drag it to the toolbar.<br /><br />I wouldn't try to argue that this is definitive list, but for an occasional user like as myself these tools work well and (with the exception of Firebug) are easy to remember how to use even after several months away from them. However if these don't meet your needs then I'd recommend checking out the <a href="https://addons.mozilla.org/en-US/firefox/extensions/web-development/">"Web Development" category of Mozilla's add-ons site</a> for many more options.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-84163425892881717102011-11-13T06:55:00.000-08:002011-11-13T13:03:47.175-08:00Creative Commons overviewA while ago I came across an interesting overview of the Creative Commons licence for digital content by Jude Umeh in the BCS "IT Now" newsletter ("Flexible Copyright", also available via the BCS website at <a href="http://www.bcs.org/content/conBlogPost/1828">http://www.bcs.org/content/conBlogPost/1828</a> as "Creative Commons: Addressing the perils of re-using digital content"), which I felt gave a very clear and concise introduction to the problem that Creative Commons (CC) is trying to solve, how it works in practice, and some of the limitations.<br /><br />Essentially, anyone who creates online content - whether a piece of writing (such as this blog), an image (such as a photo in my Flickr stream), or any other kind of media - automatically has "all rights reserved" copyright on that content. This default position means that the only way someone else can (legally) re-use that content is by explicitly seeking and obtaining the copyright owner's permission (i.e. a licence) to do so. As you might imagine this can present a significant barrier to re-using online content.<br /><br />The aim of the Creative Commons is to enable content creators to easily pre-emptively grant permissions for others to re-use their work, by providing a set of free licences which bridge the gap between the "all rights reserved" position (where the copyright owner retains all rights on their work) and "public domain" (where the copyright owner gives up those rights, and allows anyone to re-use their work in any way and for any purpose).<br /><br />These licences are intended to be easily understood and provide a graduated scale of permissiveness. According to the article the six most common are:<br /><br /><ul><li><span style="font-weight: bold;">BY ("By Attribution"):</span> this is the most permissive, as it grants permission to reuse the original work for any purpose - including making "derived works" - with no restrictions other than that it must attributed to the original author.</li></ul><ul><li><span style="font-weight: bold;">BY-SA ("By Attribution-Share Alike"):</span> the same as BY, with the additional restriction that any derived work must also be licensed as BY-SA.</li></ul><ul><li><span style="font-weight: bold;">BY-ND ("By Attribution-No Derivatives"):</span> the original work can be freely used and shared with attribution, but derivative works are not allowed.</li></ul><ul><li><span style="font-weight: bold;">BY-NC ("By Attribution-Non-Commerical"):</span> as with BY, the original work can be used, shared and used in derived works, provided attribution is made to the original author; however the original work cannot be used for commercial purposes.</li></ul><ul><li><span style="font-weight: bold;">BY-NC-SA ("By Attribution-Non-Commercial-Share Alike"):</span> similar to BY-SA, so any derived work must use the same BY-NC-SA licence, and like BY-NC, in that commercial use of the original work is not permitted.</li></ul><ul><li><span style="font-weight: bold;">BY-NC-ND ("By Attribution-Non-Commercial-No Derivatives"):</span> the most restrictive licence (short of "all rights reserved"), as this only allows re-use of the original work for non-commercial purposes, and doesn't permit derivative works to be made. Umeh states that BY-NC-ND is "often regarded as a 'free advertising' licence".<br /></li></ul><br />As Umeh points out, "CC is not a silver bullet", and his article cites examples of some of its limitations and potential pitfalls. Elsewhere I've also come across some criticisms of using the non-commercial CC licences in certain contexts: for example, the scientist Peter Murray Rust has blogged about what he sees as the negative impact of CC-NC licensing in science and teaching (see "Suboptimal/missing Open Licences by Wiley and Royal Society" <a href="http://blogs.ch.cam.ac.uk/pmr/2011/10/27/suboptimalmissing-open-licences-by-wiley-and-royal-society/">http://blogs.ch.cam.ac.uk/pmr/2011/10/27/suboptimalmissing-open-licences-by-wiley-and-royal-society/</a> and "Why you and I should avoid NC licences" <a href="http://blogs.ch.cam.ac.uk/pmr/2010/12/17/why-i-and-you-should-avoid-nc-licences/">http://blogs.ch.cam.ac.uk/pmr/2010/12/17/why-i-and-you-should-avoid-nc-licences/</a>).<br /><br />However it's arguable that these are special cases, and that more generally CC-based licensing has a significant and positive impact on enabling the legal re-use of online material that would otherwise not be possible: indeed, even the posts cited above only criticise its NC aspects, and otherwise see the CC as greatly beneficial. Certainly it's worth investigating if you're interested in allowing others to reuse digital content that you've produced (there's even a page on the CC website to help choose the appropriate CC licence based on answers to plain English questions: <a href="http://creativecommons.org/choose/">http://creativecommons.org/choose/</a>).<br /><br />As I'm not an expert on CC (or indeed on copyright law or content licensing), I'd recommend Umeh's article as the next step for a more comprehensive and expert overview; and beyond that of course more information can be found at the Creative Commons website <a href="http://www.creativecommons.org/">http://www.creativecommons.org/</a> (with the UK-specifi<span style="text-decoration: underline;"></span>c version due to become available at <a href="http://www.creativecommons.org.uk/">http://www.creativecommons.org.uk</a> later this month).pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-48736963311606106432011-10-24T12:02:00.000-07:002011-10-24T13:14:23.814-07:00Day Camp 4 Developers: Project ManagementJust over three of weeks ago I attended the third online <a href="http://daycamp4developers.com/">Day Camp 4 Developers</a> event, which this time focused on the subject of project management. The DC4D events are aimed at filling the "soft skills" gap that software developers can suffer from, and rather than being a "how-to" on project management (arguably there are already plenty of other places you can learn the basics) the six speakers covered a range of topics around the subject - some of which I wouldn't initially have thought of in this context (for example, dealing with difficult people). However as one of the speakers noted, fundamentally project management is as much about people as it is about process, and all of them delivered some interesting insights.<br /><br />The first talk (which unfortunately I missed the start of through my own disorganisation) by <a href="http://brianhprince.com/">Brian Prince</a> about <span style="font-weight: bold;">"Hands-on Agile Practices"</span> covered the practical implementation of Agile process in a lot of detail. I've never worked with Agile myself, but I have read a bit about it in the past and Brian's presentation reminded me of a few Agile concepts that sound like they could be usefully adopted elsewhere. For example: using "yesterday's weather" (i.e. statistically the weather tomorrow is likely to be the same as today's) as a way to plan ahead by considering recent performance; and the guidelines for keeping stand-up meetings concise could also be applied to any sort status meeting (each person covers the three points "what I did yesterday", "what I'm doing today and when it will be done", and "what issues I have"). The idea of focusing on "not enough time" rather than "too much to do" also appealed to me.<br /><br />The next presentation <span style="font-weight: bold;">"Dealing with Difficult People"</span> by <a href="http://naramore.net/blog/">Elizabeth Naramore</a> turned out to be an expected delight. Starting by asking what makes people you know "difficult" to interact with, she identified four broad types of behaviour:<br /><ul><li><span style="font-style: italic; font-weight: bold;">"Get it done"</span>-types are focused on getting information and acting on it quickly, so their style is terse and to-the-point,</li><li><span style="font-style: italic; font-weight: bold;">"Get it right"</span>-types are focused on detail, so their style is precise, slow and deliberate,</li><li><span style="font-style: italic; font-weight: bold;">"Get along"</span>-types are focused on making sure others are happy, so their style is touchy-feely and often sugar-coated, and<br /></li><li><span style="font-style: italic; font-weight: bold;">"Get a pat on the back"</span>-types are focused on getting their efforts recognised by others, so their style is more "person-oriented".</li></ul>The labels are instantly memorable and straight away I'm sure we can all think of people that we know who fit these categories (as well as ourselves of course). Elizabeth was at pains to point out that most people are a mixture of two or more, and that none of them are bad (except when someone is operating at an extreme all of the time). The important point is that they affect how people communicate, so if you can recognise and adapt to other people's styles, and learn to listen to them, then you'll stand a better chance of reducing your difficult interactions.<br /><br /><a href="http://akrabat.com/">Rob Allen</a> was next up with <span style="font-weight: bold;">"Getting A Website Out Of The Door"</span> (subtitled <span style="font-weight: bold; font-style: italic;">"Managing a website project"</span>), and covered the process used at <a href="http://www.bigroominternet.co.uk/about/meet-the-team">Big Room Internet</a> for spec'ing, developing and delivering website projects for external clients. Rob included a lot of detail on each part of the process, what can go wrong, and how they aim to manage and reduce the risks of that happening. One specific aspect that I found interesting was the change control procedure, which is used for all change requests from their clients regardless of the size of the change, essentially:<br /><ul><li>Write down the request</li><li>Understand the impact</li><li>Decide whether to do it</li><li>Do the work!</li></ul>I think that the second point here is key: you need to understand what the impact will be, and how much work it's really going to be (I'm sure we've all agreed at one or another to make "trivial" changes to code which have turned out in practice to be far more work than anyone first imagined). A more general point that Rob made was the importance of clear communication, particularly in emails (which should have a subject line, summary, action and deadline).<br /><br />Rob was followed by <a href="http://caseysoftware.com/home">Keith Casey</a> talking about how <span style="font-weight: bold;">"Project Management is More Than Todo Lists"</span>. One of the interesting aspects of Keith's talk was that he brought an open source perspective to the subject. In open source projects the contributors are unpaid and so understanding how their motivations differ from doing work paid work is important for the projects to successful: as Keith said early on in his talk, in this case "it's about people".<br /><br />He argued that people managing open source projects should pay attention to the uppermost levels of <a href="http://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs">Maslow's Hierarchy of Needs </a>(where the individual focuses on "esteem" and "self-actualisation"), but there was also a lot of practical advice: for example, having regular and predictable releases; ensuring that bugs and feature requests are prioritised regularly, and that developements should be driven input and involvement by the community. I particularly liked the practical suggestion that frequently asked questions can be used to identify areas of friction that need to be simplified or clarified. He also recommended Karl Fogel's book <a href="http://www.amazon.co.uk/Producing-Open-Source-Software-Successful/dp/0596007590/">"Producing Open Source Software"</a>, which looks like it would be a good read.<br /><br /><a href="http://www.thursdaybram.com/">Thursday Bram</a>'s presentation <span style="font-weight: bold;">"Project Management for Freelancers"</span> was another change of direction (and certainly the subtitle "<strong></strong>How Freelancers Can Use Project Management to Make Clients Happier than They've Ever Been Before" didn't lack ambition). She suggested that for freelancers, project management is at least in part about helping clients to recognise quality work - after all they're not experts in coding (that's why they hired you), so inevitably they have an issue with knowing "what does quality look like?". (If you've ever paid for a service such as car servicing or plumbing then I'm sure you can relate to this.) So arguably one function of project management is to provide a way to communicate the quality of your work. The key message that I took away from Thursday's talk was that "what makes people happy is 'seeing progress' on their projects". Again I felt this was an idea that I could use in my (non-freelancer) work environment.<br /><br />The last session of the day was <a href="http://paul-m-jones.com/">Paul M. Jones</a> talking about <span style="font-weight: bold;">"Estimating and Expectations"</span>. Essentially we (i.e. everyone) are terrible at making estimates, as illustrated by his "laws of scheduling and estimates":<br /><ul><li><span style="font-weight: bold;">Jones' Law:</span> "if you plan for the worst, then all surprises are good surprises"</li><li><span style="font-weight: bold;">Hofstadter's Law:</span> "it always takes longer than you expect - even when you take Hofstadter's Law into account"</li><li><span style="font-weight: bold;">Brooks' Law:</span> "adding more people to a late software project will make it even later"<br /></li></ul>However there are various strategies and methods we can use to try and make our estimates better: for example, using historical data and doing some design work up front can both provide valuable knowledge for improved estimates. In this context Paul also had my favourite quote of the day: <span style="font-style: italic;">"It's not enough to be smart; you actually have to know things"</span> (something that I think a lot of genuinely clever people can often forget, especially when they move into a domain that's new to them).<br /><br />It felt like Paul packed an immense amount of material into this talk, covering a wide range of different areas and offering a lot of practical advice drawn from various sources (Steve McConnell's <a href="http://www.amazon.co.uk/Code-Complete-Practical-Handbook-Construction/dp/0735619670/">Code Complete</a>, Fred Brooks' <a href="http://www.amazon.co.uk/Mythical-Month-Essays-Software-Engineering/dp/0201835959/">The Mythical Man Month</a> and Tom DeMarco and Timothy Lister's <a href="http://www.amazon.co.uk/Peopleware-Productive-Projects-Teams-2nd/dp/0932633439/">Peopleware: Productive Projects & Teams</a> were all mentioned) both for estimation techniques and for expectation management - where ultimately communication and trust are key (a message that seemed to be repeated throughout the day).<br /><br />In spite of a few minor technical issues (the organisers had opted to use a new service called <a href="http://www.fuzemeeting.com/">Fuzemeeting</a>, which I guess was still ironing out some wrinkles), overall everything ran smoothly, and at the end I felt I'd got some useful ideas that I feel I can apply in my own working life - in the end surely that's the whole idea. It was definitely worth a few hours of my weekend, and I'm looking forward to being able to see some of the talks again when the videocasts become available. In the meantime if any of this sounds interesting to you then I'd recommend checking out the <a href="http://daycamp4developers.com/">DC4D website</a> and watching out for the next event!pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-53238603402837914082011-10-16T09:29:00.000-07:002011-10-23T04:15:44.245-07:00Using Fedora 15 & Gnome 3: an updateFollowing up on my previous posting about the <a href="http://ironiccog.blogspot.com/2011/07/fedora-15-and-gnome-3-user-basics.html">Fedora 15/Gnome 3 user experience</a>, I've now been using it as a day-to-day working environment for the last 4 1/2 months and thought it was time to post a brief update.<br /><br />Overall the experience has been pretty good (although I gather a lot of other commentators on the web wouldn't agree). For me the least satisfying aspect is still the automated workspaces/virtual desktops, closely followed by the default left-click behaviour of icons in the favourites sidebar. Both of these continue to catch me out from time to time, but I'd class their deficiencies as merely irritating rather than unusable.<br /><br />Another aspect that I complained about in my previous post was the limited set of customisations that seemed to be available. However I've since discovered the <span style="font-weight: bold;">gnome-tweak-tool</span>, which provides access to a much wider range of customisations than is offered via the "Preferences" options. (This and many other useful features are covered in <a href="http://docs.fedoraproject.org/en-US/Fedora/15/html/Release_Notes/sect-Release_Notes-Changes_for_Desktop_Users.html">Fedora's release notes for desktop users</a>, which I should probably have read right at the start.)<br /><br />It's likely that you'll need to explicitly install it as it doesn't appear to be there by default, i.e.:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">% yum install gnome-tweak-tool</span></span><br /><br />(Nb this requires superuser privileges). To launch, start from the command line (or go to the "Applications" desktop view and <a href="http://www.flickr.com/photos/oblong_dog/6109603999/in/set-72157627125161258/">use the search box to look for "tweak"</a>). The tool itself looks like this:<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/6109604083/" title="Fedora 15: gnome-tweak-tool: "Fonts" tab by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6191/6109604083_2d6afe9181.jpg" alt="Fedora 15: gnome-tweak-tool: "Fonts" tab" height="250" width="400" /></a><span style="font-size:85%;"><br /><span style="font-style: italic;"><br />Figure 1: gnome-tweak-tool displaying the "Fonts" tab</span></span></div><br />There are various categories ("Fonts", "Interface" etc) with a set of options for each, and at first glance there doesn't seem to be that many options available. However if the one you want might doesn't appear to be there then it's worth typing in some search terms to see if something comes up (for example, this is how I found the option for displaying the full date next to the time at the top of the screen).<br /><br />Another useful utility is <span style="font-weight: bold;">gnome-session-properties</span> (again, seems easiest to launch from the command line), which really doesn't have many options but does allow you to customise which applications start up on login:<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/6109603699/" title="Fedora 15: gnome-session-properties by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6067/6109603699_c8a739fc1e.jpg" alt="Fedora 15: gnome-session-properties" height="250" width="400" /></a><span style="font-size:85%;"><br /><span style="font-style: italic;"><br />Figure 2: gnome-session-properties dialog</span></span></div><br />As you can see by the fact that I'm still using the default desktop wallpaper, I'm not big on customisations (in fact my needs are basic: web browser, email client, terminal window, Emacs and some development tools are usually sufficient), however these additional tools have helped make me feel a little more at home, and generally I'm pretty happy with the setup now.<br /><br />Finally I thought I'd give the GnomeShell Cheatsheet page at <a href="http://live.gnome.org/GnomeShell/CheatSheet">http://live.gnome.org/GnomeShell/CheatSheet</a> a quick mention. It covers similar ground to my previous post but from a more expert perspective and with some useful extra detail.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-32790629231979741492011-07-27T13:33:00.000-07:002011-07-27T14:16:03.216-07:00Book review: “Python Testing Cookbook” by Greg L. Turnquist<span style="font-style: italic;">Disclosure: a free e-copy of this book was received from the publisher for review purposes. The opinions expressed here are entirely my own; a copy of this review has also been posted at Amazon.<br /></span><br />Greg L. Turnquist’s “Python Testing Cookbook” explores automated testing at all levels, with the intention of providing the reader with the knowledge needed to implement testing using Python tools to improve software quality. To this end the book presents over 70 “recipes” in its nine chapters (ranging from the basics of unit testing, through test suites, user acceptance and web application testing, continuous integration, and methods for smoke- and load-testing), covering both tools for testing Python, and Python tools for testing. It also delivers advice about how to get the most from automated testing, which is as much an art as a science.<br /><br />The first three chapters introduce the fundamentals: writing, organising and running unit tests, comprehensively covering <a href="http://docs.python.org/library/unittest.html">unittest</a> (Python’s built-in unit testing library), <a href="http://readthedocs.org/docs/nose/en/latest/">nose</a> (a versatile tool for discovering, running and reporting tests) and doctest (which turns Python docstrings into testable code – a sample of this chapter can be downloaded from <a href="http://www.packtpub.com/python-testing-cookbook/book">http://www.packtpub.com/python-testing-cookbook/book</a>). Having established a solid foundation, subsequent chapters look at increasingly broader levels of automated testing using the appropriate relevant Python tools: for example, the “<a href="http://lettuce.it/">lettuce</a>” and “<a href="http://www.should-dsl.info/">should_DSL</a>” libraries for “<a href="http://behaviour-driven.org/">behaviour driven development</a>” (an extension of “test driven development” which aims to produce human-readable test cases and reports), and the “<a href="https://github.com/heynemann/pyccuracy/wiki/">Pyccuracy</a>” and “<a href="http://code.google.com/p/robotframework/">Robot</a>” frameworks for end-user acceptance testing of web applications. Later chapters cover higher level concepts and tools, such as using nose to hook Python tests into “continuous integration” servers (both <a href="http://jenkins-ci.org/">Jenkins</a> and <a href="http://www.jetbrains.com/teamcity/">TeamCity</a> are covered in detail), and assessing test coverage using the “<a href="http://nedbatchelder.com/code/coverage/">coverage</a>” tool (both as a metric, and to identify areas that need more tests). A detailed chapter on smoke- and load-testing includes practical advice on developing multiple test suites for different scenarios, and methods for stress-testing (for example, by capturing and replaying real world data) to discover weaknesses in a system before going to production. The final chapter distils the author’s experience into general advice on making testing a successful part of your code development methodology, both for new and legacy projects.<br /><br />There’s a lot of good stuff in this book: the initial chapters on unittest and nose are particularly strong, and I can imagine returning to these in future as a reference. There is also a lot of excellent and hard-won practical advice from the author’s own experience – not only in these early chapters but throughout the book – which is consistently valuable (in this regard the final chapter is a real highlight and could easily stand alone – I will definitely be re-reading it soon). Elsewhere the various tools and topics are presented clearly with plenty of useful detail, and in some cases have demystified things that I’d always assumed were quite esoteric and difficult to do (nose in particular was a revelation to me, but also setting up continuous integration servers and measuring test coverage).<br /><br />There are a few disappointments: the section on mock objects left me feeling baffled as to how to actually implement them in practice – a shame as it was something that I’d looked forward to learning. I’d also have liked something about approaches for handling difficult testing scenarios such as software which interacts with the file system or with large files – a few hints here would have been invaluable for me. There are typos in some commands and code in a few recipes (e.g. for nose), which meant I had to look up the correct syntax elsewhere – perhaps not so bad, but annoying (especially in a cookbook) – and since the recipes themselves aren’t numbered, this sometimes made it difficult to navigate between them.<br /><br />However these are fairly minor quibbles, and in conclusion I was impressed with both the breadth of material covered by the book and the level of detail for many topics. Moreover I enjoyed reading it and was often left feeling excited at the prospect of being able to apply the ideas to my own projects, which is I think was one of the author’s aims (and no mean feat for a technical book). I think that the combination of the detail together with the author’s practical advice make this book both an excellent introduction to testing with Python, and a valuable resource to refer back to subsequently.<br /><br />(Addendum: Greg Turnquist's blog about the book can be found at <a href="http://pythontestingcookbook.posterous.com/">http://pythontestingcookbook.posterous.com/</a> and features some interesting supplementary material.)pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-89241119308552244282011-07-10T11:08:00.000-07:002011-10-16T09:34:49.297-07:00Fedora 15 and Gnome 3: user basicsI've been using Fedora 15 for about a month now and thought it was time to write up some of my experiences with the new Gnome 3 desktop, since certain aspects are quite a bit different from the previous version. I know other people have posted details about the Fedora 15 desktop (for example Xavier Claessens' <a href="http://blogs.gnome.org/xclaesse/2011/04/25/one-week-with-gnome3/">One Week with Gnome 3</a>) but when I first installed it there didn't seem to be much from a "user basics" perspective. So this is my take, hope it's useful.<br /><br /><span style="font-weight: bold;font-size:130%;" >Getting started</span><br /><br />When you first start up Gnome 3 the desktop looks pretty empty - in fact there are no desktop icons in this new version (even when you put them in your "Desktop" subdirectory):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5905874151/" title="Fedora 15: Gnome 3 desktop by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6023/5905874151_a5b6498926.jpg" alt="Fedora 15: Gnome 3 desktop" height="250" width="400" /></a><span style="font-size:85%;"><br /><span style="font-style: italic;">Figure 1: "Empty" Gnome 3 desktop on startup. No desktop icons allowed!</span></span><br /></div><br />To get started, move the mouse to the word "Activities" at the top right-hand corner of the screen (the so-called "hot corner") - immediately changing the desktop to the "Windows" view:<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5906433032/" title="Fedora 15: "Activities" hot corner: "Windows" view by Oblong Dog, on Flickr"><img src="http://farm6.static.flickr.com/5276/5906433032_68e8e7f33e.jpg" alt="Fedora 15: "Activities" hot corner: "Windows" view" height="250" width="400" /></a><br /><span style="font-size:85%;"><span style="font-style: italic;">Figure 2: "exploded view" of the Gnome 3 desktop, accessed either by moving the mouse over the "Activities" hot corner (top-left of the screen), or by hitting the "Gnome" (i.e. Windows) key on the keyboard. The Favourites sidebar sits on the left edge of the screen, and the edge of the Workspaces sidebar peeks out on the right.</span></span><br /></div><br />In this view (figure 2) you can see the <span style="font-weight: bold;">Favourites sidebar</span> on the left side, and just the very edge of the <span style="font-weight: bold;">Workspaces sidebar</span> on the right (more about those below). I call this the <span style="font-weight: bold;">exploded view</span> of the current workspace, since (as in this example) features minatures of any windows in the workspace. The exploded view can also be toggled by pressing the <del>"Gnome key"</del> "Super key" (i.e. the Windows key).<br /><br />From this view you can toggle to the "Applications" view, by clicking on "Applications" near the top left (figure 3):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5905874335/" title="Fedora 15: "Activities" hot corner: "Applications" view by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6010/5905874335_d86f31bf11.jpg" alt="Fedora 15: "Activities" hot corner: "Applications" view" height="250" width="400" /></a><span style="font-size:85%;"><br /><span style="font-style: italic;">Figure 3: "Applications" view of the Gnome 3 desktop</span></span><br /></div><br />This shows all the applications installed on the system, with a search box and category groupings on the right to help you find the one you want.<br /><ul><li>Drag icons from this view to the "Favourites" bar to make them more easily accessible in future.<br /></li><li>The "Add/remove software" application is a graphical front end to yum for installing and managing additional packages that are weren't included by default.</li></ul><br /><span style="font-weight: bold;font-size:130%;" >The Favourites sidebar: launching and navigating applications</span><br /><br />The Favourites sidebar is the strip down the left-hand side which holds various application icons. These icons do "double-duty": if you've dragged an icon there from elsewhere (essentially "favouriting" it), then it initially acts as a launcher for that application; also the icons for any running applications (favourited or otherwise) will appear here.<br /><br />If an application already has one or more instances running then clicking on its icon takes you to the "nearest" running instance; clicking-and-holding (or right clicking) gives you more options (e.g. to start a new instance, or move to any of the currently open instances) - behaviour that is probably already familiar to users of Windows 7 and Mac OS X (figure 4):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5909977188/" title="Fedora 15: "Favourites" sidebar by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6056/5909977188_f71945c771.jpg" alt="Fedora 15: "Favourites" sidebar" height="250" width="400" /></a><span style="font-size:85%;"><br /><span style="font-style: italic;">Figure 4: The "Favourites" sidebar with dialogue (i.e. the black bubble) opened for Firefox after right-clicking on its icon. This gives options to move to a running Firefox window, or to start a new Firefox instance.</span></span><br /></div><br />Another way to navigate between running applications is to use Alt+Tab to cycle between them (figure 5):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5916621360/" title="Fedora 15: Alt-Tab cycle through applications by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6013/5916621360_002b909a19.jpg" alt="Fedora 15: Alt-Tab cycle through applications" height="250" width="400" /></a><span style="font-size:85%;"><br /><span style="font-style: italic;">Figure 5: Alt+Tab moves between running applications...</span></span><br /></div><br />Repeated Alt+Tabb'ing moves between the applications; if there's more than one instance running then these are also shown when you Alt+Tab to it, and you can use the arrow keys to navigate to the specific one you want (figure 6):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5916621484/" title="Fedora 15: Alt+Tab cycle through applications (multiple windows) by Oblong Dog, on Flickr"><img src="http://farm7.static.flickr.com/6023/5916621484_274cebf8d7.jpg" alt="Fedora 15: Alt+Tab cycle through applications (multiple windows)" height="250" width="400" /></a><br /><span style="font-style: italic;font-size:85%;" >Figure 6: ... and arrow keys allow you to select a specific instance if there are multiple instances of a particular application.</span><br /></div><br /><span style="font-weight: bold;font-size:130%;" >The Workspaces sidebar: navigating multiple desktops</span><br /><br />Workspaces provide a way to manage applications, by giving the user multiple virtual desktops. These should already be familiar to seasoned Gnome users, but they operate somewhat differently in Gnome 3: there are no longer a fixed number of workspaces, instead they are created and destroyed automatically by the system as required.<br /><br />You can move between workspaces in at least two different ways. Firstly, you can access the workspaces sidebar on the right-hand side of the screen, by moving the mouse over it in the "exploded view" of the desktop and causing it to "pop out" (figure 7):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5905894999/" title="Fedora 15: Workspaces "popped out" by Oblong Dog, on Flickr"><img src="http://farm6.static.flickr.com/5080/5905894999_a793067a29.jpg" alt="Fedora 15: Workspaces "popped out"" height="250" width="400" /></a><br /><span style="font-size:85%;"><span style="font-style: italic;">Figure 7: the Workspaces sidebar "popped out" on the right of the screen in the exploded view of the Gnome 3 desktop.</span></span><br /></div><br />The sidebar shows miniatures of each workspace, with the current workspace highlighted with a white outline. Clicking on one of the images takes you to that workspace; you can also drag application windows between the different workspaces.<br /><br />Note the sidebar also shows an extra "empty" workspace at the bottom: if an application is opened or moved into this workspace then a new empty workspace is automatically created underneath. Furthermore, there's only ever one empty workspace - so if a workspace "empties" (e.g. because you've closed all the applications it contains) then Gnome automatically removes it. This can be quite disconcerting, and is probably the feature that causes me the most confusion in practice as it often upsets my sense of where I am in the workspace order.<br /><br />The other way to navigate between workspaces is Alt+Ctrl+Up/Down Arrows, which I find myself using quite a lot (although I frequently overshoot into the empty workspace by accident) (figure 8):<br /><br /><div style="text-align: center;"><a href="http://www.flickr.com/photos/oblong_dog/5905874657/" title="Fedora 15: alt+ctrl+arrows to navigate workspaces by Oblong Dog, on Flickr"><img src="http://farm6.static.flickr.com/5235/5905874657_17d13f1cc0.jpg" alt="Fedora 15: alt+ctrl+arrows to navigate workspaces" height="250" width="400" /></a><br /><span style="font-size:85%;"><span style="font-style: italic;">Figure 8: moving between multiple desktops using Alt+Ctrl+Up/Down keys</span></span><br /></div><br /><span style="font-weight: bold;font-size:130%;" >Other observations</span><br /><ul><li><span style="font-weight: bold;">Resizing windows: </span>windows can be maximised by double-clicking on their title bar (double-click again to restore to the original size). There's no "minimise" button on the window frame, so you now have to right-click and then select the "Minimise" menu option. Also, note that dragging a window to the top of the screen automatically causes it to maximise (again similar to Windows 7, and not always what you intend). Manual resizing is also possible as always, by dragging the window edges - but this can be fiddly, as the area where an edge can be "caught" for dragging seems to be quite small.<br /></li></ul><ul><li><span style="font-weight: bold;">System notifications</span>: these now pop up rather discreetly at the bottom of the screen, but interacting with then can be frustrating at times - often they disappear before you have a chance to read them, and sometimes (counterintuitvely) disappear when clicked.<br /></li></ul><ul><li><span style="font-weight: bold;">Customisation:</span> while some preference-type options are available via the "username" menu (top right-hand corner of the screen) under "System Settings", overall the customisation options feel quite limited (for example, no screen-savers). However as a number of interfaces to system tools currently only seem to be accessible by launching from a command line, it's not clear if this a conscious design decision or whether more customisation options will be exposed in future versions.</li></ul><ul><li><span style="font-weight: bold;">Fallback mode:</span> this is a half-way house between Gnome 2 and Gnome 3, and is started by default on systems which can't support the full Gnome 3 experience (which appears to include virtual machines). However as it's much more like the old Gnome, if you really don't like the new version then you could try using fallback mode instead.</li></ul><br /><span style="font-size:130%;"><span style="font-weight: bold;">Conclusion</span></span><br /><br />Having been using Fedora 15 and Gnome 3 day-to-day for a few weeks now, I'm now largely used to its quirks and finding it overall a perfectly serviceable working environment - for me the new workspaces model and the rather random system notification mechanism have proved to be the most challenging differences from previous versions. So while it may not suit everyone's tastes it's definitely worth trying (and hopefully the more egregious foibles will be ironed out in future versions).pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com2tag:blogger.com,1999:blog-2061759690050619608.post-57560573727690423442011-05-30T10:02:00.000-07:002011-10-23T03:10:41.100-07:00Mac OS X: new user tipsOver the couple of weeks I've been using an Apple iMac, and as a Windows/Linux user I've found navigating the desktop has been something of a learning experience for me.<br /><br />As different as they are, in many ways the standard Windows and Linux desktops are idiomatically quite similar these days, and both support the standard PC three-button mouse. By contrast the Mac OS X desktop environment (and its use of the infamous one-button mouse) has a number of differences which can turn even basic operations (for example, cut-and-paste) initially into something of a challenge.<br /><br />However some basic knowledge should go a long way in helping. First, there are the three essential keys you need to know about:<br /><ul><li><a style="font-weight: bold;" href="http://en.wikipedia.org/wiki/Command_key">Command key</a>: [⌘]</li><li><a style="font-weight: bold;" href="http://en.wikipedia.org/wiki/Option_key">Option key</a>: [⌥]</li><li><a style="font-weight: bold;" href="http://en.wikipedia.org/wiki/Control_key">Control key</a>: [ctrl]</li></ul>(The links give more background but aren't essential to the following. You can think of the option key as being the same as the "Alt" key on Windows/Linux.)<br /><br />Then:<br /><ul><li><span style="font-weight: bold;">Emulating the right-hand mouse button:</span> [ctrl] + mouse click (essential for desktop and web applications that use this to activate context menus and so on)</li></ul>Basic text editing operations:<br /><ul><li><span style="font-weight: bold;">Cut: </span>[⌘] + [x]</li><li><span style="font-weight: bold;">Copy:</span> [⌘] + [x]</li><li><span style="font-weight: bold;">Paste:</span> [⌘] + [x]</li></ul>Basic keys for navigating within text documents:<br /><ul><li><span style="font-weight: bold;">Home:</span> [↖]</li><li><span style="font-weight: bold;">End:</span> [↘]</li><li><span style="font-weight: bold;">Page up:</span> [<span style="font-size:180%;">⇞</span>]<br /></li><li><span style="font-weight: bold;">Page down:</span> [<span style="font-size:180%;">⇟</span>]<br /></li></ul>Useful shortcuts for navigating the desktop:<br /><ul><li><span style="font-weight: bold;">Cycle between open windows:</span> [⌥] + [tab]</li><li><span style="font-weight: bold;">Zoom out (pulls back to show all open windows):</span> [F9]</li><li><span style="font-weight: bold;">Show desktop (hides all open windows):</span> [F11]<br /></li></ul>And finally (and essential if you're programming and find your Apple keyboard is missing a hash key):<br /><ul><li><span style="font-weight: bold;">Hash symbol ("#"):</span> [⌥] + [3]<br /></li></ul>These all work on OS X 10.4.11 ("Tiger"), which is admittedly no longer a very recent release, but hopefully they're also applicable to later Mac OSes. I can't say that I've fallen in love with Apple as a result, but they have enabled me to operate at an acceptably functional level (until I can get my Linux workstation up and running!).pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com2tag:blogger.com,1999:blog-2061759690050619608.post-20897010986969373632011-04-09T06:04:00.000-07:002011-04-11T12:37:11.584-07:00Managing Python packages: virtualenv, pip and yolkI've recently been playing with the Python <a href="http://pypi.python.org/pypi/virtualenv/">virtualenv</a> package - along with <a href="http://pypi.python.org/pypi/pip">pip</a> and <a href="http://pypi.python.org/pypi/yolk">yolk</a> - as a way of managing third-party packages. This post is my brief introduction to the basics of these three tools.<br /><br /><span style="font-weight:bold;">virtualenv</span> lets you create isolated self-contained "virtual environments" which are separate from the system Python. You can then install and manage the specific Python packages that you need for a particular application - safe from potential problems due to version incompatibilities, and without needing superuser privileges - using the <span style="font-weight:bold;">pip</span> package installer. <span style="font-weight:bold;">yolk</span> provides an extra utility to keep track of what's installed.<br /><br /><span style="font-weight: bold;">1. virtualenv: building virtual Python environments</span><br /><br />virtualenv can either be installed via your system's package manager (for example, synaptic on Ubuntu), or by using the <a href="http://peak.telecommunity.com/DevCenter/EasyInstall">easy_install</a> tool, i.e.:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">$ easy_install virtualenv</span></span><br /><br />(If you don't have the SetupTools package which provides easy_install then you can download the "bootstrap" install script from <a href="http://peak.telecommunity.com/dist/ez_setup.py">http://peak.telecommunity.com/dist/ez_setup.py</a>. Save as <span style="font-size:85%;"><span style="font-family:courier new;">ez_setup.py</span></span> and run using <span style="font-size:85%;"><span style="font-family:courier new;">/path/to/python ez_setup.py</span></span>.)<br /><br />Once virtualenv is installed you can create a new virtual environment (called in this example, "myenv") as follows:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">$ virtualenv --no-site-packages myenv</span></span><br /><br />This makes a new directory myenv in the current directory (which will contain bin, include and lib subdirectories) based on the system version of Python. The <span style="font-size:85%;"><span style="font-family:courier new;">--no-site-packages</span></span> option tells virtualenv not to include any third-party packages which might have been installed into the system Python (see the <a href="http://www.virtualenv.org/en/latest/">virtualenv documentation</a> for details of other options).<br /><br />To start using the new environment, run the environment's "activate" command e.g.:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">$ source myenv/bin/activate</span></span><br /><br />The shell command prompt will change from e.g. <span style="font-family:courier new;">$</span> to <span style="font-family:courier new;">(myenv)</span><span style="font-family:courier new;">$</span>, indicating that the "myenv" environment (and any packages installed in it) will be used instead of the system Python for applications run in this shell. (Note that the Python application code doesn't need to be inside the virtual environment directory; in fact this directory is just using for the packages associated with the virtual environment.)<br /><br />Finally, when you've finished working with the virtual environment you can leave it by running the <span style="font-family:courier new;"></span><span style="font-size:85%;"><span style="font-family:courier new;">deactivate</span></span> command (also in the bin directory).<br /><br />(On Windows you may have to specify the full path to the "Scripts" directory of your Python installation when invoking the easy_install and virtualenv commands above, e.g. <span style="font-size:85%;"><span style="font-family:courier new;">C:\Python27\Scripts\virtualenv</span></span>. Also, note that when a virtual environment is created it won't contain a "bin" directory - instead it's activated by invoking the <span style="font-size:85%;"><span style="font-family:courier new;">Scripts\activate</span></span> batch file in the virtual environment directory. Invoking the <span style="font-size:85%;"><span style="font-family:courier new;">deactivate</span></span> command exits the environment as before.)<br /><br /><span style="font-weight: bold;">2. pip: installing Python packages</span><br /><br />Once you're created a virtual environment you can start to add packages (which is really the point of doing this in the first place). virtualenv automatically includes both easy_install and an alternative package installer called <a href="http://pypi.python.org/pypi/pip">pip</a> (at least, for virtualenv 1.4.1 and up; earlier versions only have easy_install, so you'll need to run <span style="font-size:85%;"><span style="font-family:courier new;">easy_install pip</span></span> within the virtual environment in order to get it).<br /><br />Most packages that are easy_installable can also be installed using pip, and it's designed to work well with virtualenv. However I think its main advantage is that it offers some useful functionality that's missing from easy_install - most significantly, the ability to uninstall previously installed packages. (Other useful features include the ability to explicitly control and export versions of third-party package dependencies via "requirements files" - see the <a href="http://www.pip-installer.org/en/latest/index.html">pip documentation</a> for more details.)<br /><br />Basic pip usage looks like this:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">(myenv)$ pip install python-dateutil <span style="font-style: italic;"># install latest version of a package</span></span><br /><br /></span><span style="font-size:85%;"><span style="font-family:courier new;">(myenv)$ pip uninstall python-dateutil <span style="font-style: italic;"># remove package</span></span></span><br /><span style="font-size:85%;"><span style="font-family:courier new;"><br />(myenv)$ pip install python-dateutil==1.5 <span style="font-style: italic;"># install specific version</span></span></span><br /><br />(As an aside, the <a href="http://labix.org/python-dateutil">python-dateutil</a> package is illustrative of one of the advantages of using pip over easy_install: after installing the latest version of python-dateutil, I discovered that it's only compatible with Python 3 - an earlier 1.* version is required to work with Python 2. pip let me uninstall the newer version and reinstall the older one.)<br /><br /><span style="font-weight: bold;">3. yolk: checking Python packages installed on your system</span><br /><br />The final utility I'd recommend is <a href="http://pypi.python.org/pypi/yolk">yolk</a>, which provides a way of querying which packages (and versions) have been installed in the current environment. It also has options to query <a href="http://pypi.python.org/pypi">PyPI (the Python Package Index)</a>. Installing it is easy:<br /><br /><span style=";font-family:courier new;font-size:85%;" >(myenv)$ pip install yolk</span><br /><br />Running it with the -l option (for "list") then shows us what packages are available:<br /><pre style="width: 450px; overflow: scroll; overflow-y:auto; background-color:#F5F5DC;"><span style="font-size:85%;">(myenv)$ yolk -l<br />Python - 2.6.4 - active development (/usr/lib/python2.6/lib-dynload)<br />pip - 1.0 - active<br />python-dateutil - 1.5 - active<br />setuptools - 0.6c9 - active<br />wsgiref - 0.1.2 - active development (/usr/lib/python2.6)<br />yolk - 0.4.1 - active<br /></span></pre>(See the <a href="http://pypi.python.org/pypi/yolk">yolk documentation</a> to learn more about its other features.)<br /><br /><span style="font-weight: bold;">Summary</span><br /><br />Obviously the above is just an introduction to the basics of virtualenv, pip and yolk for managing and working with third-party packages - but hopefully it's enough to get started. If you're interested in using virtualenv in practice then Doug Hellman's article about working with <a href="http://www.doughellmann.com/articles/pythonmagazine/completely-different/2008-05-virtualenvwrapper/index.html">multiple virtual environments</a> (and his <a href="http://www.doughellmann.com/projects/virtualenvwrapper/">virtualenvwrapper</a> project, which provides tools to help) is recommended as a starting point for further reading.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-64937858204389915692011-04-04T09:16:00.000-07:002011-04-05T05:55:36.570-07:00Richard Stallman: "A Free Digital Society?"About a month ago I was fortunate to attend an IET-hosted lecture by Richard Stallman, entitled "A Free Digital Society?". Probably most famous as the originator of the <a href="http://www.gnu.org/">GNU project</a> (out of which came <a href="http://www.gnu.org/gnu/why-gnu-linux.html">GNU/Linux</a>) and initiator of the <a href="http://www.gnu.org/philosophy/free-software-intro.html">free software movement</a>, Stallman has for many years been an active and vocal advocate for free software, and has a campaigned against excessive extension of copyright laws<br /><br />He began the talk with the observation that there is an implicit assumption in the recent movement towards "digital inclusion", that using computers and the internet is inherently good and beneficial. However, as the question mark in the title of his talk indicated this assumption merits closer attention, as (in his opinion) there are various issues and threats associated with these technologies. These include:<br /><ul><li><span style="font-weight: bold;">Survelliance:</span> technology now makes it possible for ISPs, websites and other organisations to monitor and analyse what individuals do online (e.g. the sites that they visit, things they buy, search terms they use etc) to an extent to which (in Stallman's words) "Stalin could only dream".</li></ul><ul><li><span style="font-weight: bold;">Censorship:</span> for example, governments or corporations blocking access to particular websites (think Google in China), or even forcing them to close.</li></ul><ul><li><span style="font-weight: bold;">Restrictions on users imposed by data formats:</span> both proprietary (e.g. Silverlight) and patented data formats (e.g. <a href="http://en.wikipedia.org/wiki/MP3">MP3</a>) restrict what the end user is able to do with the data they encode.</li></ul><ul><li><span style="font-weight: bold;">Non-free software:</span> here "free" is in the sense of "freedom", rather than price. Non-free software is essentially software that isn't under the control of you, the user - in the case of proprietary software, it's controlled by the owner (for example Microsoft, Apple, Amazon) who is able to insert features (e.g. to track user behaviour) that serves their interests rather than those of the user. By contrast, free software - which by the way <a href="http://www.gnu.org/philosophy/selling.html">you can still charge money for</a> - gives the user four basic freedoms: 0. to run the software for any purpose; 1. to study how the software works, and make changes to it; 2. to redistribute the software as-is; 3. to redistribute the software with your changes (see the <a href="http://www.gnu.org/philosophy/free-sw.html">free software definition</a>). In this way malicious features can be detected and removed, and control is returned to the user.<br /></li></ul><ul><li><span style="font-weight: bold;">"Software as a service" (SaaS)</span>: in Stallman's definition, "software as a service" is anything where the computation is done by programs that you can't control - this is like non-free software above, because someone else has control and can change how your computing is done at any time without your permission. He made a distinction between things like e-commerce, online storage storage (e.g. Dropbox), publishing (e.g. Twitter) and search (which are about "data" or "communication", and so are not SaaS), and e.g. Google Docs (which does do computation for you, and so is SaaS). (See Stallman's article <a href="http://www.gnu.org/philosophy/who-does-that-server-really-serve.html">Who does that server really serve?</a>)<br /></li></ul><ul><li><span style="font-weight: bold;">Misuse of an individual's data:</span> essentially doing something with your data without your permission, or even your knowledge - for example, passing on personal data to the authorities, unilaterally modifying your data, or even (for example in the case of Facebook) <a href="http://www.wired.com/epicenter/2008/01/facebook-ads-ma/">using it for commercial purposes</a>.</li></ul><ul><li><span style="font-weight: bold;">"The War on Sharing":</span> according to Stallman, sharing is "using the internet for what it's best at", and the war on sharing - whether <a href="http://en.wikipedia.org/wiki/Digital_rights_management">digital rights management</a> (DRM) technology or threatening internet users with disconnection (as under the UK's <a href="http://en.wikipedia.org/wiki/Digital_Economy_Act_2010">Digital Economy Act</a>) - is an attempt by commercial interests to unfairly restrict what users are allowed to do (see Stallman's article <a href="http://stallman.org/articles/end-war-on-sharing.html">Ending the War on Sharing</a>).</li></ul><ul><li><span style="font-weight: bold;">Users don't have a postive right to do things on the internet:</span> essentially, all the activities that users perform on the internet - communications, payment etc - are dependent on organisations who have no obligation to continue providing those services to you.</li></ul>This is a pretty long list of issues (hopefully I've accurately captured the essence of each), and while many of them can be mitigated by moving to free software; others (for example, monitoring by ISPs) require other solutions - and Stallman admitted that he's quite pessimistic about the future. Aside from that, it was a fascinating and entertaining talk (including the auctioning of a <a href="http://shop.fsf.org/category/stuffed-gnu/">GNU gnu soft toy</a> to raise funds for the Free Software Foundation) and the subsequent audience Q&A session provided many opportunities for elaboration and clarification on many of the issues.<br /><br />I'm still mulling over many of the issues raised. On the one hand there is a fundamental question about what moral rights you believe individuals should have, both generally and with specific regard to the digital world; and on the other there is the question of what you should do if you feel those rights are not being upheld. Stallman's position is clear and uncompromising: for example, not owning a mobile phone and not using a key card to enter his office (to avoid the possibility of being tracked), and using a netbook that allows him to run 100% free software (down to the BIOS level). It's certainly given me plenty to think about, and I'm looking forward to reading his book of collected essays <a href="http://shop.fsf.org/product/free-software-free-society-2/">"Free Software, Free Society"</a> - which might be a good place to start if you're also interested in learning more.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-54801138124441661562011-04-03T11:51:00.000-07:002011-04-08T08:31:28.667-07:00Book review: "Python 2.6 Text Processing: Beginner’s Guide" by Jeff McNeilJeff McNeil’s “Python 2.6 Text Processing: Beginner’s Guide” is a practical introduction to a wide range of methods for reading, processing and writing textual data from a variety of structured and unstructured data formats. Aimed primarily at novice Python programmers who have some elementary knowledge of the language basics but without prior experience in text processing, the book offers hands-on examples for each of the techniques it discusses – ranging from Python’s built-in libraries for handling strings, regular expressions, and formats such as JSON, XML and HTML, through to more advanced topics such as parsing custom grammars, and efficiently searching large text archives. In addition it contains a great deal of general supporting material on working with Python, including installing packages and third-party libraries, and working with Python 3.<br /><br />The first three chapters lay the foundations, covering a number of Python basics including a crash course in file and URL I/O, and the essentials of Python’s built-in string handling functions. Useful background topics – such as installing packages with easy_install, and using virtualenv – are also introduced here. (A sample of the first chapter can be freely downloaded from the book’s website at <a href="https://www.packtpub.com/python-2-6-text-processing-beginners-guide/book">https://www.packtpub.com/python-2-6-text-processing-beginners-guide/book</a>). The next three cover: using the standard library to work with simple structured data formats (delimited “CSV” data, “ini”-style configuration files, and JSON-formatted data); working with Python regular expressions (a stand out chapter for me); and handling structured markup (specifically, XML and HTML). Subsequent chapters on using the Mako templating package (the default system for the Pylons web framework) to generate emails and web pages, and on writing more advanced data formats (PDF, Excel and OpenDocument), are separated by an excellent overview of understanding and working with Unicode, encodings and application internationalization (“i18n”).<br /><br />The remaining two chapters cover more advanced topics, with some good background theory supplementing the practical examples: using the PyParsing package to create parsers for custom grammars (with a brief nod to the basics of natural language processing using the Natural Language Toolkit, NLTK); and the Nucular package for indexing large quantities of textual data (not necessarily just plain text) to enable highly efficient searching. Finally, an appendix offers a grab-bag of general Python resources, references to some more advanced text processing tools (such as Apache’s Lucene/Solr), and an excellent overview of the differences between Python 2 and 3 (including a hands-on example of migrating code from 2 to 3).<br /><br />The book covers a lot of ground and moves fairly quickly; however it adopts a largely successful hands-on approach, engaging the reader with working examples at each stage to illustrate the key points, and this certainly helped me keep up. I was also impressed by the clear and concise quality of code in the examples, and the very natural way that general Python concepts and principles – generators, duck typing, packaging and so on – were introduced as asides. (One very minor criticism is that the layout of the example code could have been improved, as the indentation levels weren’t always immediately obvious to me.) Aside from a surprisingly unsatisfying chapter on structured markup (reluctantly, I would recommend looking elsewhere for an introduction to XML processing with Python) and a few niggling typos, there’s a lot of excellent material in this book, and the author has a knack for presenting some tricky concepts in a deceptively easy-to-understand manner. I think that the chapter on regular expressions is possibly one of the best introductions to the subject that I’ve ever seen; other chapters on encodings and internationalization, advanced parsing, and indexing and searching were also highlights for me (as was the section on Python 3 in the appendix).<br /><br />Overall I really enjoyed working through the book and felt I learned a lot. I think it’s fair to say that given the rather ambitious range of techniques presented, in many cases (particularly for the more advanced or specialised topics) that the chapters are inevitably more introductory than definitive in nature: the reader is given enough information to grasp the background concepts and get started, with pointers to external resources to learn more. In conclusion, I think this is a great introduction to a wide range of text processing techniques in Python, both for novice Pythonistas (who will undoubtedly also benefit from the more general Python tips and tricks presented in the book) and more experienced programmers who are looking for a place to start learning about text processing.<br /><br /><span style="font-style: italic;">Disclosure: a free e-copy of this book was received from the publisher for review purposes; this review has also been submitted to Amazon.</span>pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-52427022102579986282011-03-18T11:37:00.000-07:002011-03-18T16:07:14.906-07:00Day Camp 4 Developers: TelecommutingAbout two weeks ago I took part in the second online <a href="http://daycamp4developers.com/">Day Camp 4 Developers</a>, on the topic of telecommuting. The idea behind the Day Camp events is to provide software developers with practical knowledge and advice in the area of "soft" skills, to complement their expertise with "hard" skills (i.e. actual coding). In this case five speakers gave consistently excellent web presentations (slides and audio) with different perspectives on remote working, while an IRC chatroom gave all participants a forum to discuss the issues behind the scenes.<br /><br /><a href="http://lornajane.net/">Lorna Jane Mitchell</a> started off by asking "<span style="font-weight: bold;">Could You Telecommute?</span>". As a teleworker herself, Lorna Jane looked at the environmental, organisational and personal factors that influence the happiness and productivity of the remote worker: for example, ensuring you have a good home working space, and set clear boundaries between work and personal life (both for yourself and for others). In particular you have to be aware of the tendency for other people to think that working from home is easy, and that your time is infinitely flexible. She also noted that there are some big differences between being part of a distributed team and being a telecommuting member of a co-located team (where you risk feeling isolated), and further differences between employees and freelancers. Particularly for lone telecommuters, it's important to build professional and social support networks that might otherwise be taken for granted in more conventional work settings.<br /><br />Next self-described "entreprenerd" <a href="http://www.jansch.nl/">Ivo Jansch</a> talked about "<span style="font-weight: bold;">The Business Case For Telecommuting</span>". Ivo's company <a href="http://www.egeniq.com/">Egeniq</a> is built around a distributed team (essentially using remote working as an organisational model) - so in addition to benefiting individual workers, he suggested ways that telecommuting could positively impact the company's bottom line, for example enabling access to an bigger talent pool and increasing its geographical reach (if providing consultancy services). He acknowledged that this distributed model won't suit every company or industry however, and success requires (amongst other things) a results-driven culture where individuals are trusted to self-manage and have a sense of shared responsibility. Ultimately good communication between team members is paramount.<br /><br />After the lunch break, Jack G. Ford gave a manager's perspective on setting up a telecommuting programme in his presentation "<span style="font-weight: bold;">Can I Work From Home Tomorrow?</span>". Jack introduced himself as an ex-coder who is now the manager for 17 developers in a more conventional environment than Ivo's, but in spite of that his key points seemed remarkably similar: beyond asking whether the company infrastructure can support remote working, the main issues are trust (both with the manager and with the team) and good communication between the manager and the individual. Jack emphasised that as a manager, when you telecommute, "I can't see you," so the telecommuter must stay connected, keep the manager informed, and must not only act professionally but be seen to do so. Although it might seem obvious, this was a fascinating insight into telecommuting from the other side of the management chain.<br /><br /><a href="http://www.khankennels.com/blog/">Ligaya Turmelle</a>'s presentation on "<span style="font-weight: bold;">Managing the Work/Life Balance</span>" emphasised the challenges of balancing work and home life, with her lists of "the good, the bad and the ugly" of remote working from a teleworker perspective. Ligaya focused especially on balancing family commitments with work commitments, and among some interesting observations (for example, no longer doing the daily commute means you lose some "me time" to yourself), I was most struck by the admission that if you love your work then it can mean sometimes that you <span style="font-style: italic;">want</span> to go on working, and are in danger of not respecting your own ground rules. While noting that situations can differ both for individuals and companies, her advice was: clarify everyone's expectations (e.g. policies for "on-call" hours, weekends, and holidays); set up ground rules and limits (and be disciplined in adhering to them); and try to be flexible and imaginative in how you approach your work.<br /><br />The final presentation was <a href="http://about.avdi.org/">Avdi Grimm</a> talking about "<span style="font-weight: bold;">The Well-Equipped Remote Worker</span>". Avdi is a freelance software developer who is also a "dispersed teams facilitator" and runs the <a href="http://wideteams.com/">Wide Teams blog</a>. As might be expected from the title, some of the focus was on the hardware and software tools that can help with remote working, but there was just as much information on practices that can support distributed teams. Once again promoting communication is key, and using tools and practices that help team members create good working relationships (for example, utilising social media like Twitter and Facebook, and holding regular face-to-face meetings) can really contribute to this.<br /><br />Looking back over all the talks, a few common themes had emerged for me:<br /><ul><li>Good communication (both with managers and with other team members) to build trust, keep people informed and avoid misunderstandings;</li><li>Clarify expectations on all sides, and establishing well-defined boundaries between work and personal life. Set ground rules to ensure that those boundaries are respected by others (your boss, your family and friends) and have the discipline to also respect them yourself;</li><li>Build and maintain your social and professional support networks for when there are problem times;</li><li>Provide yourself with a good working environment and (software and hardware) tools.<br /></li></ul>I was also able to relate some points to my own experiences: when I worked briefly as a remote member of a co-located team, I did feel a real sense of isolation; another time as a home teleworker I got the impression from some people that they assumed (not maliciously) that I only did a few hours work a day; and previous experience as part of a large organisation makes me feel that there was some truth in Ivo's comment that "co-location is over-rated", in that it doesn't automatically lead to great communication between individuals or groups.<br /><br />Overall it was an excellent event and a good use of 8 hours of my Saturday - although the time difference (coincidentally another telecommuting issue) meant that it didn't finish until 10pm UK time I surprised myself by staying with it to the end. Hats off to Cal and Kathy Evans for organising the day and to the speakers for their excellent presentations. Here's waiting for the next Day Camp 4 Developers!pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-27474584434429502532011-02-27T11:55:00.000-08:002011-02-28T12:46:22.786-08:00MadLab: pancake café and the Omniversity of ManchesterYesterday I dropped into the <a href="http://madlab.org.uk/">Manchester Digital Laboratory</a> (aka MadLab) in Edge Street for the <a href="http://madlab.org.uk/content/madlab-cafe-pancake-day/">MadLab Café Pancake Day</a>, and enjoyed a couple of hours chatting to various friendly people while eating an extremely tasty pancake and drinking cups of tea (one of my favourite pastimes), and at one point even discussing <a href="http://en.wikipedia.org/wiki/Outkast">Outkast</a>'s back catalogue.<br /><br />MadLab describes itself as "a community space for people who want to do and make interesting stuff - a place for geeks, artists, designers, illustrators, hackers, tinkerers, innovators and idle dreamers; an autonomous R&D laboratory and a release valve for Manchester's creative communities." I'm not sure precisely where I'd put myself in that list - I've only been there a couple of times before, for the <a href="http://groups.google.com/group/python-north-west">Python Northwest</a> user group meetings - but the folks I met seemed to be a representative cross section of the target community.<br /><br />There's a packed and eclectic schedule of (mostly free) events hosted there, which is well-worth checking out (see <a href="http://madlab.org.uk/events/">http://madlab.org.uk/events/</a>), but their most recent new development is the <a href="http://omniversity.madlab.org.uk/">Omniversity of Manchester</a> - a programme of professional-level training courses that so far have covered experimental film making and physical computing with Arduino, with plans to extend to topics as diverse as web design, Ruby on Rails, writing workshops and urban gardening. These courses won't be free, but the fees will go towards keeping MadLab sustainable and supporting the other free events.<br /><br />If you're interested in learning more then you can <a href="http://omniversity.madlab.org.uk/2011/02/omniversity-newsflash/">watch out a video</a>, and register the subjects you'd like to see covered by taking a moment to fill in their survey:<br /><ul><li><a href="http://s.madlab.org.uk/curriculum">The Omniversity Crowd Sourced Cirriculum Survey</a></li></ul>Personally I think it's a really exciting idea - I'm generally a fan of courses, and many of the proposed workshops are things that I'd love to learn more about, so it would also be great to see the Omniversity take off and help MadLab expand and flourish as a focal point for Manchester's digital community - the more people who find out about it and get involved the better. And in the meantime I'll be looking forward to the next (undoubtedly tasty) MadLab café event.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-80876892974950578522011-02-25T06:00:00.000-08:002011-02-25T09:15:09.479-08:00Book review: "Simply SQL" by Rudy LimebackRudy Limeback's "Simply SQL" (Sitepoint) is an overview of SQL targeted at web application developers, and intended to fill a gap between the basic "SQL 101"-type tutorials (seemingly compulsory in just about every introductory article or book about web programming) and more advanced texts covering topics which at first glance don't seem so relevant to the straightforward day-to-day requirements of many web applications.<br /><br />The chapters are grouped into two main sections. The first deals with the details of the SQL language and comprises the bulk of the book. It starts with a short introduction to the SQL commands most commonly needed by web developers to create and modify data within the database (all the usual suspects - CREATE, ALTER, INSERT, UPDATE, DELETE and so on - are quickly dealt with here). The rest of this section focuses on the SELECT command (the one used to retrieve information), with each chapter covering one specific clause - FROM, WHERE, GROUP BY and so on - in quite extensive detail, and illustrated with examples from sample applications.<br /><br />The second section of the book has three chapters covering some basic database design concepts, specifically SQL data types, relational integrity, and the use of "special structures" (such as tables that refer to themselves) for particular situations. The appendices then outline the basics of using some specific SQL implementations, along with details of the sample applications and scripts used in the main part of the book.<br /><br />The heavy emphasis on the SELECT statement might seem odd, but it makes a lot of sense in the context of web applications where data is typically read from the database far more than it's written. The detailed examples are also excellent - at times invaluable - for clarifying things like (for example) the nuances of the different types of JOINS, the subtleties of the GROUP BY and HAVING clauses (useful for aggregating data from subsets of rows in conjunction with summing and averaging functions), and the issues with working with time data. I certainly learnt a few things - the GROUP BY clause was completely new to me, as were the distinctions between the FLOAT and DECIMAL data types (DECIMALs are exact - within certain limits - while FLOATs are approximate). I found the brief sections on views, derived tables and subqueries extremely enlightening, as was the discussion of foreign keys in the chapter on relational integrity, and the clear writing style throughout made the book a pleasure to read.<br /><br />It's important to note that "Simply SQL" is based on the SQL standard, rather than the syntax of specific implementations (although in places it does indicate where there are notable deviations from the standard, particularly for MySQL) - also it doesn't cover any of the programming APIs, so it's not really a reference text (admittedly it doesn't claim to be). However with its clear and detailed explanations it looks like it would be a useful companion to more traditional reference or cookbooks and will definitely reward re-reading - least ways, I'm sure I'll be squeezing plenty more juice out of it in the future. So overall highly recommended.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com2tag:blogger.com,1999:blog-2061759690050619608.post-75162106212682938182011-02-11T08:09:00.000-08:002011-02-16T02:40:11.966-08:00Don Knuth: BCS/IET Turing LectureEarlier this week was the annual Manchester <a href="http://www.cs.manchester.ac.uk/aboutus/events/Turing/">BCS/IET Turing Lecture</a>, and this year's guest speaker was <a href="http://www-cs-faculty.stanford.edu/%7Eknuth/">Don Knuth</a>. Possibly he's best known (at least to me) as the author of the seminal <a href="http://en.wikipedia.org/wiki/The_Art_of_Computer_Programming">"The Art of Computer Programming"</a> (a multi-volume book which he began in 1962, and continues to work on to this day - subvolume 4A is the most recently published, with another 5 sections still to come), and the typesetting system <a href="http://en.wikipedia.org/wiki/TeX">TeX</a> (pronounced "tek", and used for typesetting countless Ph.D theses - including mine). However Knuth's contributions to computer science throughout his long career (he's now in his seventies) are staggering - as are his "extra-curricular" activities, which include writing novels and playing the pipe organ.<br /><br />So it was quite an opportunity to be able to listen to this giant of computing first hand - even more so since rather than a straightforward lecture, this was actually a Q&A, with Knuth taking questions from the audience. After opening with a concise explanation of the significance of the number 885205232 (which I won't spoil by revealing here, since it's a puzzle in his book "Selected Papers on Fun & Games", other than noting that it involves Alan Turing's manual for programming the <a href="http://en.wikipedia.org/wiki/Ferranti_Mark_1">Ferranti Mk. I computer</a>), Knuth fielded questions on various topics including: elegance in programming languages, the public's fear of computers, <a href="http://en.wikipedia.org/wiki/Busy_Beaver_Number">"busy beaver" numbers</a>, the best way to teach programming to elementary schoolchildren, and whether an aptitude for programming is an art or a "genetic defect".<br /><br />Throughout his answers were thoughtful, often surprising (for example, making a case for <a href="http://en.wikipedia.org/wiki/C_%28programming_language%29#Pointers">pointers in C</a> as an elegant language feature), consistently interesting, and delivered with characteristic humour (Knuth was once published in <a href="http://en.wikipedia.org/wiki/Mad_%28magazine%29">MAD magazine</a>, and is famous for the quote "Beware of bugs in the above code; I have only proven it correct, not tried it", <a href="http://www.brainyquote.com/quotes/authors/d/donald_knuth.html">amongst others</a>). In response to a question about "what are we 'enabling the information society' to do" (a reference to the BCS's current mission statement), Knuth initially replied "to have jobs", before more seriously reflecting that "there's a long way to go improving what we already have at the moment."<br /><br />Although Knuth's world of computing feels like it's a long way from the one I inhabit, it was a great privilege to see and hear such a legendary figure - in spite of his age he seems as lively as ever, both physically and intellectually, and still enjoying it - and his career is truly inspiring: when asked what he'd do differently if he had his time again, his reply was that he wouldn't change anything. "In my case," he said, "Murphy's Law hasn't worked - so many things that could have gone wrong didn't."pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-60807562693946841762011-01-31T08:27:00.001-08:002011-01-31T13:28:56.744-08:00The What, Where and How of Open DataLast week I attended a seminar at the <a href="http://www.ccsr.ac.uk/">Cathie Marsh Centre for Census and Survey Research</a>, given by <a href="http://rufuspollock.org/about/">Rufus Pollock</a> of the <a href="http://okfn.org/">Open Knowledge Foundation</a> (OKFN) on the topic of "open data"<a href="http://www.ccsr.ac.uk/"></a>.<br /><br />Rufus started by showing two example applications built using open data. <a href="http://yourtopia.net/">Yourtopia</a> makes use of data from the World Bank that measures individual nations progress towards the Millennium Development Goals. Visitors to the site balance the relative importance of different factors (for example, "health", "economy" and "education"), and their preferences are matched with the data in order to suggest which country meets them most closely. <a href="http://wheredoesmymoneygo.org/">Where Does My Money Go?</a> offers various breakdowns of UK government spending and presents these in a way that allows the site visitor to see (for example) how much of the tax they pay is used for things such as defence, environment, culture and so on.<br /><br />Both sites are eye-catching and fun (and can provide some surprising insights), while at the same time serving more serious purposes. In the context of the seminar Rufus noted that building the two sites also highlighted some key issues when working with these kinds of datasets:<br /><ul><li><span style="font-weight: bold;">Completeness:</span> i.e. the data are not always complete</li><li><span style="font-weight: bold;">Correctness:</span> i.e. the data are not always correct</li><li><span style="font-weight: bold;">Ease-of-use:</span> it can take a lot of effort to put the data into a format where it can actually be used (for example an estimated 90% of the time developing Where Does My Money Go?, as opposed to 10% actually building the site)<br /></li></ul>These issues can largely be mitigated by "open data", which has two key characteristics:<br /><ul><li><span style="font-weight: bold;">Legal openness: </span>the data must be provided with a licence that allows anyone to use, reuse and redistribute the data, for any purpose. ("Reuse" in this context can include combining it with other datasets and redistributing that.) An explicit open licence is required (such as those offered at <a href="http://www.opendatacommons.org/">Open Data Commons</a>) because the default legal position for any data - even that posted "openly" on the web - doesn't entitle someone else to reuse or redistribute. </li><li><span style="font-weight: bold;">Technical openness:</span> the data should be in a format that means that it's easy to access and work with, that it should be possible to obtain the data in bulk, and in a machine-readable, open format. These are pre-requisites for the data to be useful in a practical sense: for example, it's not sufficient to provide the data via a website that only returns subsets of that data via a form submission.</li></ul>(See the official definition at <a href="http://www.opendefinition.org/">http://www.opendefinition.org/</a>.)<br /><br />The data itself can be about almost anything: geographical (for example, mapping postcodes to a latitude and longitude), statistical, electoral, legal, financial - the OKFN's <a href="http://www.ckan.net/">CKAN</a> (Comprehensive Knowledge Archive Network) site has many examples. The key point is that the data should not be personal - that is, it shouldn't enable individuals to be identified, either directly or indirectly.<br /><br />The motivation for making data open goes back to the initial issues of completeness, correctness and ease-of-use - it can take a lot of time to assemble a dataset (for example, the Government already collects a lot data), but once the effort has been made then the added cost of releasing it is small, and then sharing it reduces the cost of merging, filling gaps and correcting errors. To make an analogy with open source software, it's a essentially <a href="http://en.wikipedia.org/wiki/Linus%27_Law">Linus' Law</a> for data: "given enough eyeballs, all bugs are shallow". Rufus also talked about a corollary to this, the "many minds" principle: the best use of the data you produce will probably be thought of by someone else (and vice versa).<br /><br />One argument against openness is that it precludes the possibility of commercial exploitation in order to offset the costs of compiling the data, and is a topical point given the current economic climate. Rufus's counter-argument is that there are many other ways to fund the creation of data aside from making it proprietary, by considering the data as a platform (rather than as a product), and building on that platform to sell extensions or complementary services (such as consultancy - again there are parallels with open source software). (Some of the audience expressed also concerns that in principle at least, open data is might be used irresponsibly - but arguably if the data is available to all then it means that others could challenge that interpretation.)<br /><br />The final point that Rufus's talk addressed is how to actually build the open data ecosystem. To some degree it's up to the people who hold the data, but his suggestions are:<br /><ul><li>Start small and simple (which I took to mean, start with small sets of data rather than doing everything all at once).<br /></li><li>If you're using someone else's dataset then you can make an enquiry via the OKFN website to find out what the licensing situation is<span style="font-style: italic;">.</span><br /></li><li>If you have your datasets then put them under an open data licence and can register it at CKAN so that others can find it.<br /></li><li>"Componentize" your data to make it easier to reuse (which I took to mean, divide the datasets up into sensible subsets).<br /></li><li>Make the case with whoever holds the data you want (government, business etc) to release it openly.</li></ul>For me as a "lay person", this was a fascinating introduction to the world of open data. Not unreasonably the seminar didn't go into details of actually working with such data (I think many of the seminar audience members were researchers already familiar with the available tools). However afterwards Rufus made the point that writing a paragraph of text after looking at the data is just as valid as the slick visualisations provided by Where Does My Money Go? and other sites. Ultimately it's having open access to the data in the first place that counts.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com0tag:blogger.com,1999:blog-2061759690050619608.post-30428188504152750152011-01-23T07:24:00.000-08:002011-01-23T10:55:02.327-08:00Python North-West: The Python ChallengeLast week I went to my first-ever <a href="http://groups.google.com/group/python-north-west">Python North-West</a> meeting, at the <a href="http://madlab.org.uk/">Manchester Digital Laboratory</a> (aka MadLab). The webpage describes it as a "user group for Pythoneers and Pythonistas of all levels and ages, open to everyone coding 'the way Guido indented it'", and meetings alternate between talks and coding dojos (group coding sessions where people get to share code and ideas with the aim of improving their knowledge and skills - see <a href="http://codingdojo.org/cgi-bin/wiki.pl?CodingDojo">http://codingdojo.org/cgi-bin/wiki.pl?CodingDojo</a> for more information).<br /><br />This particular meeting was a coding dojo and so as a group we worked through The Python Challenge (<a href="http://www.pythonchallenge.com/">http://www.pythonchallenge.com/</a>), which is a series of puzzles that can be solved using Python programming combined with some imagination and lateral thinking. While most people had come with their own laptops, the format that developed was for one person to "drive" the laptop connected to the overhead projector, typing in code and taking suggestions from the others.<br /><br />Although I'd already looked at the first two challenges earlier in the day to get an idea of what was involved, the group setting provided a great opportunity to see how other people worked, and to learn about bits of Python that I was unfamiliar with - one example for me was being introduced to <a href="http://docs.python.org/tutorial/datastructures.html#list-comprehensions">list comprehensions</a>, which are concise ways to generate lists, e.g.:<br /><br /><span style="font-family:courier new;">>>> [[x,x**2] for x in vec]</span><br /><span style="font-family:courier new;">[[2, 4], [4, 16], [6, 36]]</span><br /><br />(although there were several other examples which I won't write about here so as not to spoil the challenges for others). Also, as many of the challenges began with having to figure out what the programming problem actually was, it meant that collectively we didn't get stuck for too long on any particular puzzle - I know that at least a couple would have had me completely stumped if I'd been on my own. For me personally it was also an opportunity to play with <a href="http://wiki.python.org/moin/IDLE">IDLE</a> - Python's <a href="http://en.wikipedia.org/wiki/Integrated_development_environment">IDE</a> - under Windows (not an environment that I've used much in the past but quite handy for this kind of exploratory programming process.)<br /><br />Overall it was great to get out and interact with other Python developers in an enthusiastic and friendly atmosphere, while at the same time broadening my knowledge of the language - and now I've had a taste I'll definitely be back for future meetings.pjbhttp://www.blogger.com/profile/02877142465318426440noreply@blogger.com2