Upgrading Fedora Linux 20 (Heisenbug) to 21 & 22

Although I've recently switched to Ubuntu for my work, I'm still using Fedora (dual-boot with Windows 7) on my home workstation. I've been running Fedora 20 ("Heisenbug", horrible name) since mid-2014 so my installation was pretty aged - and with the release of Fedora 22 in May has been end-of-life'd since June this year. So it was definitely time for an update.

When I moved previously to F20 I did a fresh install (although I was able to keep my /home from the previous F16 installation). However this time I decided to go with an upgrade using FedUp (FEDora UPgrader), especially attractive since it promised to also update the installed packages - meaning that I wouldn't need to start over again with installing and configuring all the applications and tools that I use.

The upgrade procedure is covered pretty comprehensively in the FedUp documentation at https://fedoraproject.org/wiki/FedUp. As I was two releases adrift I decided to perform the upgrade twice (i.e. going from F20->F21->F22) as recommended. This seemed the safest route, especially given two of the significant changes introduced with each of these realeases, specifically:

• Fedora 21 introduced the idea of "products", which can be thought of as sub-releases specifically targeted for particular user needs. When upgrading from F20, you need to specify which product sub-release you're moving to: the available options are "workstation" (which looks like the best choice for desktop use), "server" or "cloud".
• Fedora 22 switched from the old yum package manager to a newer version called dnf ("dandified yum") - see https://fedoraproject.org/wiki/Changes/ReplaceYumWithDNF

(As an aside, another change is that from F21 the releases will no longer have names - so F21 is just "Twenty One", and F22 is "Twenty Two".)

The upgrade process for each revision was then:

1. Install the fedup package:

sudo yum install fedup

2. Perform all updates & backup your system (my preference is still for CloneZilla Live)

3. Perform a network-based upgrade (the recommended method over e.g. upgrading from an ISO image):

sudo yum update fedup fedora-release

sudo fedup --network 21 --product=workstation

(As noted earlier, for F20-to-F21, you need to specify which product line you are moving to - hence the --product option above. For F21-to-F22 the second line can be reduced to simply sudo fedup --network 22)

4. Reboot and select "System Upgrade (fedup)" from the GRUB menu, which will boot into an upgrader environment and then churn through all the packages. (Note that this can take some time: for F20-to-F21 this has a textual display and for me the screen would go black periodically during this process, however moving the mouse seemed to restore the display. For F21-to-F22 there is a nice graphical progress bar; using F1 toggled between this and the textual display.)

5. If the previous step completes successfully then the system will reboot automatically and there should be an option for the new Fedora version in the boot menu. Select this to boot into the new version and check that it looks as you expect.

At this point the documentation outlines some post-upgrade clean-up actions that need to be performed manually:

1. Rebuild the RPM database:

sudo rpm --rebuilddb
sudo yum distro-sync --setopt=deltarpm=0

2. Install and run rpmconf to identify packages that have more recent versions, and interactively decide what to do for each:

sudo yum install rpmconf

sudo rpmconf -a

(After upgrading to F22 replace the first line with
sudo dnf install rpmconf
However for me when running the subsequent step, rpmconf crashed out with an error about a missing file, so I wasn't able to complete this step.)

Overall the process was pretty painless, and after a week or so I haven't noticed any issues (other than my continued inability to get a working version of Google Chrome - but that problem predates this upgrade), with everything appearing to work at least as well as before.

Also while Fedora 22 looks very much like Fedora 20, there have been some visual tweaks which I feel make it a bit nicer to look at than before (however this is perhaps a personal thing - see Chris Duckett's A month with Fedora 22 leaves me hungry for 23 for a counter-view), and for now I'm enjoying my "refreshed" Fedora experience.

Checking the RAID level with MegaCLI on Ubuntu 14.04

I recently installed Ubuntu 14.04 on a new PC (Dell Precision Tower 7910) and wanted to check the RAID configuration that the system had been set up to use. Unfortunately none of the BIOS or system configuration menus seemed to give me access to this information.

Luckily a colleague pointed me towards MegaCLI, a utility for obtaining information about (and troubleshooting) LSI RAID controllers. Instructions on how to install and use MegaCLI are given in this Nerdy Notes blog post: http://blog.nold.ca/2012/10/installing-megacli.html. However as this dates from 2012, doesn't cover use on Ubuntu, and doesn't include information on how to interpret the outputs from the utility, I've written up what I did below.

0. Check you're using a MegaRAID controller

Directly from Nerdy Notes: do

lspci -nn | grep RAID

and check that the output for MegaRAID, e.g.

02:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS-3 3008 [Fury] [1000:005f] (rev 02)

1. Obtain and install MegaCLI

The first problem is actually obtaining a copy of MegaCLI. According to the Nerdy Notes post, it should be available for download by searching on the LSI website, however when I tried this the download link was non-functional. Subsequently I've discovered that it can be located by searching on the Avago site (go to http://www.avagotech.com/ and enter "megacli" into the search box).

Opening the downloaded zip file reveals a set of subdirectories with versions of the utility for different OSes, with the Linux directory only holding a "noarch" RPM file. This should be straightforward to install using yum on Redhat-based systems (such as Fedora), but for Ubuntu it's necessary to extract the MegaCLI executables using the rpm2cpio utility:

rpm2cpio MegaCli-8.07.14-1.noarch.rpm | cpio -dimv

(The rpm2cpio utility can be installed via the rpm2cpio package using Synaptic or apt-get.)

This should pull out the MegaCli64 executable. Note that it doesn't need to be installed in any special location, you can run it from wherever you extracted it to using the above command, however you need to run with superuser privileges otherwise the output is blank.

2. Run MegaCli6 to probe the RAID information

To get information on the adapter and see which RAID levels the system supports, do:

sudo MegaCli64 -AdpAllInfo -aAll

and grep the output for "RAID"; this should give you output of the following form:

RAID Level Supported            : RAID0, RAID1, RAID5, RAID00, RAID10, RAID50, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span

Then to get information on which RAID level is actually being used:

sudo MegaCli64 -LDInfo -Lall -aAll

and again grep for "RAID":

RAID Level          : Primary-5, Secondary-0, RAID Level Qualifier-3

This doesn't explicitly name a "standard" RAID level, but I found this ServerFault post which gives details on how to interpret the output: http://serverfault.com/questions/385796/how-to-interpret-this-output-from-megacli

In this case "Primary-5, Secondary-0, RAID Level Qualifier-3" turns out to be equivalent to RAID 5
(https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5), which is suitable for my needs - so I was able to rest a bit easier.

I'd like to acknowledge my colleague Ian Donaldson for tipping me off about MegaCLI.

Enabling correct monitor drivers on Ubuntu 14.04 LTS

Having recently installed Ubuntu 14.04 LTS on a new PC, I had some troubles configuring the display to operate correctly with a Dell 24" monitor: specifically, the monitor aspect ratio of 1920x1200 (16:10) didn't seem to be supported by the default install from the Live CD. In fact it was more than a little unstable (to say the least: for example, attempting to change the display aspect ratio rendered the display unusable several times, necessitating a complete reinstall to recover).

This appeared to be a missing driver issue, but it was difficult to find out what drivers were needed. Searching on the web for the specific problem was rather fruitless, and the many suggestions to look at xrand (for example https://wiki.ubuntu.com/X/Config/Resolution) didn't work for me either. In the end I stumbled across a post which suggested using the command:

ubuntu-drivers devices

from a terminal, which shows all devices which need drivers, and which packages apply to them. For example for my system the output looks like:

== /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0 ==
vendor  : NVIDIA Corporation
modalias : pci:v000010DEd000013BAsv000010DEsd00001097bc03sc00i00
driver  : nvidia-346-updates - distro non-free
driver  : nvidia-346 - third-party free
driver  : nvidia-340 - third-party free
driver  : nvidia-352 - third-party free
driver  : xserver-xorg-video-nouveau - distro free builtin
driver  : nvidia-349 - third-party non-free
driver  : nvidia-355 - third-party free recommended
driver  : nvidia-340-updates - distro non-free

When I ran this it marked 'nvidia-349' as the recommended driver (nb the output above is from a subsequent run of ubuntu-drivers and doesn't show the mark). This was easily installed using:

apt-get install nvidia-349

and appeared to solve my issues. Unfortunately I haven't been able to find the post again that originally suggested this, however if you are experiencing similar resolution issues then it might be worth a try before grappling with xrand.

Keeping a GitHub fork up-to-date with the original repo

Forking is a standard way to make changes to a third party repository on GitHub. Typically a developer makes a fork (essentially a clone on the GitHub server side) of someone else's repo in order to make their own changes to the project. These changes might be for private use, or they could subsequently be submitted back to the original repo for evaluation and possible inclusion via the pull request mechanism.

However: once the fork is created, any further commits or other changes made to the original repo will not automatically be reflected in the fork. This post describes how to keep the fork up-to-date with the original repository (which is normally referred to as the upstream repo) by pulling in the changes manually via a clone of the fork, using the process described below.

The update procedure is:

0. Make a clone of your fork onto your local machine (or use one you've already made), and move into the cloned directory.

1. Add a new remote repository to the clone which points to the upstream repo. The general syntax to do this is:

git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPO.git

A remote is simply a version of the project which is hosted on the Internet or elsewhere on the network, and a repo can have multiple remotes (use git remote -v to list all the ones that are defined; you should that origin is also a remote). Conventionally the remote used for this syncing is given the name upstream, however it could be called anything you like.

For example:

git remote add upstream https://github.com/fls-bioinformatics-core/genomics

This step only needs to be done once for any given clone.

2. Update the local clone by fetching and merging in any changes from the upstream repo, using the procedure:

git fetch upstream
git checkout master
git merge upstream/master

If you have made your changes in your own dedicated branches and avoided making commits to the fork's master branch then the upstream changes should merge cleanly without any conflicts. (It's considered best practice that changes made to the fork should always be done in a purpose-made branch; working directly in master makes it harder to make clean pull requests and can cause conflicts when trying to merge the upstream changes into your fork.)

3. Push the updated master branch back to your fork on GitHub using:

git push origin master

Once again, if you have avoided committing local changes to master then the push should be drama-free.

Finally, you will need to repeat steps 2 and 3 in order to stay up to date whenever new changes are made to the upstream repo.

Inline images in HTML tags with Python

I recently discovered a neat trick for embedding images within HTML documents, really useful if you've got an application where you would like the HTML files to be portable (in the sense of being moved from one location to another) and not have to rely on also moving a bunch of related image files.

Essentially the inlining is achieved by base64 encoding the data from the image file into an ASCII string, which can then be copied into the src attribute of an <img> tag with the following general syntax:

<img src="data:image/image_type;base64,base64_encoded_string" />

For example to embed a PNG image:

<img src="data:image/png;base64,iVBORw...." />

i.e. image_type is png and iVBORw... is the base64 encoded string (truncated here for readability).

If you're familiar with Python then it's straightforward to encode any file using the base64 module, e.g. (for a PNG image):

>>> import base64
>>> pngdata = base64.b64encode(open("cog.png",'rb').read())
>>> print "<img src='data:image/png;base64,%s' />" % pngdata

And here's an example inlined PNG generated using this method:

In fact this inlining is an example of the general data URI scheme for embedding data directly in HTML documents, and can also be used for example in CSS to set an inline background image - the Wikipedia entry on the "Data URI scheme" is a good place to start for more detailed information.

There are some caveats, particularly if you're interested in cross-browser compatibility: older versions of Internet Explorer (version 7 and older) don't support data URIs at all, while version 8 only supports encoded strings up to 32KB. More generally the encoded strings can be around 1/3 larger than the original images and are implicitly downloaded each time the document is refreshed; these are definitely considerations if you're concerned about bandwidth. However if these aren't issues for your application then this can be a handy trick to have in your toolbox.

Update 24/10/2012: another caveat I've discovered since is that at least some command line HTML-to-PDF converters (for example wkhtmltopdf) aren't able to convert the encoded images, so it's worth bearing this in mind if you plan to use them. (On the other hand the PDF conversion in Firefox - via "Print to file" - works fine but can't be run from the command line AFAIK.)

Installing Fedora 16 (Verne) dual-boot with Windows 7

Back in February I finally got around to installing Fedora 16 (Verne) as a dual-boot option on my Windows 7 home machine. I was helped considerably by an impressively detailed tutorial on How to dual-boot Fedora 15 and Windows 7 posted on the LinuxBSDos.com blog, which was invaluable for the tricky steps of partitioning and boot loader set up.

In this post I give an overview of what I did for my own dual-boot installation, with some hints and tips that I picked up in the process.

Preparation: back up the existing system

My first step was to make an image of the existing system on an external hard drive so that I could recover everything if the installation should fail for any reason. To do this I used Clonezilla Live, which can be burned onto a CD or other bootable media (I used the 1.2.12-10 AMD64 iso). I found the Clonezilla website quite difficult to navigate but there are very detailed usage instructions at http://www.clonezilla.org/clonezilla-live-doc.php, and while Clonezilla Live's menu-driven structure might feel a bit retro to those used to a slicker UI, the program itself turns out to be very easy to use.

Note that the size of the resulting image file (and hence the amount of space required on the external drive) depends on how much of the disk actually contains data, rather than the size of the disk or partition being imaged. So although the Windows partitions had been allocated the whole of my 1 TB hard drive, in this case only about 70 GB was in use and the corresponding image file was even smaller at around 45 GB.

Repartitioning: create space for the Fedora installation

Having created the image the next step was to free up some space by shrinking the size allocated to the Windows partitions (the LinuxBSDos dual-booting tutorial assumes there's already some free space available for the Fedora install). To do this I used GParted Live (version 0.11.0-10), a free partition editor that runs from bootable media and works with 64-bit systems. Launching the GParted application from the bootable CD is covered at http://gparted.sourceforge.net/display-doc.php?name=gparted-live-manual and there is comprehensive documentation on how to use it (with screenshots) at gparted.sourceforge.net/display-doc.php?name=help-manual. However I think the user interface is pretty intuitive on its own. For my installation I reduced the Windows allocation down to 50% of the disk space, leaving around 500 GB of unallocated space.

Installing Fedora 16

I was now in a position to install Fedora into the free space. I downloaded and burned the Fedora 16 installable live media (aka Live Desktop CD) from http://fedoraproject.org/en/get-fedora (make sure you get the appropriate 32- or 64-bit version for your system), which also allows you to try out Fedora without installing (see my previous post on Fedora 15 and Gnome 3 user basics if you're unfamiliar with the Gnome 3 desktop).

Detailed information about the installation procedure can be found in the extensive Fedora 16 Installation Guide but in summary this is what I did:

• Booted the system from the Fedora Live Desktop CD and started the installer via Activies/Applications/Install to Hard Drive.

• Clicked through the installer screens and set up keyboard layout, computer name, network setup, computer name, and timezone etc (generally if you're installing a vanilla home desktop system then the default options should be okay) until I reached the Type of installation screen. Here I needed to tell the installer where to put Fedora:
  
  • Choose the Use Free Space option to use the unallocated disk space, and
  • Make sure the Review and modify partitioning layout option is checked.
  before clicking through to the next screen.

• The next step was to modify the assigned logical volumes to my preferred layout (here the detailed instructions in the LinuxBSDos blog post were invaluable; I strongly recommend consulting them if you're unfamiliar with logical volumes and partitions). For my installation I changed the Fedora defaults to set up the following new partitions:
  
  • lv_root (/): 10 GB
  • lv_home:
  • /home: 75 GB
  • /boot: 500 MB
  • lv_swap: 16 GB
  Generally it's recommended to only use the space you need for the new installation and leave unused capacity unallocated; keeping /home separate from the root partition means that I can share user directories between multiple Linux installations in future; and the recommendation is that swap should be allocated around twice the size of the system RAM. However more elaborate partitioning schemes can be created, with various pros and cons - there's useful information on this in the Recommended Partitioning Scheme section of the Fedora documentation.

• Having set up the partition layout, I then needed to change the location of the Fedora boot loader (GRUB) from the default of Master Boot Record to First sector of boot partition. The reason for this is that any changes that are made to the Master Boot Record (MBR) (such as installing GRUB) are liable to be overwritten by future Windows updates, so it's better to manage the boot options from within Windows after Fedora has been installed:
  
  • Click Change Device and select First sector of boot partition (I'd recommend verifying that the correct location is shown before clicking Next to complete the installation, and also making a note of the boot loader location for reference later).

At this point I was finished and the installation could complete, which was a relatively quick process (in my case it took around 5 minutes).

Configuring the Windows boot manager

Once the installation process had completed the computer rebooted directly into Windows, so to be able to access Fedora I needed to update the Windows boot manager: for this I used EasyBCD 2.1.2 (EasyBCD is a Windows bootloader editor freely available from NeoSmart Technologies) as recommended in the LinuxBSDos.com tutorial. Once installed in Windows I was able to add a new entry for Fedora 16:

• Click on the Add New Entry tab
• Select the Linux/BSD tab and select the partition that the GRUB boot loader was installed to (which I made a note of earlier)
• Select Edit Boot Menu to view the change

Then restarting the system then gave me the option of booting either Windows or Fedora 16.

(Note that using this method, when Fedora 16 is selected you get the Linux GRUB boot loader rather than just booting directly into Linux; however it should be possible to configure GRUB to boot Fedora immediately.)

Post-installation Fedora set-up

Once Fedora 16 was booted for the first time, there were a few standard post-installation steps to complete: agreeing to the user license, setting up the date and time - usually I select the option to synchronise over the network - and creating a non-root user account. I also ran the Software Update application to pull in any updates (this also fixed the settings for my monitor, which initially weren't properly configured). It was then possible to start building up the software inventory and configuring the system to my personal preferences.

The final thing I did was to make donations to the Clonezilla, GParted and EasyBCD projects, as I would have been unable to set up my dual-boot system without them. It's also worth taking a moment to acknowledge the efforts of the Fedora community and the wider Linux community for making a complete operating system available and easy to install at no cost to the end user.

For my part I hope that this overview will be useful to someone else - and good luck with your own dual-booting installations.

Fedora 15 and Gnome 3: user basics

I've been using Fedora 15 for about a month now and thought it was time to write up some of my experiences with the new Gnome 3 desktop, since certain aspects are quite a bit different from the previous version. I know other people have posted details about the Fedora 15 desktop (for example Xavier Claessens' One Week with Gnome 3) but when I first installed it there didn't seem to be much from a "user basics" perspective. So this is my take, hope it's useful.

Getting started

When you first start up Gnome 3 the desktop looks pretty empty - in fact there are no desktop icons in this new version (even when you put them in your "Desktop" subdirectory):

Figure 1: "Empty" Gnome 3 desktop on startup. No desktop icons allowed!

To get started, move the mouse to the word "Activities" at the top right-hand corner of the screen (the so-called "hot corner") - immediately changing the desktop to the "Windows" view:

Figure 2: "exploded view" of the Gnome 3 desktop, accessed either by moving the mouse over the "Activities" hot corner (top-left of the screen), or by hitting the "Gnome" (i.e. Windows) key on the keyboard. The Favourites sidebar sits on the left edge of the screen, and the edge of the Workspaces sidebar peeks out on the right.

In this view (figure 2) you can see the Favourites sidebar on the left side, and just the very edge of the Workspaces sidebar on the right (more about those below). I call this the exploded view of the current workspace, since (as in this example) features minatures of any windows in the workspace. The exploded view can also be toggled by pressing the "Gnome key" "Super key" (i.e. the Windows key).

From this view you can toggle to the "Applications" view, by clicking on "Applications" near the top left (figure 3):

Figure 3: "Applications" view of the Gnome 3 desktop

This shows all the applications installed on the system, with a search box and category groupings on the right to help you find the one you want.

• Drag icons from this view to the "Favourites" bar to make them more easily accessible in future.
  
• The "Add/remove software" application is a graphical front end to yum for installing and managing additional packages that are weren't included by default.

The Favourites sidebar: launching and navigating applications

The Favourites sidebar is the strip down the left-hand side which holds various application icons. These icons do "double-duty": if you've dragged an icon there from elsewhere (essentially "favouriting" it), then it initially acts as a launcher for that application; also the icons for any running applications (favourited or otherwise) will appear here.

If an application already has one or more instances running then clicking on its icon takes you to the "nearest" running instance; clicking-and-holding (or right clicking) gives you more options (e.g. to start a new instance, or move to any of the currently open instances) - behaviour that is probably already familiar to users of Windows 7 and Mac OS X (figure 4):

Figure 4: The "Favourites" sidebar with dialogue (i.e. the black bubble) opened for Firefox after right-clicking on its icon. This gives options to move to a running Firefox window, or to start a new Firefox instance.

Another way to navigate between running applications is to use Alt+Tab to cycle between them (figure 5):

Figure 5: Alt+Tab moves between running applications...

Repeated Alt+Tabb'ing moves between the applications; if there's more than one instance running then these are also shown when you Alt+Tab to it, and you can use the arrow keys to navigate to the specific one you want (figure 6):

Figure 6: ... and arrow keys allow you to select a specific instance if there are multiple instances of a particular application.

The Workspaces sidebar: navigating multiple desktops

Workspaces provide a way to manage applications, by giving the user multiple virtual desktops. These should already be familiar to seasoned Gnome users, but they operate somewhat differently in Gnome 3: there are no longer a fixed number of workspaces, instead they are created and destroyed automatically by the system as required.

You can move between workspaces in at least two different ways. Firstly, you can access the workspaces sidebar on the right-hand side of the screen, by moving the mouse over it in the "exploded view" of the desktop and causing it to "pop out" (figure 7):

Figure 7: the Workspaces sidebar "popped out" on the right of the screen in the exploded view of the Gnome 3 desktop.

The sidebar shows miniatures of each workspace, with the current workspace highlighted with a white outline. Clicking on one of the images takes you to that workspace; you can also drag application windows between the different workspaces.

Note the sidebar also shows an extra "empty" workspace at the bottom: if an application is opened or moved into this workspace then a new empty workspace is automatically created underneath. Furthermore, there's only ever one empty workspace - so if a workspace "empties" (e.g. because you've closed all the applications it contains) then Gnome automatically removes it. This can be quite disconcerting, and is probably the feature that causes me the most confusion in practice as it often upsets my sense of where I am in the workspace order.

The other way to navigate between workspaces is Alt+Ctrl+Up/Down Arrows, which I find myself using quite a lot (although I frequently overshoot into the empty workspace by accident) (figure 8):

Figure 8: moving between multiple desktops using Alt+Ctrl+Up/Down keys

Other observations

• Resizing windows: windows can be maximised by double-clicking on their title bar (double-click again to restore to the original size). There's no "minimise" button on the window frame, so you now have to right-click and then select the "Minimise" menu option. Also, note that dragging a window to the top of the screen automatically causes it to maximise (again similar to Windows 7, and not always what you intend). Manual resizing is also possible as always, by dragging the window edges - but this can be fiddly, as the area where an edge can be "caught" for dragging seems to be quite small.
  

• System notifications: these now pop up rather discreetly at the bottom of the screen, but interacting with then can be frustrating at times - often they disappear before you have a chance to read them, and sometimes (counterintuitvely) disappear when clicked.
  

• Customisation: while some preference-type options are available via the "username" menu (top right-hand corner of the screen) under "System Settings", overall the customisation options feel quite limited (for example, no screen-savers). However as a number of interfaces to system tools currently only seem to be accessible by launching from a command line, it's not clear if this a conscious design decision or whether more customisation options will be exposed in future versions.

• Fallback mode: this is a half-way house between Gnome 2 and Gnome 3, and is started by default on systems which can't support the full Gnome 3 experience (which appears to include virtual machines). However as it's much more like the old Gnome, if you really don't like the new version then you could try using fallback mode instead.

Conclusion

Having been using Fedora 15 and Gnome 3 day-to-day for a few weeks now, I'm now largely used to its quirks and finding it overall a perfectly serviceable working environment - for me the new workspaces model and the rather random system notification mechanism have proved to be the most challenging differences from previous versions. So while it may not suit everyone's tastes it's definitely worth trying (and hopefully the more egregious foibles will be ironed out in future versions).

Mac OS X: new user tips

Over the couple of weeks I've been using an Apple iMac, and as a Windows/Linux user I've found navigating the desktop has been something of a learning experience for me.

As different as they are, in many ways the standard Windows and Linux desktops are idiomatically quite similar these days, and both support the standard PC three-button mouse. By contrast the Mac OS X desktop environment (and its use of the infamous one-button mouse) has a number of differences which can turn even basic operations (for example, cut-and-paste) initially into something of a challenge.

However some basic knowledge should go a long way in helping. First, there are the three essential keys you need to know about:

• Command key: [⌘]
• Option key: [⌥]
• Control key: [ctrl]

(The links give more background but aren't essential to the following. You can think of the option key as being the same as the "Alt" key on Windows/Linux.)

Then:

• Emulating the right-hand mouse button: [ctrl] + mouse click (essential for desktop and web applications that use this to activate context menus and so on)

Basic text editing operations:

• Cut: [⌘] + [x]
• Copy: [⌘] + [x]
• Paste: [⌘] + [x]

Basic keys for navigating within text documents:

• Home: [↖]
• End: [↘]
• Page up: [⇞]
  
• Page down: [⇟]
  

Useful shortcuts for navigating the desktop:

• Cycle between open windows: [⌥] + [tab]
• Zoom out (pulls back to show all open windows): [F9]
• Show desktop (hides all open windows): [F11]
  

And finally (and essential if you're programming and find your Apple keyboard is missing a hash key):

• Hash symbol ("#"): [⌥] + [3]
  

These all work on OS X 10.4.11 ("Tiger"), which is admittedly no longer a very recent release, but hopefully they're also applicable to later Mac OSes. I can't say that I've fallen in love with Apple as a result, but they have enabled me to operate at an acceptably functional level (until I can get my Linux workstation up and running!).

Managing Python packages: virtualenv, pip and yolk

I've recently been playing with the Python virtualenv package - along with pip and yolk - as a way of managing third-party packages. This post is my brief introduction to the basics of these three tools.

virtualenv lets you create isolated self-contained "virtual environments" which are separate from the system Python. You can then install and manage the specific Python packages that you need for a particular application - safe from potential problems due to version incompatibilities, and without needing superuser privileges - using the pip package installer. yolk provides an extra utility to keep track of what's installed.

1. virtualenv: building virtual Python environments

virtualenv can either be installed via your system's package manager (for example, synaptic on Ubuntu), or by using the easy_install tool, i.e.:

$ easy_install virtualenv

(If you don't have the SetupTools package which provides easy_install then you can download the "bootstrap" install script from http://peak.telecommunity.com/dist/ez_setup.py. Save as ez_setup.py and run using /path/to/python ez_setup.py.)

Once virtualenv is installed you can create a new virtual environment (called in this example, "myenv") as follows:

$ virtualenv --no-site-packages myenv

This makes a new directory myenv in the current directory (which will contain bin, include and lib subdirectories) based on the system version of Python. The --no-site-packages option tells virtualenv not to include any third-party packages which might have been installed into the system Python (see the virtualenv documentation for details of other options).

To start using the new environment, run the environment's "activate" command e.g.:

$ source myenv/bin/activate

The shell command prompt will change from e.g. $ to (myenv)$, indicating that the "myenv" environment (and any packages installed in it) will be used instead of the system Python for applications run in this shell. (Note that the Python application code doesn't need to be inside the virtual environment directory; in fact this directory is just using for the packages associated with the virtual environment.)

Finally, when you've finished working with the virtual environment you can leave it by running the deactivate command (also in the bin directory).

(On Windows you may have to specify the full path to the "Scripts" directory of your Python installation when invoking the easy_install and virtualenv commands above, e.g. C:\Python27\Scripts\virtualenv. Also, note that when a virtual environment is created it won't contain a "bin" directory - instead it's activated by invoking the Scripts\activate batch file in the virtual environment directory. Invoking the deactivate command exits the environment as before.)

2. pip: installing Python packages

Once you're created a virtual environment you can start to add packages (which is really the point of doing this in the first place). virtualenv automatically includes both easy_install and an alternative package installer called pip (at least, for virtualenv 1.4.1 and up; earlier versions only have easy_install, so you'll need to run easy_install pip within the virtual environment in order to get it).

Most packages that are easy_installable can also be installed using pip, and it's designed to work well with virtualenv. However I think its main advantage is that it offers some useful functionality that's missing from easy_install - most significantly, the ability to uninstall previously installed packages. (Other useful features include the ability to explicitly control and export versions of third-party package dependencies via "requirements files" - see the pip documentation for more details.)

Basic pip usage looks like this:

(myenv)$ pip install python-dateutil # install latest version of a package

(myenv)$ pip uninstall python-dateutil # remove package

(myenv)$ pip install python-dateutil==1.5 # install specific version

(As an aside, the python-dateutil package is illustrative of one of the advantages of using pip over easy_install: after installing the latest version of python-dateutil, I discovered that it's only compatible with Python 3 - an earlier 1.* version is required to work with Python 2. pip let me uninstall the newer version and reinstall the older one.)

3. yolk: checking Python packages installed on your system

The final utility I'd recommend is yolk, which provides a way of querying which packages (and versions) have been installed in the current environment. It also has options to query PyPI (the Python Package Index). Installing it is easy:

(myenv)$ pip install yolk

Running it with the -l option (for "list") then shows us what packages are available:

(myenv)$ yolk -l
Python          - 2.6.4        - active development (/usr/lib/python2.6/lib-dynload)
pip             - 1.0          - active
python-dateutil - 1.5          - active
setuptools      - 0.6c9        - active
wsgiref         - 0.1.2        - active development (/usr/lib/python2.6)
yolk            - 0.4.1        - active

(See the yolk documentation to learn more about its other features.)

Summary

Obviously the above is just an introduction to the basics of virtualenv, pip and yolk for managing and working with third-party packages - but hopefully it's enough to get started. If you're interested in using virtualenv in practice then Doug Hellman's article about working with multiple virtual environments (and his virtualenvwrapper project, which provides tools to help) is recommended as a starting point for further reading.

MySQL user administration basics

Last week I spent a frustrating few hours setting up a script to copy a MySQL-backed web application from CVS into a local test environment. In the process I somehow broke the database access settings in a way I didn't understand, and - despite the fact that it was probably quite a trivial error - all my attempts at manually resetting the MySQL user and password failed. Finally I resorted to using phpMyAdmin to sort out the immediate problem, but afterwards I felt I needed to learn more about the basics of MySQL user administration.

First, a bit of background: it's typical for applications such as mine to access the database via a dedicated MySQL user explicitly created for the purpose, and with access rights restricted to only the data that it actually needs. This acts as a security measure to limit the potential damage to the database as a whole if the application is compromised. (Any application that uses MySQL for its database backend should also implement its own independent application-specific user management system, but this is a separate issue not covered here.)

The first point is that MySQL's users are entirely distinct and separate from the users defined on the host system - so they also need to be managed separately. The essential SQL commands for doing this are:

• GRANT: creates a new user and sets its privileges (if the username/hostname pair specified doesn't already exist); modifies the privileges for an existing user (if it does exist):
  
  GRANT privileges ON data TO user IDENTIFIED BY password;
  
  For example:
  
  GRANT SELECT,INSERT,UPDATE,DELETE ON myappdb.* TO 'myappdbuser'@'localhost' IDENTIFIED BY 'quitesecret';
  
  gives the user myappdbuser@localhost a limited set of privileges for all tables in the myappdb database. (To grant all permissions, privileges can be specified as ALL; data can be specified as *.* to grant the rights to all tables in all databases. Also, note that more recent versions of MySQL seem to support an explicit CREATE USER command in addition to GRANT.)


• REVOKE: the opposite of GRANT; removes a user's privileges:
  
  REVOKE privileges ON data FROM user;
  
  For example:
  
  REVOKE INSERT,UPDATE,DELETE ON myappdb.* FROM 'myappdbuser'@'localhost';


• DROP USER: removes a user, for example:
  
  DROP USER 'myappdbuser'@'localhost';


• SET PASSWORD sets or changes the password for a user, for example:
  
  SET PASSWORD FOR 'myappdbuser'@'localhost' = PASSWORD('newsecret');


• To get a list of users as username/host pairs: execute a SELECT query on the user table of MySQL's mysql administrative database (where the user data is stored):
  
  SELECT User,Host FROM mysql.users;

These commmands are actually very straightforward, however based on recent experience two particular points are worth emphasizing:

1. Each user is uniquely identified by a username and host pair (not just by the username) - so to MySQL fred@localhost, fred@example.com and fred@% are all different and distinct users.
2. Each user has its own set of associated privileges, specifying which database operations the user is allowed to perform and on which data - so it's possible for each of fred@localhost, fred@example.com and fred@% to have different privileges.

These are significant when considering connection requests where there is an ambiguity in the user specification (for example, omitting the hostname) which results in more than one potential match in the user table. In this case MySQL uses the most specific match, which might not be the one that was intended. The connection might then be denied (if the intended and actual users have different passwords), or have subsequent problems executing SQL queries (if the two users have different privileges). So I'd also recommend always specifying users explicitly whenever possible with the full username@hostname pair.

Hopefully this overview has given some insight into the basics of MySQL user administration, although obviously there's a lot more than I've outlined here (for example I've skipped over many details are for the GRANT command - see the section Account Management Statements in the MySQL manual for more comprehensive information) - and I'd still recommend using phpMyAdmin or similar for database user management (especially if you don't do it that often). However I hope it's still useful - and I'd welcome any comments or corrections.

Basic CVS from Emacs

I've been a long time user of Emacs both on UNIX/Linux and on Windows, and I'm reasonably proficient now I suppose - I have a small set of keyboard short cuts that I use regularly for things like switching between buffers and doing query-replace operations - but I'm aware that I've only scratched the surface of what the program is capable of (after all, I still think it's "just" an editor).

So my most recent discovery is probably not news to a lot of people, but I still think it's pretty neat: specifically, the ability to access CVS version control operations on a file directly from within Emacs using its standard version control interface, vc.el.

A quick summary of the basics: to load vc.el and activate the functions, first do Ctrl-x Ctrl-q, then:

• Use the sequence Ctrl-x v = to see differences between the current buffer and the last CVS version (i.e. cvs diff), and
  
• Use the sequence Ctrl-x v v to commit your changes - this opens a new buffer to write the commit message, and when you're done use Ctrl-c Ctrl-c to finish the commit.

Other operations are available but I think these are the ones that will be most useful to me, as it means I won't need to leave the editor in order to commit changes from the command line. (Also with this method I don't need to "revert" the buffer after CVS has updated any keywords in the file, which I always have to do when using editor-plus-command-line.)

So far I've been using this with GNU Emacs 22.2.1 and CVS 1.12.3 on Ubuntu Linux. I'm guessing that Emacs works out that the file is under version control by detecting the CVS subdirectory, and that the same (or similar) key sequences might also work for SVN - but I have no idea yet if this will work on Windows using Xemacs and TortoiseCVS (I will report back once I've had a chance to investigate further).

Simple distributed version control using Dropbox

Over the last 2 years I've been developing a few personal software projects, and while I'd happily been using CVS for version control on one machine (a laptop running Windows XP), there were two significant friction points: 1) how to transfer code to my other machines (I'm running two other separate operating systems - Ubuntu Linux and Windows XP - on a single dual-booting desktop PC), and then merge any changes or fixes back again, and 2) how to ensure that the CVS repositories were being regularly and reliably backed up.

One of the issues with the first problem is that while it's irritating and error-prone to work around (and has a deterrent effect on performing cross-system testing), it doesn't cause enough real pain to force you to properly deal with it. The second problem is that hardy perennial, that making back ups never really feels that urgent (until immediately after your hard drive fails).

So I had been limping along for a while until a few months ago when I read an article called "Easy Version Control" by Ryan Taylor (havocinspired.co.uk) in August's .net magazine. Ryan's article covers a lot of other things, but for me the key suggestion is to use the free Dropbox file synchronisation service (which also provides you with up to 2GB of online storage) to back up and share the version control repositories.

The actual set up process is simple:

1. Install the Dropbox client on the first machine,
2. Put your repositories inside the special Dropbox directory/folder that is created when the client is installed,
3. Install Dropbox clients on each of the other machines where you want to access the repositories.

The Dropbox clients do the rest: they automatically synchronise the Dropbox directories/folders - including the repositories - across all machines. You can then check out working copies of your code and commit changes back to the repository on any machine, which will be automatically reflected on all the others. Additionally, a synchronised copy of the repository is also held on the Dropbox servers (thus taking care of the back up issue).

In Ryan's article he works with SVN rather than CVS but this doesn't appear to be a problem - neither does sharing repositories between Linux and Windows (where I'm using TortoiseCVS ). (He also suggests that several people could work with the same repository at once by enabling sharing in Dropbox for the appropriate directories/folders.) So all in all this seems like an ideal solution for simple distributed version control for personal projects like mine.