Disclosure: a free e-copy of this book was received from the publisher for review purposes. The opinions expressed here are entirely my own; a copy of this review has also been posted at Amazon.
Greg L. Turnquist’s “Python Testing Cookbook” explores automated testing at all levels, with the intention of providing the reader with the knowledge needed to implement testing using Python tools to improve software quality. To this end the book presents over 70 “recipes” in its nine chapters (ranging from the basics of unit testing, through test suites, user acceptance and web application testing, continuous integration, and methods for smoke- and load-testing), covering both tools for testing Python, and Python tools for testing. It also delivers advice about how to get the most from automated testing, which is as much an art as a science.
The first three chapters introduce the fundamentals: writing, organising and running unit tests, comprehensively covering unittest (Python’s built-in unit testing library), nose (a versatile tool for discovering, running and reporting tests) and doctest (which turns Python docstrings into testable code – a sample of this chapter can be downloaded from http://www.packtpub.com/python-testing-cookbook/book). Having established a solid foundation, subsequent chapters look at increasingly broader levels of automated testing using the appropriate relevant Python tools: for example, the “lettuce” and “should_DSL” libraries for “behaviour driven development” (an extension of “test driven development” which aims to produce human-readable test cases and reports), and the “Pyccuracy” and “Robot” frameworks for end-user acceptance testing of web applications. Later chapters cover higher level concepts and tools, such as using nose to hook Python tests into “continuous integration” servers (both Jenkins and TeamCity are covered in detail), and assessing test coverage using the “coverage” tool (both as a metric, and to identify areas that need more tests). A detailed chapter on smoke- and load-testing includes practical advice on developing multiple test suites for different scenarios, and methods for stress-testing (for example, by capturing and replaying real world data) to discover weaknesses in a system before going to production. The final chapter distils the author’s experience into general advice on making testing a successful part of your code development methodology, both for new and legacy projects.
There’s a lot of good stuff in this book: the initial chapters on unittest and nose are particularly strong, and I can imagine returning to these in future as a reference. There is also a lot of excellent and hard-won practical advice from the author’s own experience – not only in these early chapters but throughout the book – which is consistently valuable (in this regard the final chapter is a real highlight and could easily stand alone – I will definitely be re-reading it soon). Elsewhere the various tools and topics are presented clearly with plenty of useful detail, and in some cases have demystified things that I’d always assumed were quite esoteric and difficult to do (nose in particular was a revelation to me, but also setting up continuous integration servers and measuring test coverage).
There are a few disappointments: the section on mock objects left me feeling baffled as to how to actually implement them in practice – a shame as it was something that I’d looked forward to learning. I’d also have liked something about approaches for handling difficult testing scenarios such as software which interacts with the file system or with large files – a few hints here would have been invaluable for me. There are typos in some commands and code in a few recipes (e.g. for nose), which meant I had to look up the correct syntax elsewhere – perhaps not so bad, but annoying (especially in a cookbook) – and since the recipes themselves aren’t numbered, this sometimes made it difficult to navigate between them.
However these are fairly minor quibbles, and in conclusion I was impressed with both the breadth of material covered by the book and the level of detail for many topics. Moreover I enjoyed reading it and was often left feeling excited at the prospect of being able to apply the ideas to my own projects, which is I think was one of the author’s aims (and no mean feat for a technical book). I think that the combination of the detail together with the author’s practical advice make this book both an excellent introduction to testing with Python, and a valuable resource to refer back to subsequently.
(Addendum: Greg Turnquist's blog about the book can be found at http://pythontestingcookbook.posterous.com/ and features some interesting supplementary material.)
Wednesday, July 27, 2011
Sunday, July 10, 2011
Fedora 15 and Gnome 3: user basics
I've been using Fedora 15 for about a month now and thought it was time to write up some of my experiences with the new Gnome 3 desktop, since certain aspects are quite a bit different from the previous version. I know other people have posted details about the Fedora 15 desktop (for example Xavier Claessens' One Week with Gnome 3) but when I first installed it there didn't seem to be much from a "user basics" perspective. So this is my take, hope it's useful.
Getting started
When you first start up Gnome 3 the desktop looks pretty empty - in fact there are no desktop icons in this new version (even when you put them in your "Desktop" subdirectory):
Figure 1: "Empty" Gnome 3 desktop on startup. No desktop icons allowed!
To get started, move the mouse to the word "Activities" at the top right-hand corner of the screen (the so-called "hot corner") - immediately changing the desktop to the "Windows" view:
Figure 2: "exploded view" of the Gnome 3 desktop, accessed either by moving the mouse over the "Activities" hot corner (top-left of the screen), or by hitting the "Gnome" (i.e. Windows) key on the keyboard. The Favourites sidebar sits on the left edge of the screen, and the edge of the Workspaces sidebar peeks out on the right.
In this view (figure 2) you can see the Favourites sidebar on the left side, and just the very edge of the Workspaces sidebar on the right (more about those below). I call this the exploded view of the current workspace, since (as in this example) features minatures of any windows in the workspace. The exploded view can also be toggled by pressing the"Gnome key" "Super key" (i.e. the Windows key).
From this view you can toggle to the "Applications" view, by clicking on "Applications" near the top left (figure 3):
Figure 3: "Applications" view of the Gnome 3 desktop
This shows all the applications installed on the system, with a search box and category groupings on the right to help you find the one you want.
The Favourites sidebar: launching and navigating applications
The Favourites sidebar is the strip down the left-hand side which holds various application icons. These icons do "double-duty": if you've dragged an icon there from elsewhere (essentially "favouriting" it), then it initially acts as a launcher for that application; also the icons for any running applications (favourited or otherwise) will appear here.
If an application already has one or more instances running then clicking on its icon takes you to the "nearest" running instance; clicking-and-holding (or right clicking) gives you more options (e.g. to start a new instance, or move to any of the currently open instances) - behaviour that is probably already familiar to users of Windows 7 and Mac OS X (figure 4):
Figure 4: The "Favourites" sidebar with dialogue (i.e. the black bubble) opened for Firefox after right-clicking on its icon. This gives options to move to a running Firefox window, or to start a new Firefox instance.
Another way to navigate between running applications is to use Alt+Tab to cycle between them (figure 5):
Figure 5: Alt+Tab moves between running applications...
Repeated Alt+Tabb'ing moves between the applications; if there's more than one instance running then these are also shown when you Alt+Tab to it, and you can use the arrow keys to navigate to the specific one you want (figure 6):
Figure 6: ... and arrow keys allow you to select a specific instance if there are multiple instances of a particular application.
The Workspaces sidebar: navigating multiple desktops
Workspaces provide a way to manage applications, by giving the user multiple virtual desktops. These should already be familiar to seasoned Gnome users, but they operate somewhat differently in Gnome 3: there are no longer a fixed number of workspaces, instead they are created and destroyed automatically by the system as required.
You can move between workspaces in at least two different ways. Firstly, you can access the workspaces sidebar on the right-hand side of the screen, by moving the mouse over it in the "exploded view" of the desktop and causing it to "pop out" (figure 7):
Figure 7: the Workspaces sidebar "popped out" on the right of the screen in the exploded view of the Gnome 3 desktop.
The sidebar shows miniatures of each workspace, with the current workspace highlighted with a white outline. Clicking on one of the images takes you to that workspace; you can also drag application windows between the different workspaces.
Note the sidebar also shows an extra "empty" workspace at the bottom: if an application is opened or moved into this workspace then a new empty workspace is automatically created underneath. Furthermore, there's only ever one empty workspace - so if a workspace "empties" (e.g. because you've closed all the applications it contains) then Gnome automatically removes it. This can be quite disconcerting, and is probably the feature that causes me the most confusion in practice as it often upsets my sense of where I am in the workspace order.
The other way to navigate between workspaces is Alt+Ctrl+Up/Down Arrows, which I find myself using quite a lot (although I frequently overshoot into the empty workspace by accident) (figure 8):
Figure 8: moving between multiple desktops using Alt+Ctrl+Up/Down keys
Other observations
Conclusion
Having been using Fedora 15 and Gnome 3 day-to-day for a few weeks now, I'm now largely used to its quirks and finding it overall a perfectly serviceable working environment - for me the new workspaces model and the rather random system notification mechanism have proved to be the most challenging differences from previous versions. So while it may not suit everyone's tastes it's definitely worth trying (and hopefully the more egregious foibles will be ironed out in future versions).
Getting started
When you first start up Gnome 3 the desktop looks pretty empty - in fact there are no desktop icons in this new version (even when you put them in your "Desktop" subdirectory):
Figure 1: "Empty" Gnome 3 desktop on startup. No desktop icons allowed!
To get started, move the mouse to the word "Activities" at the top right-hand corner of the screen (the so-called "hot corner") - immediately changing the desktop to the "Windows" view:
Figure 2: "exploded view" of the Gnome 3 desktop, accessed either by moving the mouse over the "Activities" hot corner (top-left of the screen), or by hitting the "Gnome" (i.e. Windows) key on the keyboard. The Favourites sidebar sits on the left edge of the screen, and the edge of the Workspaces sidebar peeks out on the right.
In this view (figure 2) you can see the Favourites sidebar on the left side, and just the very edge of the Workspaces sidebar on the right (more about those below). I call this the exploded view of the current workspace, since (as in this example) features minatures of any windows in the workspace. The exploded view can also be toggled by pressing the
From this view you can toggle to the "Applications" view, by clicking on "Applications" near the top left (figure 3):
Figure 3: "Applications" view of the Gnome 3 desktop
This shows all the applications installed on the system, with a search box and category groupings on the right to help you find the one you want.
- Drag icons from this view to the "Favourites" bar to make them more easily accessible in future.
- The "Add/remove software" application is a graphical front end to yum for installing and managing additional packages that are weren't included by default.
The Favourites sidebar: launching and navigating applications
The Favourites sidebar is the strip down the left-hand side which holds various application icons. These icons do "double-duty": if you've dragged an icon there from elsewhere (essentially "favouriting" it), then it initially acts as a launcher for that application; also the icons for any running applications (favourited or otherwise) will appear here.
If an application already has one or more instances running then clicking on its icon takes you to the "nearest" running instance; clicking-and-holding (or right clicking) gives you more options (e.g. to start a new instance, or move to any of the currently open instances) - behaviour that is probably already familiar to users of Windows 7 and Mac OS X (figure 4):
Figure 4: The "Favourites" sidebar with dialogue (i.e. the black bubble) opened for Firefox after right-clicking on its icon. This gives options to move to a running Firefox window, or to start a new Firefox instance.
Another way to navigate between running applications is to use Alt+Tab to cycle between them (figure 5):
Figure 5: Alt+Tab moves between running applications...
Repeated Alt+Tabb'ing moves between the applications; if there's more than one instance running then these are also shown when you Alt+Tab to it, and you can use the arrow keys to navigate to the specific one you want (figure 6):
Figure 6: ... and arrow keys allow you to select a specific instance if there are multiple instances of a particular application.
The Workspaces sidebar: navigating multiple desktops
Workspaces provide a way to manage applications, by giving the user multiple virtual desktops. These should already be familiar to seasoned Gnome users, but they operate somewhat differently in Gnome 3: there are no longer a fixed number of workspaces, instead they are created and destroyed automatically by the system as required.
You can move between workspaces in at least two different ways. Firstly, you can access the workspaces sidebar on the right-hand side of the screen, by moving the mouse over it in the "exploded view" of the desktop and causing it to "pop out" (figure 7):
Figure 7: the Workspaces sidebar "popped out" on the right of the screen in the exploded view of the Gnome 3 desktop.
The sidebar shows miniatures of each workspace, with the current workspace highlighted with a white outline. Clicking on one of the images takes you to that workspace; you can also drag application windows between the different workspaces.
Note the sidebar also shows an extra "empty" workspace at the bottom: if an application is opened or moved into this workspace then a new empty workspace is automatically created underneath. Furthermore, there's only ever one empty workspace - so if a workspace "empties" (e.g. because you've closed all the applications it contains) then Gnome automatically removes it. This can be quite disconcerting, and is probably the feature that causes me the most confusion in practice as it often upsets my sense of where I am in the workspace order.
The other way to navigate between workspaces is Alt+Ctrl+Up/Down Arrows, which I find myself using quite a lot (although I frequently overshoot into the empty workspace by accident) (figure 8):
Figure 8: moving between multiple desktops using Alt+Ctrl+Up/Down keys
Other observations
- Resizing windows: windows can be maximised by double-clicking on their title bar (double-click again to restore to the original size). There's no "minimise" button on the window frame, so you now have to right-click and then select the "Minimise" menu option. Also, note that dragging a window to the top of the screen automatically causes it to maximise (again similar to Windows 7, and not always what you intend). Manual resizing is also possible as always, by dragging the window edges - but this can be fiddly, as the area where an edge can be "caught" for dragging seems to be quite small.
- System notifications: these now pop up rather discreetly at the bottom of the screen, but interacting with then can be frustrating at times - often they disappear before you have a chance to read them, and sometimes (counterintuitvely) disappear when clicked.
- Customisation: while some preference-type options are available via the "username" menu (top right-hand corner of the screen) under "System Settings", overall the customisation options feel quite limited (for example, no screen-savers). However as a number of interfaces to system tools currently only seem to be accessible by launching from a command line, it's not clear if this a conscious design decision or whether more customisation options will be exposed in future versions.
- Fallback mode: this is a half-way house between Gnome 2 and Gnome 3, and is started by default on systems which can't support the full Gnome 3 experience (which appears to include virtual machines). However as it's much more like the old Gnome, if you really don't like the new version then you could try using fallback mode instead.
Conclusion
Having been using Fedora 15 and Gnome 3 day-to-day for a few weeks now, I'm now largely used to its quirks and finding it overall a perfectly serviceable working environment - for me the new workspaces model and the rather random system notification mechanism have proved to be the most challenging differences from previous versions. So while it may not suit everyone's tastes it's definitely worth trying (and hopefully the more egregious foibles will be ironed out in future versions).
Monday, May 30, 2011
Mac OS X: new user tips
Over the couple of weeks I've been using an Apple iMac, and as a Windows/Linux user I've found navigating the desktop has been something of a learning experience for me.
As different as they are, in many ways the standard Windows and Linux desktops are idiomatically quite similar these days, and both support the standard PC three-button mouse. By contrast the Mac OS X desktop environment (and its use of the infamous one-button mouse) has a number of differences which can turn even basic operations (for example, cut-and-paste) initially into something of a challenge.
However some basic knowledge should go a long way in helping. First, there are the three essential keys you need to know about:
Then:
As different as they are, in many ways the standard Windows and Linux desktops are idiomatically quite similar these days, and both support the standard PC three-button mouse. By contrast the Mac OS X desktop environment (and its use of the infamous one-button mouse) has a number of differences which can turn even basic operations (for example, cut-and-paste) initially into something of a challenge.
However some basic knowledge should go a long way in helping. First, there are the three essential keys you need to know about:
- Command key: [⌘]
- Option key: [⌥]
- Control key: [ctrl]
Then:
- Emulating the right-hand mouse button: [ctrl] + mouse click (essential for desktop and web applications that use this to activate context menus and so on)
- Cut: [⌘] + [x]
- Copy: [⌘] + [x]
- Paste: [⌘] + [x]
- Home: [↖]
- End: [↘]
- Page up: [⇞]
- Page down: [⇟]
- Cycle between open windows: [⌥] + [tab]
- Zoom out (pulls back to show all open windows): [F9]
- Show desktop (hides all open windows): [F11]
- Hash symbol ("#"): [⌥] + [3]
Saturday, April 9, 2011
Managing Python packages: virtualenv, pip and yolk
I've recently been playing with the Python virtualenv package - along with pip and yolk - as a way of managing third-party packages. This post is my brief introduction to the basics of these three tools.
virtualenv lets you create isolated self-contained "virtual environments" which are separate from the system Python. You can then install and manage the specific Python packages that you need for a particular application - safe from potential problems due to version incompatibilities, and without needing superuser privileges - using the pip package installer. yolk provides an extra utility to keep track of what's installed.
1. virtualenv: building virtual Python environments
virtualenv can either be installed via your system's package manager (for example, synaptic on Ubuntu), or by using the easy_install tool, i.e.:
$ easy_install virtualenv
(If you don't have the SetupTools package which provides easy_install then you can download the "bootstrap" install script from http://peak.telecommunity.com/dist/ez_setup.py. Save as ez_setup.py and run using /path/to/python ez_setup.py.)
Once virtualenv is installed you can create a new virtual environment (called in this example, "myenv") as follows:
$ virtualenv --no-site-packages myenv
This makes a new directory myenv in the current directory (which will contain bin, include and lib subdirectories) based on the system version of Python. The --no-site-packages option tells virtualenv not to include any third-party packages which might have been installed into the system Python (see the virtualenv documentation for details of other options).
To start using the new environment, run the environment's "activate" command e.g.:
$ source myenv/bin/activate
The shell command prompt will change from e.g. $ to (myenv)$, indicating that the "myenv" environment (and any packages installed in it) will be used instead of the system Python for applications run in this shell. (Note that the Python application code doesn't need to be inside the virtual environment directory; in fact this directory is just using for the packages associated with the virtual environment.)
Finally, when you've finished working with the virtual environment you can leave it by running the deactivate command (also in the bin directory).
(On Windows you may have to specify the full path to the "Scripts" directory of your Python installation when invoking the easy_install and virtualenv commands above, e.g. C:\Python27\Scripts\virtualenv. Also, note that when a virtual environment is created it won't contain a "bin" directory - instead it's activated by invoking the Scripts\activate batch file in the virtual environment directory. Invoking the deactivate command exits the environment as before.)
2. pip: installing Python packages
Once you're created a virtual environment you can start to add packages (which is really the point of doing this in the first place). virtualenv automatically includes both easy_install and an alternative package installer called pip (at least, for virtualenv 1.4.1 and up; earlier versions only have easy_install, so you'll need to run easy_install pip within the virtual environment in order to get it).
Most packages that are easy_installable can also be installed using pip, and it's designed to work well with virtualenv. However I think its main advantage is that it offers some useful functionality that's missing from easy_install - most significantly, the ability to uninstall previously installed packages. (Other useful features include the ability to explicitly control and export versions of third-party package dependencies via "requirements files" - see the pip documentation for more details.)
Basic pip usage looks like this:
(myenv)$ pip install python-dateutil # install latest version of a package
(myenv)$ pip uninstall python-dateutil # remove package
(myenv)$ pip install python-dateutil==1.5 # install specific version
(As an aside, the python-dateutil package is illustrative of one of the advantages of using pip over easy_install: after installing the latest version of python-dateutil, I discovered that it's only compatible with Python 3 - an earlier 1.* version is required to work with Python 2. pip let me uninstall the newer version and reinstall the older one.)
3. yolk: checking Python packages installed on your system
The final utility I'd recommend is yolk, which provides a way of querying which packages (and versions) have been installed in the current environment. It also has options to query PyPI (the Python Package Index). Installing it is easy:
(myenv)$ pip install yolk
Running it with the -l option (for "list") then shows us what packages are available:
Summary
Obviously the above is just an introduction to the basics of virtualenv, pip and yolk for managing and working with third-party packages - but hopefully it's enough to get started. If you're interested in using virtualenv in practice then Doug Hellman's article about working with multiple virtual environments (and his virtualenvwrapper project, which provides tools to help) is recommended as a starting point for further reading.
virtualenv lets you create isolated self-contained "virtual environments" which are separate from the system Python. You can then install and manage the specific Python packages that you need for a particular application - safe from potential problems due to version incompatibilities, and without needing superuser privileges - using the pip package installer. yolk provides an extra utility to keep track of what's installed.
1. virtualenv: building virtual Python environments
virtualenv can either be installed via your system's package manager (for example, synaptic on Ubuntu), or by using the easy_install tool, i.e.:
$ easy_install virtualenv
(If you don't have the SetupTools package which provides easy_install then you can download the "bootstrap" install script from http://peak.telecommunity.com/dist/ez_setup.py. Save as ez_setup.py and run using /path/to/python ez_setup.py.)
Once virtualenv is installed you can create a new virtual environment (called in this example, "myenv") as follows:
$ virtualenv --no-site-packages myenv
This makes a new directory myenv in the current directory (which will contain bin, include and lib subdirectories) based on the system version of Python. The --no-site-packages option tells virtualenv not to include any third-party packages which might have been installed into the system Python (see the virtualenv documentation for details of other options).
To start using the new environment, run the environment's "activate" command e.g.:
$ source myenv/bin/activate
The shell command prompt will change from e.g. $ to (myenv)$, indicating that the "myenv" environment (and any packages installed in it) will be used instead of the system Python for applications run in this shell. (Note that the Python application code doesn't need to be inside the virtual environment directory; in fact this directory is just using for the packages associated with the virtual environment.)
Finally, when you've finished working with the virtual environment you can leave it by running the deactivate command (also in the bin directory).
(On Windows you may have to specify the full path to the "Scripts" directory of your Python installation when invoking the easy_install and virtualenv commands above, e.g. C:\Python27\Scripts\virtualenv. Also, note that when a virtual environment is created it won't contain a "bin" directory - instead it's activated by invoking the Scripts\activate batch file in the virtual environment directory. Invoking the deactivate command exits the environment as before.)
2. pip: installing Python packages
Once you're created a virtual environment you can start to add packages (which is really the point of doing this in the first place). virtualenv automatically includes both easy_install and an alternative package installer called pip (at least, for virtualenv 1.4.1 and up; earlier versions only have easy_install, so you'll need to run easy_install pip within the virtual environment in order to get it).
Most packages that are easy_installable can also be installed using pip, and it's designed to work well with virtualenv. However I think its main advantage is that it offers some useful functionality that's missing from easy_install - most significantly, the ability to uninstall previously installed packages. (Other useful features include the ability to explicitly control and export versions of third-party package dependencies via "requirements files" - see the pip documentation for more details.)
Basic pip usage looks like this:
(myenv)$ pip install python-dateutil # install latest version of a package
(myenv)$ pip uninstall python-dateutil # remove package
(myenv)$ pip install python-dateutil==1.5 # install specific version
(As an aside, the python-dateutil package is illustrative of one of the advantages of using pip over easy_install: after installing the latest version of python-dateutil, I discovered that it's only compatible with Python 3 - an earlier 1.* version is required to work with Python 2. pip let me uninstall the newer version and reinstall the older one.)
3. yolk: checking Python packages installed on your system
The final utility I'd recommend is yolk, which provides a way of querying which packages (and versions) have been installed in the current environment. It also has options to query PyPI (the Python Package Index). Installing it is easy:
(myenv)$ pip install yolk
Running it with the -l option (for "list") then shows us what packages are available:
(myenv)$ yolk -l
Python - 2.6.4 - active development (/usr/lib/python2.6/lib-dynload)
pip - 1.0 - active
python-dateutil - 1.5 - active
setuptools - 0.6c9 - active
wsgiref - 0.1.2 - active development (/usr/lib/python2.6)
yolk - 0.4.1 - active
(See the yolk documentation to learn more about its other features.)Summary
Obviously the above is just an introduction to the basics of virtualenv, pip and yolk for managing and working with third-party packages - but hopefully it's enough to get started. If you're interested in using virtualenv in practice then Doug Hellman's article about working with multiple virtual environments (and his virtualenvwrapper project, which provides tools to help) is recommended as a starting point for further reading.
Monday, April 4, 2011
Richard Stallman: "A Free Digital Society?"
About a month ago I was fortunate to attend an IET-hosted lecture by Richard Stallman, entitled "A Free Digital Society?". Probably most famous as the originator of the GNU project (out of which came GNU/Linux) and initiator of the free software movement, Stallman has for many years been an active and vocal advocate for free software, and has a campaigned against excessive extension of copyright laws
He began the talk with the observation that there is an implicit assumption in the recent movement towards "digital inclusion", that using computers and the internet is inherently good and beneficial. However, as the question mark in the title of his talk indicated this assumption merits closer attention, as (in his opinion) there are various issues and threats associated with these technologies. These include:
I'm still mulling over many of the issues raised. On the one hand there is a fundamental question about what moral rights you believe individuals should have, both generally and with specific regard to the digital world; and on the other there is the question of what you should do if you feel those rights are not being upheld. Stallman's position is clear and uncompromising: for example, not owning a mobile phone and not using a key card to enter his office (to avoid the possibility of being tracked), and using a netbook that allows him to run 100% free software (down to the BIOS level). It's certainly given me plenty to think about, and I'm looking forward to reading his book of collected essays "Free Software, Free Society" - which might be a good place to start if you're also interested in learning more.
He began the talk with the observation that there is an implicit assumption in the recent movement towards "digital inclusion", that using computers and the internet is inherently good and beneficial. However, as the question mark in the title of his talk indicated this assumption merits closer attention, as (in his opinion) there are various issues and threats associated with these technologies. These include:
- Survelliance: technology now makes it possible for ISPs, websites and other organisations to monitor and analyse what individuals do online (e.g. the sites that they visit, things they buy, search terms they use etc) to an extent to which (in Stallman's words) "Stalin could only dream".
- Censorship: for example, governments or corporations blocking access to particular websites (think Google in China), or even forcing them to close.
- Restrictions on users imposed by data formats: both proprietary (e.g. Silverlight) and patented data formats (e.g. MP3) restrict what the end user is able to do with the data they encode.
- Non-free software: here "free" is in the sense of "freedom", rather than price. Non-free software is essentially software that isn't under the control of you, the user - in the case of proprietary software, it's controlled by the owner (for example Microsoft, Apple, Amazon) who is able to insert features (e.g. to track user behaviour) that serves their interests rather than those of the user. By contrast, free software - which by the way you can still charge money for - gives the user four basic freedoms: 0. to run the software for any purpose; 1. to study how the software works, and make changes to it; 2. to redistribute the software as-is; 3. to redistribute the software with your changes (see the free software definition). In this way malicious features can be detected and removed, and control is returned to the user.
- "Software as a service" (SaaS): in Stallman's definition, "software as a service" is anything where the computation is done by programs that you can't control - this is like non-free software above, because someone else has control and can change how your computing is done at any time without your permission. He made a distinction between things like e-commerce, online storage storage (e.g. Dropbox), publishing (e.g. Twitter) and search (which are about "data" or "communication", and so are not SaaS), and e.g. Google Docs (which does do computation for you, and so is SaaS). (See Stallman's article Who does that server really serve?)
- Misuse of an individual's data: essentially doing something with your data without your permission, or even your knowledge - for example, passing on personal data to the authorities, unilaterally modifying your data, or even (for example in the case of Facebook) using it for commercial purposes.
- "The War on Sharing": according to Stallman, sharing is "using the internet for what it's best at", and the war on sharing - whether digital rights management (DRM) technology or threatening internet users with disconnection (as under the UK's Digital Economy Act) - is an attempt by commercial interests to unfairly restrict what users are allowed to do (see Stallman's article Ending the War on Sharing).
- Users don't have a postive right to do things on the internet: essentially, all the activities that users perform on the internet - communications, payment etc - are dependent on organisations who have no obligation to continue providing those services to you.
I'm still mulling over many of the issues raised. On the one hand there is a fundamental question about what moral rights you believe individuals should have, both generally and with specific regard to the digital world; and on the other there is the question of what you should do if you feel those rights are not being upheld. Stallman's position is clear and uncompromising: for example, not owning a mobile phone and not using a key card to enter his office (to avoid the possibility of being tracked), and using a netbook that allows him to run 100% free software (down to the BIOS level). It's certainly given me plenty to think about, and I'm looking forward to reading his book of collected essays "Free Software, Free Society" - which might be a good place to start if you're also interested in learning more.
Sunday, April 3, 2011
Book review: "Python 2.6 Text Processing: Beginner’s Guide" by Jeff McNeil
Jeff McNeil’s “Python 2.6 Text Processing: Beginner’s Guide” is a practical introduction to a wide range of methods for reading, processing and writing textual data from a variety of structured and unstructured data formats. Aimed primarily at novice Python programmers who have some elementary knowledge of the language basics but without prior experience in text processing, the book offers hands-on examples for each of the techniques it discusses – ranging from Python’s built-in libraries for handling strings, regular expressions, and formats such as JSON, XML and HTML, through to more advanced topics such as parsing custom grammars, and efficiently searching large text archives. In addition it contains a great deal of general supporting material on working with Python, including installing packages and third-party libraries, and working with Python 3.
The first three chapters lay the foundations, covering a number of Python basics including a crash course in file and URL I/O, and the essentials of Python’s built-in string handling functions. Useful background topics – such as installing packages with easy_install, and using virtualenv – are also introduced here. (A sample of the first chapter can be freely downloaded from the book’s website at https://www.packtpub.com/python-2-6-text-processing-beginners-guide/book). The next three cover: using the standard library to work with simple structured data formats (delimited “CSV” data, “ini”-style configuration files, and JSON-formatted data); working with Python regular expressions (a stand out chapter for me); and handling structured markup (specifically, XML and HTML). Subsequent chapters on using the Mako templating package (the default system for the Pylons web framework) to generate emails and web pages, and on writing more advanced data formats (PDF, Excel and OpenDocument), are separated by an excellent overview of understanding and working with Unicode, encodings and application internationalization (“i18n”).
The remaining two chapters cover more advanced topics, with some good background theory supplementing the practical examples: using the PyParsing package to create parsers for custom grammars (with a brief nod to the basics of natural language processing using the Natural Language Toolkit, NLTK); and the Nucular package for indexing large quantities of textual data (not necessarily just plain text) to enable highly efficient searching. Finally, an appendix offers a grab-bag of general Python resources, references to some more advanced text processing tools (such as Apache’s Lucene/Solr), and an excellent overview of the differences between Python 2 and 3 (including a hands-on example of migrating code from 2 to 3).
The book covers a lot of ground and moves fairly quickly; however it adopts a largely successful hands-on approach, engaging the reader with working examples at each stage to illustrate the key points, and this certainly helped me keep up. I was also impressed by the clear and concise quality of code in the examples, and the very natural way that general Python concepts and principles – generators, duck typing, packaging and so on – were introduced as asides. (One very minor criticism is that the layout of the example code could have been improved, as the indentation levels weren’t always immediately obvious to me.) Aside from a surprisingly unsatisfying chapter on structured markup (reluctantly, I would recommend looking elsewhere for an introduction to XML processing with Python) and a few niggling typos, there’s a lot of excellent material in this book, and the author has a knack for presenting some tricky concepts in a deceptively easy-to-understand manner. I think that the chapter on regular expressions is possibly one of the best introductions to the subject that I’ve ever seen; other chapters on encodings and internationalization, advanced parsing, and indexing and searching were also highlights for me (as was the section on Python 3 in the appendix).
Overall I really enjoyed working through the book and felt I learned a lot. I think it’s fair to say that given the rather ambitious range of techniques presented, in many cases (particularly for the more advanced or specialised topics) that the chapters are inevitably more introductory than definitive in nature: the reader is given enough information to grasp the background concepts and get started, with pointers to external resources to learn more. In conclusion, I think this is a great introduction to a wide range of text processing techniques in Python, both for novice Pythonistas (who will undoubtedly also benefit from the more general Python tips and tricks presented in the book) and more experienced programmers who are looking for a place to start learning about text processing.
Disclosure: a free e-copy of this book was received from the publisher for review purposes; this review has also been submitted to Amazon.
The first three chapters lay the foundations, covering a number of Python basics including a crash course in file and URL I/O, and the essentials of Python’s built-in string handling functions. Useful background topics – such as installing packages with easy_install, and using virtualenv – are also introduced here. (A sample of the first chapter can be freely downloaded from the book’s website at https://www.packtpub.com/python-2-6-text-processing-beginners-guide/book). The next three cover: using the standard library to work with simple structured data formats (delimited “CSV” data, “ini”-style configuration files, and JSON-formatted data); working with Python regular expressions (a stand out chapter for me); and handling structured markup (specifically, XML and HTML). Subsequent chapters on using the Mako templating package (the default system for the Pylons web framework) to generate emails and web pages, and on writing more advanced data formats (PDF, Excel and OpenDocument), are separated by an excellent overview of understanding and working with Unicode, encodings and application internationalization (“i18n”).
The remaining two chapters cover more advanced topics, with some good background theory supplementing the practical examples: using the PyParsing package to create parsers for custom grammars (with a brief nod to the basics of natural language processing using the Natural Language Toolkit, NLTK); and the Nucular package for indexing large quantities of textual data (not necessarily just plain text) to enable highly efficient searching. Finally, an appendix offers a grab-bag of general Python resources, references to some more advanced text processing tools (such as Apache’s Lucene/Solr), and an excellent overview of the differences between Python 2 and 3 (including a hands-on example of migrating code from 2 to 3).
The book covers a lot of ground and moves fairly quickly; however it adopts a largely successful hands-on approach, engaging the reader with working examples at each stage to illustrate the key points, and this certainly helped me keep up. I was also impressed by the clear and concise quality of code in the examples, and the very natural way that general Python concepts and principles – generators, duck typing, packaging and so on – were introduced as asides. (One very minor criticism is that the layout of the example code could have been improved, as the indentation levels weren’t always immediately obvious to me.) Aside from a surprisingly unsatisfying chapter on structured markup (reluctantly, I would recommend looking elsewhere for an introduction to XML processing with Python) and a few niggling typos, there’s a lot of excellent material in this book, and the author has a knack for presenting some tricky concepts in a deceptively easy-to-understand manner. I think that the chapter on regular expressions is possibly one of the best introductions to the subject that I’ve ever seen; other chapters on encodings and internationalization, advanced parsing, and indexing and searching were also highlights for me (as was the section on Python 3 in the appendix).
Overall I really enjoyed working through the book and felt I learned a lot. I think it’s fair to say that given the rather ambitious range of techniques presented, in many cases (particularly for the more advanced or specialised topics) that the chapters are inevitably more introductory than definitive in nature: the reader is given enough information to grasp the background concepts and get started, with pointers to external resources to learn more. In conclusion, I think this is a great introduction to a wide range of text processing techniques in Python, both for novice Pythonistas (who will undoubtedly also benefit from the more general Python tips and tricks presented in the book) and more experienced programmers who are looking for a place to start learning about text processing.
Disclosure: a free e-copy of this book was received from the publisher for review purposes; this review has also been submitted to Amazon.
Friday, March 18, 2011
Day Camp 4 Developers: Telecommuting
About two weeks ago I took part in the second online Day Camp 4 Developers, on the topic of telecommuting. The idea behind the Day Camp events is to provide software developers with practical knowledge and advice in the area of "soft" skills, to complement their expertise with "hard" skills (i.e. actual coding). In this case five speakers gave consistently excellent web presentations (slides and audio) with different perspectives on remote working, while an IRC chatroom gave all participants a forum to discuss the issues behind the scenes.
Lorna Jane Mitchell started off by asking "Could You Telecommute?". As a teleworker herself, Lorna Jane looked at the environmental, organisational and personal factors that influence the happiness and productivity of the remote worker: for example, ensuring you have a good home working space, and set clear boundaries between work and personal life (both for yourself and for others). In particular you have to be aware of the tendency for other people to think that working from home is easy, and that your time is infinitely flexible. She also noted that there are some big differences between being part of a distributed team and being a telecommuting member of a co-located team (where you risk feeling isolated), and further differences between employees and freelancers. Particularly for lone telecommuters, it's important to build professional and social support networks that might otherwise be taken for granted in more conventional work settings.
Next self-described "entreprenerd" Ivo Jansch talked about "The Business Case For Telecommuting". Ivo's company Egeniq is built around a distributed team (essentially using remote working as an organisational model) - so in addition to benefiting individual workers, he suggested ways that telecommuting could positively impact the company's bottom line, for example enabling access to an bigger talent pool and increasing its geographical reach (if providing consultancy services). He acknowledged that this distributed model won't suit every company or industry however, and success requires (amongst other things) a results-driven culture where individuals are trusted to self-manage and have a sense of shared responsibility. Ultimately good communication between team members is paramount.
After the lunch break, Jack G. Ford gave a manager's perspective on setting up a telecommuting programme in his presentation "Can I Work From Home Tomorrow?". Jack introduced himself as an ex-coder who is now the manager for 17 developers in a more conventional environment than Ivo's, but in spite of that his key points seemed remarkably similar: beyond asking whether the company infrastructure can support remote working, the main issues are trust (both with the manager and with the team) and good communication between the manager and the individual. Jack emphasised that as a manager, when you telecommute, "I can't see you," so the telecommuter must stay connected, keep the manager informed, and must not only act professionally but be seen to do so. Although it might seem obvious, this was a fascinating insight into telecommuting from the other side of the management chain.
Ligaya Turmelle's presentation on "Managing the Work/Life Balance" emphasised the challenges of balancing work and home life, with her lists of "the good, the bad and the ugly" of remote working from a teleworker perspective. Ligaya focused especially on balancing family commitments with work commitments, and among some interesting observations (for example, no longer doing the daily commute means you lose some "me time" to yourself), I was most struck by the admission that if you love your work then it can mean sometimes that you want to go on working, and are in danger of not respecting your own ground rules. While noting that situations can differ both for individuals and companies, her advice was: clarify everyone's expectations (e.g. policies for "on-call" hours, weekends, and holidays); set up ground rules and limits (and be disciplined in adhering to them); and try to be flexible and imaginative in how you approach your work.
The final presentation was Avdi Grimm talking about "The Well-Equipped Remote Worker". Avdi is a freelance software developer who is also a "dispersed teams facilitator" and runs the Wide Teams blog. As might be expected from the title, some of the focus was on the hardware and software tools that can help with remote working, but there was just as much information on practices that can support distributed teams. Once again promoting communication is key, and using tools and practices that help team members create good working relationships (for example, utilising social media like Twitter and Facebook, and holding regular face-to-face meetings) can really contribute to this.
Looking back over all the talks, a few common themes had emerged for me:
Overall it was an excellent event and a good use of 8 hours of my Saturday - although the time difference (coincidentally another telecommuting issue) meant that it didn't finish until 10pm UK time I surprised myself by staying with it to the end. Hats off to Cal and Kathy Evans for organising the day and to the speakers for their excellent presentations. Here's waiting for the next Day Camp 4 Developers!
Lorna Jane Mitchell started off by asking "Could You Telecommute?". As a teleworker herself, Lorna Jane looked at the environmental, organisational and personal factors that influence the happiness and productivity of the remote worker: for example, ensuring you have a good home working space, and set clear boundaries between work and personal life (both for yourself and for others). In particular you have to be aware of the tendency for other people to think that working from home is easy, and that your time is infinitely flexible. She also noted that there are some big differences between being part of a distributed team and being a telecommuting member of a co-located team (where you risk feeling isolated), and further differences between employees and freelancers. Particularly for lone telecommuters, it's important to build professional and social support networks that might otherwise be taken for granted in more conventional work settings.
Next self-described "entreprenerd" Ivo Jansch talked about "The Business Case For Telecommuting". Ivo's company Egeniq is built around a distributed team (essentially using remote working as an organisational model) - so in addition to benefiting individual workers, he suggested ways that telecommuting could positively impact the company's bottom line, for example enabling access to an bigger talent pool and increasing its geographical reach (if providing consultancy services). He acknowledged that this distributed model won't suit every company or industry however, and success requires (amongst other things) a results-driven culture where individuals are trusted to self-manage and have a sense of shared responsibility. Ultimately good communication between team members is paramount.
After the lunch break, Jack G. Ford gave a manager's perspective on setting up a telecommuting programme in his presentation "Can I Work From Home Tomorrow?". Jack introduced himself as an ex-coder who is now the manager for 17 developers in a more conventional environment than Ivo's, but in spite of that his key points seemed remarkably similar: beyond asking whether the company infrastructure can support remote working, the main issues are trust (both with the manager and with the team) and good communication between the manager and the individual. Jack emphasised that as a manager, when you telecommute, "I can't see you," so the telecommuter must stay connected, keep the manager informed, and must not only act professionally but be seen to do so. Although it might seem obvious, this was a fascinating insight into telecommuting from the other side of the management chain.
Ligaya Turmelle's presentation on "Managing the Work/Life Balance" emphasised the challenges of balancing work and home life, with her lists of "the good, the bad and the ugly" of remote working from a teleworker perspective. Ligaya focused especially on balancing family commitments with work commitments, and among some interesting observations (for example, no longer doing the daily commute means you lose some "me time" to yourself), I was most struck by the admission that if you love your work then it can mean sometimes that you want to go on working, and are in danger of not respecting your own ground rules. While noting that situations can differ both for individuals and companies, her advice was: clarify everyone's expectations (e.g. policies for "on-call" hours, weekends, and holidays); set up ground rules and limits (and be disciplined in adhering to them); and try to be flexible and imaginative in how you approach your work.
The final presentation was Avdi Grimm talking about "The Well-Equipped Remote Worker". Avdi is a freelance software developer who is also a "dispersed teams facilitator" and runs the Wide Teams blog. As might be expected from the title, some of the focus was on the hardware and software tools that can help with remote working, but there was just as much information on practices that can support distributed teams. Once again promoting communication is key, and using tools and practices that help team members create good working relationships (for example, utilising social media like Twitter and Facebook, and holding regular face-to-face meetings) can really contribute to this.
Looking back over all the talks, a few common themes had emerged for me:
- Good communication (both with managers and with other team members) to build trust, keep people informed and avoid misunderstandings;
- Clarify expectations on all sides, and establishing well-defined boundaries between work and personal life. Set ground rules to ensure that those boundaries are respected by others (your boss, your family and friends) and have the discipline to also respect them yourself;
- Build and maintain your social and professional support networks for when there are problem times;
- Provide yourself with a good working environment and (software and hardware) tools.
Overall it was an excellent event and a good use of 8 hours of my Saturday - although the time difference (coincidentally another telecommuting issue) meant that it didn't finish until 10pm UK time I surprised myself by staying with it to the end. Hats off to Cal and Kathy Evans for organising the day and to the speakers for their excellent presentations. Here's waiting for the next Day Camp 4 Developers!
Subscribe to:
Posts (Atom)