"Instant Flask Web Development" by Ron DuPlain (Packt Publishing http://www.packtpub.com/flask-web-development/book) is intended to be an
introduction to Flask, a lightweight web application framework written
in Python and based on the Werkzeug WSGI toolkit and Jinja2 template
engine.
The book takes a tutorial style approach, building
up an example appointment-management web application using Flask and
introducing various features of the framework on the way. As the example
application becomes more complicated, additional Python packages are
covered which are not part of the Flask framework (for example SQLAlchemy for managing interactions with a
database backend, and WTForm for handling form generation and
validation) along with various Flask extensions that can be used for
more complicated tasks (for example managing user logins and sessions). The final section of the book gives an overview of how to deploy the
application in a production environment, using gunicorn (a Python WSGI server) and nginx.
Given
its length (just short of 70 pages) the book is quite ambitious in the
amount of ground that it attempts to cover, and it's quite impressive
how much the author has managed to pack in whilst maintaining a light
touch with the material. So while inevitably there is a limit to the level of detail that
can be fitted in, there are some excellent and concise overviews of
many of the topics that could act as excellent starting points for more
experienced developers (for me the section on SQLAlchemy is a particular
highlight). Overall the pacing of the book is also quite sprightly and
conveys a sense of how quickly and easily Flask could be used to build a
web application from scratch.
The flipside of the
book's brevity is that it cannot possibly contain everything that a
developer needs to know (although this is mitigated to some extent by
extensive references to online documentation and resources). In this
regard it is really more a showcase for Flask, and is best viewed as a
good starting point for someone wishing to quickly get up to speed with the framework's potential. I'd also question how suitable this is for
newcomers to either Python, or to web programming in general - I felt that some of the concepts and example code (for example the sudden appearance of a feature implemented using Ajax) might be a bit of a stretch for a novice. Also there are some occasional frustrating glitches in the text and example code which meant
it took a bit of additional reading and debugging in places to get the
example application working in practice.
In summary
then: I'd recommend this book as a good starting point for developers
who already have some familiarity with web application development, and
who are interested in a quick introduction to the key components of Flask and how they're used - with the
caveat that you will most likely have to refer to other resources to get
the most out of it.
Disclosure: a free e-copy of this book
was received from the publisher for review purposes; this review has
also been submitted to Amazon. The opinions expressed here are entirely my own.
Showing posts with label python. Show all posts
Showing posts with label python. Show all posts
Monday, December 9, 2013
Friday, April 19, 2013
Software Carpentry Bootcamp Manchester
I've spent the last two days as a helper at a Software Carpentry Bootcamp held at the University of Manchester, and it's been really interesting and fun. Software Carpentry is a volunteer organisation and runs the bootcamps with the aim of helping postgraduate students and scientists become more productive
by teaching them basic computing skills like program design, version
control, testing, and task automation. Many of the materials are freely available online via the bootcamp's Github page: along with transcipts of some of the tutorials there are some excellent supporting materials including hints and tips on common Bash and editor commands (there's even more on the main Software Carpentry website).
The bootcamp format consisted of short tutorials alternating with hands-on practical exercises, and as a helper the main task was to support the instructors by offering assistance to participants if they found themselves stuck for some reason in the exercises. I'll admit I felt some trepidation beforehand about being a helper, as being put on the spot to debug something is very different to doing it from the relaxed privacy of my desk. However it turned out to be a both very enjoyable and very educational experience; even though I consider myself to be quiet a proficient and experienced shell and Python programmer, I learned some new things from helping the participants both with understanding some of the concepts and with getting their examples to work.
There were certainly lots of fresh insights and I learned some new things from the taught sessions too, including:
If any of this sounds interesting to you then the Software Carpentry bootcamp calendar shows future events planned in both Europe and the US, so it's worth a look to see if there's one coming up near your location. Otherwise you could consider hosting or running your own bootcamp. Either way I'd very much recommend taking part to any researchers who want to make a positive impact on their work with software.
The bootcamp format consisted of short tutorials alternating with hands-on practical exercises, and as a helper the main task was to support the instructors by offering assistance to participants if they found themselves stuck for some reason in the exercises. I'll admit I felt some trepidation beforehand about being a helper, as being put on the spot to debug something is very different to doing it from the relaxed privacy of my desk. However it turned out to be a both very enjoyable and very educational experience; even though I consider myself to be quiet a proficient and experienced shell and Python programmer, I learned some new things from helping the participants both with understanding some of the concepts and with getting their examples to work.
There were certainly lots of fresh insights and I learned some new things from the taught sessions too, including:
- Bash/shell scripting: using $(...) instead of "backtick" notation to execute a command or pipeline within a shell script;
- Version control: learning that Bitbucket now offers free private repositories (and a reminder that git push doesn't automatically push tags to the origin, for that you also need to explicitly use git push --tags);
- Python: a reminder that slice notation [i:j] is inclusive of the first index i but exclusive of the second index j, and independently that string methods often don't play well with Unicode;
- Testing: a reminder that writing and running tests doesn't have to impose a big overhead - good test functions can be implemented just with assert statements, and by observing a simple naming convention (i.e. put tests in a test_<module>.py file, and name test functions test_<name>), Python nose can run them automatically without any additional infrastructure.
- Make: good to finally have an introduction to the basic mechanics of Makefiles (including targets, dependencies, automatic variables, wildcards and macros), after all these years!
If any of this sounds interesting to you then the Software Carpentry bootcamp calendar shows future events planned in both Europe and the US, so it's worth a look to see if there's one coming up near your location. Otherwise you could consider hosting or running your own bootcamp. Either way I'd very much recommend taking part to any researchers who want to make a positive impact on their work with software.
Sunday, August 19, 2012
Inline images in HTML ![]()
tags with Python
I recently discovered a neat trick for embedding images within HTML documents, really useful if you've got an application where you would like the HTML files to be portable (in the sense of being moved from one location to another) and not have to rely on also moving a bunch of related image files.
Essentially the inlining is achieved by base64 encoding the data from the image file into an ASCII string, which can then be copied into the src attribute of an <img> tag with the following general syntax:
i.e. image_type is png and iVBORw... is the base64 encoded string (truncated here for readability).
If you're familiar with Python then it's straightforward to encode any file using the base64 module, e.g. (for a PNG image):
In fact this inlining is an example of the general data URI scheme for embedding data directly in HTML documents, and can also be used for example in CSS to set an inline background image - the Wikipedia entry on the "Data URI scheme" is a good place to start for more detailed information.
There are some caveats, particularly if you're interested in cross-browser compatibility: older versions of Internet Explorer (version 7 and older) don't support data URIs at all, while version 8 only supports encoded strings up to 32KB. More generally the encoded strings can be around 1/3 larger than the original images and are implicitly downloaded each time the document is refreshed; these are definitely considerations if you're concerned about bandwidth. However if these aren't issues for your application then this can be a handy trick to have in your toolbox.
Update 24/10/2012: another caveat I've discovered since is that at least some command line HTML-to-PDF converters (for example wkhtmltopdf) aren't able to convert the encoded images, so it's worth bearing this in mind if you plan to use them. (On the other hand the PDF conversion in Firefox - via "Print to file" - works fine but can't be run from the command line AFAIK.)
Essentially the inlining is achieved by base64 encoding the data from the image file into an ASCII string, which can then be copied into the src attribute of an <img> tag with the following general syntax:
<img src="data:image/image_type;base64,base64_encoded_string" />
For example to embed a PNG image:
<img src="data:image/png;base64,iVBORw...." />
i.e. image_type is png and iVBORw... is the base64 encoded string (truncated here for readability).
If you're familiar with Python then it's straightforward to encode any file using the base64 module, e.g. (for a PNG image):
>>> import base64
>>> pngdata = base64.b64encode(open("cog.png",'rb').read())
>>> print "<img src='data:image/png;base64,%s' />" % pngdata
And here's an example inlined PNG generated using this method:
In fact this inlining is an example of the general data URI scheme for embedding data directly in HTML documents, and can also be used for example in CSS to set an inline background image - the Wikipedia entry on the "Data URI scheme" is a good place to start for more detailed information.
There are some caveats, particularly if you're interested in cross-browser compatibility: older versions of Internet Explorer (version 7 and older) don't support data URIs at all, while version 8 only supports encoded strings up to 32KB. More generally the encoded strings can be around 1/3 larger than the original images and are implicitly downloaded each time the document is refreshed; these are definitely considerations if you're concerned about bandwidth. However if these aren't issues for your application then this can be a handy trick to have in your toolbox.
Update 24/10/2012: another caveat I've discovered since is that at least some command line HTML-to-PDF converters (for example wkhtmltopdf) aren't able to convert the encoded images, so it's worth bearing this in mind if you plan to use them. (On the other hand the PDF conversion in Firefox - via "Print to file" - works fine but can't be run from the command line AFAIK.)
Wednesday, July 27, 2011
Book review: “Python Testing Cookbook” by Greg L. Turnquist
Disclosure: a free e-copy of this book was received from the publisher for review purposes. The opinions expressed here are entirely my own; a copy of this review has also been posted at Amazon.
Greg L. Turnquist’s “Python Testing Cookbook” explores automated testing at all levels, with the intention of providing the reader with the knowledge needed to implement testing using Python tools to improve software quality. To this end the book presents over 70 “recipes” in its nine chapters (ranging from the basics of unit testing, through test suites, user acceptance and web application testing, continuous integration, and methods for smoke- and load-testing), covering both tools for testing Python, and Python tools for testing. It also delivers advice about how to get the most from automated testing, which is as much an art as a science.
The first three chapters introduce the fundamentals: writing, organising and running unit tests, comprehensively covering unittest (Python’s built-in unit testing library), nose (a versatile tool for discovering, running and reporting tests) and doctest (which turns Python docstrings into testable code – a sample of this chapter can be downloaded from http://www.packtpub.com/python-testing-cookbook/book). Having established a solid foundation, subsequent chapters look at increasingly broader levels of automated testing using the appropriate relevant Python tools: for example, the “lettuce” and “should_DSL” libraries for “behaviour driven development” (an extension of “test driven development” which aims to produce human-readable test cases and reports), and the “Pyccuracy” and “Robot” frameworks for end-user acceptance testing of web applications. Later chapters cover higher level concepts and tools, such as using nose to hook Python tests into “continuous integration” servers (both Jenkins and TeamCity are covered in detail), and assessing test coverage using the “coverage” tool (both as a metric, and to identify areas that need more tests). A detailed chapter on smoke- and load-testing includes practical advice on developing multiple test suites for different scenarios, and methods for stress-testing (for example, by capturing and replaying real world data) to discover weaknesses in a system before going to production. The final chapter distils the author’s experience into general advice on making testing a successful part of your code development methodology, both for new and legacy projects.
There’s a lot of good stuff in this book: the initial chapters on unittest and nose are particularly strong, and I can imagine returning to these in future as a reference. There is also a lot of excellent and hard-won practical advice from the author’s own experience – not only in these early chapters but throughout the book – which is consistently valuable (in this regard the final chapter is a real highlight and could easily stand alone – I will definitely be re-reading it soon). Elsewhere the various tools and topics are presented clearly with plenty of useful detail, and in some cases have demystified things that I’d always assumed were quite esoteric and difficult to do (nose in particular was a revelation to me, but also setting up continuous integration servers and measuring test coverage).
There are a few disappointments: the section on mock objects left me feeling baffled as to how to actually implement them in practice – a shame as it was something that I’d looked forward to learning. I’d also have liked something about approaches for handling difficult testing scenarios such as software which interacts with the file system or with large files – a few hints here would have been invaluable for me. There are typos in some commands and code in a few recipes (e.g. for nose), which meant I had to look up the correct syntax elsewhere – perhaps not so bad, but annoying (especially in a cookbook) – and since the recipes themselves aren’t numbered, this sometimes made it difficult to navigate between them.
However these are fairly minor quibbles, and in conclusion I was impressed with both the breadth of material covered by the book and the level of detail for many topics. Moreover I enjoyed reading it and was often left feeling excited at the prospect of being able to apply the ideas to my own projects, which is I think was one of the author’s aims (and no mean feat for a technical book). I think that the combination of the detail together with the author’s practical advice make this book both an excellent introduction to testing with Python, and a valuable resource to refer back to subsequently.
(Addendum: Greg Turnquist's blog about the book can be found at http://pythontestingcookbook.posterous.com/ and features some interesting supplementary material.)
Greg L. Turnquist’s “Python Testing Cookbook” explores automated testing at all levels, with the intention of providing the reader with the knowledge needed to implement testing using Python tools to improve software quality. To this end the book presents over 70 “recipes” in its nine chapters (ranging from the basics of unit testing, through test suites, user acceptance and web application testing, continuous integration, and methods for smoke- and load-testing), covering both tools for testing Python, and Python tools for testing. It also delivers advice about how to get the most from automated testing, which is as much an art as a science.
The first three chapters introduce the fundamentals: writing, organising and running unit tests, comprehensively covering unittest (Python’s built-in unit testing library), nose (a versatile tool for discovering, running and reporting tests) and doctest (which turns Python docstrings into testable code – a sample of this chapter can be downloaded from http://www.packtpub.com/python-testing-cookbook/book). Having established a solid foundation, subsequent chapters look at increasingly broader levels of automated testing using the appropriate relevant Python tools: for example, the “lettuce” and “should_DSL” libraries for “behaviour driven development” (an extension of “test driven development” which aims to produce human-readable test cases and reports), and the “Pyccuracy” and “Robot” frameworks for end-user acceptance testing of web applications. Later chapters cover higher level concepts and tools, such as using nose to hook Python tests into “continuous integration” servers (both Jenkins and TeamCity are covered in detail), and assessing test coverage using the “coverage” tool (both as a metric, and to identify areas that need more tests). A detailed chapter on smoke- and load-testing includes practical advice on developing multiple test suites for different scenarios, and methods for stress-testing (for example, by capturing and replaying real world data) to discover weaknesses in a system before going to production. The final chapter distils the author’s experience into general advice on making testing a successful part of your code development methodology, both for new and legacy projects.
There’s a lot of good stuff in this book: the initial chapters on unittest and nose are particularly strong, and I can imagine returning to these in future as a reference. There is also a lot of excellent and hard-won practical advice from the author’s own experience – not only in these early chapters but throughout the book – which is consistently valuable (in this regard the final chapter is a real highlight and could easily stand alone – I will definitely be re-reading it soon). Elsewhere the various tools and topics are presented clearly with plenty of useful detail, and in some cases have demystified things that I’d always assumed were quite esoteric and difficult to do (nose in particular was a revelation to me, but also setting up continuous integration servers and measuring test coverage).
There are a few disappointments: the section on mock objects left me feeling baffled as to how to actually implement them in practice – a shame as it was something that I’d looked forward to learning. I’d also have liked something about approaches for handling difficult testing scenarios such as software which interacts with the file system or with large files – a few hints here would have been invaluable for me. There are typos in some commands and code in a few recipes (e.g. for nose), which meant I had to look up the correct syntax elsewhere – perhaps not so bad, but annoying (especially in a cookbook) – and since the recipes themselves aren’t numbered, this sometimes made it difficult to navigate between them.
However these are fairly minor quibbles, and in conclusion I was impressed with both the breadth of material covered by the book and the level of detail for many topics. Moreover I enjoyed reading it and was often left feeling excited at the prospect of being able to apply the ideas to my own projects, which is I think was one of the author’s aims (and no mean feat for a technical book). I think that the combination of the detail together with the author’s practical advice make this book both an excellent introduction to testing with Python, and a valuable resource to refer back to subsequently.
(Addendum: Greg Turnquist's blog about the book can be found at http://pythontestingcookbook.posterous.com/ and features some interesting supplementary material.)
Saturday, April 9, 2011
Managing Python packages: virtualenv, pip and yolk
I've recently been playing with the Python virtualenv package - along with pip and yolk - as a way of managing third-party packages. This post is my brief introduction to the basics of these three tools.
virtualenv lets you create isolated self-contained "virtual environments" which are separate from the system Python. You can then install and manage the specific Python packages that you need for a particular application - safe from potential problems due to version incompatibilities, and without needing superuser privileges - using the pip package installer. yolk provides an extra utility to keep track of what's installed.
1. virtualenv: building virtual Python environments
virtualenv can either be installed via your system's package manager (for example, synaptic on Ubuntu), or by using the easy_install tool, i.e.:
$ easy_install virtualenv
(If you don't have the SetupTools package which provides easy_install then you can download the "bootstrap" install script from http://peak.telecommunity.com/dist/ez_setup.py. Save as ez_setup.py and run using /path/to/python ez_setup.py.)
Once virtualenv is installed you can create a new virtual environment (called in this example, "myenv") as follows:
$ virtualenv --no-site-packages myenv
This makes a new directory myenv in the current directory (which will contain bin, include and lib subdirectories) based on the system version of Python. The --no-site-packages option tells virtualenv not to include any third-party packages which might have been installed into the system Python (see the virtualenv documentation for details of other options).
To start using the new environment, run the environment's "activate" command e.g.:
$ source myenv/bin/activate
The shell command prompt will change from e.g. $ to (myenv)$, indicating that the "myenv" environment (and any packages installed in it) will be used instead of the system Python for applications run in this shell. (Note that the Python application code doesn't need to be inside the virtual environment directory; in fact this directory is just using for the packages associated with the virtual environment.)
Finally, when you've finished working with the virtual environment you can leave it by running the deactivate command (also in the bin directory).
(On Windows you may have to specify the full path to the "Scripts" directory of your Python installation when invoking the easy_install and virtualenv commands above, e.g. C:\Python27\Scripts\virtualenv. Also, note that when a virtual environment is created it won't contain a "bin" directory - instead it's activated by invoking the Scripts\activate batch file in the virtual environment directory. Invoking the deactivate command exits the environment as before.)
2. pip: installing Python packages
Once you're created a virtual environment you can start to add packages (which is really the point of doing this in the first place). virtualenv automatically includes both easy_install and an alternative package installer called pip (at least, for virtualenv 1.4.1 and up; earlier versions only have easy_install, so you'll need to run easy_install pip within the virtual environment in order to get it).
Most packages that are easy_installable can also be installed using pip, and it's designed to work well with virtualenv. However I think its main advantage is that it offers some useful functionality that's missing from easy_install - most significantly, the ability to uninstall previously installed packages. (Other useful features include the ability to explicitly control and export versions of third-party package dependencies via "requirements files" - see the pip documentation for more details.)
Basic pip usage looks like this:
(myenv)$ pip install python-dateutil # install latest version of a package
(myenv)$ pip uninstall python-dateutil # remove package
(myenv)$ pip install python-dateutil==1.5 # install specific version
(As an aside, the python-dateutil package is illustrative of one of the advantages of using pip over easy_install: after installing the latest version of python-dateutil, I discovered that it's only compatible with Python 3 - an earlier 1.* version is required to work with Python 2. pip let me uninstall the newer version and reinstall the older one.)
3. yolk: checking Python packages installed on your system
The final utility I'd recommend is yolk, which provides a way of querying which packages (and versions) have been installed in the current environment. It also has options to query PyPI (the Python Package Index). Installing it is easy:
(myenv)$ pip install yolk
Running it with the -l option (for "list") then shows us what packages are available:
Summary
Obviously the above is just an introduction to the basics of virtualenv, pip and yolk for managing and working with third-party packages - but hopefully it's enough to get started. If you're interested in using virtualenv in practice then Doug Hellman's article about working with multiple virtual environments (and his virtualenvwrapper project, which provides tools to help) is recommended as a starting point for further reading.
virtualenv lets you create isolated self-contained "virtual environments" which are separate from the system Python. You can then install and manage the specific Python packages that you need for a particular application - safe from potential problems due to version incompatibilities, and without needing superuser privileges - using the pip package installer. yolk provides an extra utility to keep track of what's installed.
1. virtualenv: building virtual Python environments
virtualenv can either be installed via your system's package manager (for example, synaptic on Ubuntu), or by using the easy_install tool, i.e.:
$ easy_install virtualenv
(If you don't have the SetupTools package which provides easy_install then you can download the "bootstrap" install script from http://peak.telecommunity.com/dist/ez_setup.py. Save as ez_setup.py and run using /path/to/python ez_setup.py.)
Once virtualenv is installed you can create a new virtual environment (called in this example, "myenv") as follows:
$ virtualenv --no-site-packages myenv
This makes a new directory myenv in the current directory (which will contain bin, include and lib subdirectories) based on the system version of Python. The --no-site-packages option tells virtualenv not to include any third-party packages which might have been installed into the system Python (see the virtualenv documentation for details of other options).
To start using the new environment, run the environment's "activate" command e.g.:
$ source myenv/bin/activate
The shell command prompt will change from e.g. $ to (myenv)$, indicating that the "myenv" environment (and any packages installed in it) will be used instead of the system Python for applications run in this shell. (Note that the Python application code doesn't need to be inside the virtual environment directory; in fact this directory is just using for the packages associated with the virtual environment.)
Finally, when you've finished working with the virtual environment you can leave it by running the deactivate command (also in the bin directory).
(On Windows you may have to specify the full path to the "Scripts" directory of your Python installation when invoking the easy_install and virtualenv commands above, e.g. C:\Python27\Scripts\virtualenv. Also, note that when a virtual environment is created it won't contain a "bin" directory - instead it's activated by invoking the Scripts\activate batch file in the virtual environment directory. Invoking the deactivate command exits the environment as before.)
2. pip: installing Python packages
Once you're created a virtual environment you can start to add packages (which is really the point of doing this in the first place). virtualenv automatically includes both easy_install and an alternative package installer called pip (at least, for virtualenv 1.4.1 and up; earlier versions only have easy_install, so you'll need to run easy_install pip within the virtual environment in order to get it).
Most packages that are easy_installable can also be installed using pip, and it's designed to work well with virtualenv. However I think its main advantage is that it offers some useful functionality that's missing from easy_install - most significantly, the ability to uninstall previously installed packages. (Other useful features include the ability to explicitly control and export versions of third-party package dependencies via "requirements files" - see the pip documentation for more details.)
Basic pip usage looks like this:
(myenv)$ pip install python-dateutil # install latest version of a package
(myenv)$ pip uninstall python-dateutil # remove package
(myenv)$ pip install python-dateutil==1.5 # install specific version
(As an aside, the python-dateutil package is illustrative of one of the advantages of using pip over easy_install: after installing the latest version of python-dateutil, I discovered that it's only compatible with Python 3 - an earlier 1.* version is required to work with Python 2. pip let me uninstall the newer version and reinstall the older one.)
3. yolk: checking Python packages installed on your system
The final utility I'd recommend is yolk, which provides a way of querying which packages (and versions) have been installed in the current environment. It also has options to query PyPI (the Python Package Index). Installing it is easy:
(myenv)$ pip install yolk
Running it with the -l option (for "list") then shows us what packages are available:
(myenv)$ yolk -l
Python - 2.6.4 - active development (/usr/lib/python2.6/lib-dynload)
pip - 1.0 - active
python-dateutil - 1.5 - active
setuptools - 0.6c9 - active
wsgiref - 0.1.2 - active development (/usr/lib/python2.6)
yolk - 0.4.1 - active
(See the yolk documentation to learn more about its other features.)Summary
Obviously the above is just an introduction to the basics of virtualenv, pip and yolk for managing and working with third-party packages - but hopefully it's enough to get started. If you're interested in using virtualenv in practice then Doug Hellman's article about working with multiple virtual environments (and his virtualenvwrapper project, which provides tools to help) is recommended as a starting point for further reading.
Sunday, April 3, 2011
Book review: "Python 2.6 Text Processing: Beginner’s Guide" by Jeff McNeil
Jeff McNeil’s “Python 2.6 Text Processing: Beginner’s Guide” is a practical introduction to a wide range of methods for reading, processing and writing textual data from a variety of structured and unstructured data formats. Aimed primarily at novice Python programmers who have some elementary knowledge of the language basics but without prior experience in text processing, the book offers hands-on examples for each of the techniques it discusses – ranging from Python’s built-in libraries for handling strings, regular expressions, and formats such as JSON, XML and HTML, through to more advanced topics such as parsing custom grammars, and efficiently searching large text archives. In addition it contains a great deal of general supporting material on working with Python, including installing packages and third-party libraries, and working with Python 3.
The first three chapters lay the foundations, covering a number of Python basics including a crash course in file and URL I/O, and the essentials of Python’s built-in string handling functions. Useful background topics – such as installing packages with easy_install, and using virtualenv – are also introduced here. (A sample of the first chapter can be freely downloaded from the book’s website at https://www.packtpub.com/python-2-6-text-processing-beginners-guide/book). The next three cover: using the standard library to work with simple structured data formats (delimited “CSV” data, “ini”-style configuration files, and JSON-formatted data); working with Python regular expressions (a stand out chapter for me); and handling structured markup (specifically, XML and HTML). Subsequent chapters on using the Mako templating package (the default system for the Pylons web framework) to generate emails and web pages, and on writing more advanced data formats (PDF, Excel and OpenDocument), are separated by an excellent overview of understanding and working with Unicode, encodings and application internationalization (“i18n”).
The remaining two chapters cover more advanced topics, with some good background theory supplementing the practical examples: using the PyParsing package to create parsers for custom grammars (with a brief nod to the basics of natural language processing using the Natural Language Toolkit, NLTK); and the Nucular package for indexing large quantities of textual data (not necessarily just plain text) to enable highly efficient searching. Finally, an appendix offers a grab-bag of general Python resources, references to some more advanced text processing tools (such as Apache’s Lucene/Solr), and an excellent overview of the differences between Python 2 and 3 (including a hands-on example of migrating code from 2 to 3).
The book covers a lot of ground and moves fairly quickly; however it adopts a largely successful hands-on approach, engaging the reader with working examples at each stage to illustrate the key points, and this certainly helped me keep up. I was also impressed by the clear and concise quality of code in the examples, and the very natural way that general Python concepts and principles – generators, duck typing, packaging and so on – were introduced as asides. (One very minor criticism is that the layout of the example code could have been improved, as the indentation levels weren’t always immediately obvious to me.) Aside from a surprisingly unsatisfying chapter on structured markup (reluctantly, I would recommend looking elsewhere for an introduction to XML processing with Python) and a few niggling typos, there’s a lot of excellent material in this book, and the author has a knack for presenting some tricky concepts in a deceptively easy-to-understand manner. I think that the chapter on regular expressions is possibly one of the best introductions to the subject that I’ve ever seen; other chapters on encodings and internationalization, advanced parsing, and indexing and searching were also highlights for me (as was the section on Python 3 in the appendix).
Overall I really enjoyed working through the book and felt I learned a lot. I think it’s fair to say that given the rather ambitious range of techniques presented, in many cases (particularly for the more advanced or specialised topics) that the chapters are inevitably more introductory than definitive in nature: the reader is given enough information to grasp the background concepts and get started, with pointers to external resources to learn more. In conclusion, I think this is a great introduction to a wide range of text processing techniques in Python, both for novice Pythonistas (who will undoubtedly also benefit from the more general Python tips and tricks presented in the book) and more experienced programmers who are looking for a place to start learning about text processing.
Disclosure: a free e-copy of this book was received from the publisher for review purposes; this review has also been submitted to Amazon.
The first three chapters lay the foundations, covering a number of Python basics including a crash course in file and URL I/O, and the essentials of Python’s built-in string handling functions. Useful background topics – such as installing packages with easy_install, and using virtualenv – are also introduced here. (A sample of the first chapter can be freely downloaded from the book’s website at https://www.packtpub.com/python-2-6-text-processing-beginners-guide/book). The next three cover: using the standard library to work with simple structured data formats (delimited “CSV” data, “ini”-style configuration files, and JSON-formatted data); working with Python regular expressions (a stand out chapter for me); and handling structured markup (specifically, XML and HTML). Subsequent chapters on using the Mako templating package (the default system for the Pylons web framework) to generate emails and web pages, and on writing more advanced data formats (PDF, Excel and OpenDocument), are separated by an excellent overview of understanding and working with Unicode, encodings and application internationalization (“i18n”).
The remaining two chapters cover more advanced topics, with some good background theory supplementing the practical examples: using the PyParsing package to create parsers for custom grammars (with a brief nod to the basics of natural language processing using the Natural Language Toolkit, NLTK); and the Nucular package for indexing large quantities of textual data (not necessarily just plain text) to enable highly efficient searching. Finally, an appendix offers a grab-bag of general Python resources, references to some more advanced text processing tools (such as Apache’s Lucene/Solr), and an excellent overview of the differences between Python 2 and 3 (including a hands-on example of migrating code from 2 to 3).
The book covers a lot of ground and moves fairly quickly; however it adopts a largely successful hands-on approach, engaging the reader with working examples at each stage to illustrate the key points, and this certainly helped me keep up. I was also impressed by the clear and concise quality of code in the examples, and the very natural way that general Python concepts and principles – generators, duck typing, packaging and so on – were introduced as asides. (One very minor criticism is that the layout of the example code could have been improved, as the indentation levels weren’t always immediately obvious to me.) Aside from a surprisingly unsatisfying chapter on structured markup (reluctantly, I would recommend looking elsewhere for an introduction to XML processing with Python) and a few niggling typos, there’s a lot of excellent material in this book, and the author has a knack for presenting some tricky concepts in a deceptively easy-to-understand manner. I think that the chapter on regular expressions is possibly one of the best introductions to the subject that I’ve ever seen; other chapters on encodings and internationalization, advanced parsing, and indexing and searching were also highlights for me (as was the section on Python 3 in the appendix).
Overall I really enjoyed working through the book and felt I learned a lot. I think it’s fair to say that given the rather ambitious range of techniques presented, in many cases (particularly for the more advanced or specialised topics) that the chapters are inevitably more introductory than definitive in nature: the reader is given enough information to grasp the background concepts and get started, with pointers to external resources to learn more. In conclusion, I think this is a great introduction to a wide range of text processing techniques in Python, both for novice Pythonistas (who will undoubtedly also benefit from the more general Python tips and tricks presented in the book) and more experienced programmers who are looking for a place to start learning about text processing.
Disclosure: a free e-copy of this book was received from the publisher for review purposes; this review has also been submitted to Amazon.
Sunday, January 23, 2011
Python North-West: The Python Challenge
Last week I went to my first-ever Python North-West meeting, at the Manchester Digital Laboratory (aka MadLab). The webpage describes it as a "user group for Pythoneers and Pythonistas of all levels and ages, open to everyone coding 'the way Guido indented it'", and meetings alternate between talks and coding dojos (group coding sessions where people get to share code and ideas with the aim of improving their knowledge and skills - see http://codingdojo.org/cgi-bin/wiki.pl?CodingDojo for more information).
This particular meeting was a coding dojo and so as a group we worked through The Python Challenge (http://www.pythonchallenge.com/), which is a series of puzzles that can be solved using Python programming combined with some imagination and lateral thinking. While most people had come with their own laptops, the format that developed was for one person to "drive" the laptop connected to the overhead projector, typing in code and taking suggestions from the others.
Although I'd already looked at the first two challenges earlier in the day to get an idea of what was involved, the group setting provided a great opportunity to see how other people worked, and to learn about bits of Python that I was unfamiliar with - one example for me was being introduced to list comprehensions, which are concise ways to generate lists, e.g.:
>>> [[x,x**2] for x in vec]
[[2, 4], [4, 16], [6, 36]]
(although there were several other examples which I won't write about here so as not to spoil the challenges for others). Also, as many of the challenges began with having to figure out what the programming problem actually was, it meant that collectively we didn't get stuck for too long on any particular puzzle - I know that at least a couple would have had me completely stumped if I'd been on my own. For me personally it was also an opportunity to play with IDLE - Python's IDE - under Windows (not an environment that I've used much in the past but quite handy for this kind of exploratory programming process.)
Overall it was great to get out and interact with other Python developers in an enthusiastic and friendly atmosphere, while at the same time broadening my knowledge of the language - and now I've had a taste I'll definitely be back for future meetings.
This particular meeting was a coding dojo and so as a group we worked through The Python Challenge (http://www.pythonchallenge.com/), which is a series of puzzles that can be solved using Python programming combined with some imagination and lateral thinking. While most people had come with their own laptops, the format that developed was for one person to "drive" the laptop connected to the overhead projector, typing in code and taking suggestions from the others.
Although I'd already looked at the first two challenges earlier in the day to get an idea of what was involved, the group setting provided a great opportunity to see how other people worked, and to learn about bits of Python that I was unfamiliar with - one example for me was being introduced to list comprehensions, which are concise ways to generate lists, e.g.:
>>> [[x,x**2] for x in vec]
[[2, 4], [4, 16], [6, 36]]
(although there were several other examples which I won't write about here so as not to spoil the challenges for others). Also, as many of the challenges began with having to figure out what the programming problem actually was, it meant that collectively we didn't get stuck for too long on any particular puzzle - I know that at least a couple would have had me completely stumped if I'd been on my own. For me personally it was also an opportunity to play with IDLE - Python's IDE - under Windows (not an environment that I've used much in the past but quite handy for this kind of exploratory programming process.)
Overall it was great to get out and interact with other Python developers in an enthusiastic and friendly atmosphere, while at the same time broadening my knowledge of the language - and now I've had a taste I'll definitely be back for future meetings.
Subscribe to:
Posts (Atom)