Quandy Factory Newsfeed - All

East Side Mario Napolitana Sauce

2010-03-05T12:00:00Z

Ingredients

Chopped Tomatoes
Tomato Puree
Onions
Soybean Oil
Garlic
Salt
Extra Virgin Olive Oil
Basil
Black Pepper

May Contain Citric Acid

MPAA and Piracy

2010-02-26T12:00:00Z

When I was a kid, I had a Fat Albert book in which one of the Cosby Kids (for details of who did what I'm going on memory, but I think it was Bucky) gets upset about something and runs away.

The other kids all search the projects for him, but evening is coming on and it's getting harder to see.

At one point, Weird Harold is walking from streetlamp to streetlamp, peering into the pool of light under each pool. Rudy walks up and asks Harold what he's doing. Harold says he's looking for Bucky.

But why are you looking under the streetlamps? asks Rudy.

Harold answers, That's the only place I can see.

Which brings us to:

(source)

Lost opportunities can tell us much

2010-02-22T12:00:00Z

Published on February 19, 2010 in the Hamilton Spectator

Recent quotes from business owners on the rapid transit choices facing Hamilton add up to a dispiriting - but perhaps inevitable - lack of imagination and understanding.

At issue are the three options in the Metrolinx Business Case Analysis for the east-west route:

Full Light Rail Transit (LRT) from McMaster University to Eastgate Square
Bus Rapid Transit (BRT) from McMaster to Eastgate
Phased LRT from McMaster to Ottawa Street, with BRT from Ottawa to Eastgate until LRT is built in 2030. [*]

Hamilton needs transformative change. Incrementing the status quo has not served us well - particularly the city's poorest neighbourhoods.

LRT doesn't just carry more people more quickly. It attracts hundreds of millions of dollars in new investment and draws many more people to live and work in the area, frequent its businesses, generate demand for new businesses, and interact creatively and productively.

Higher-order urban activity raises infrastructure productivity and boosts the rate of creative economic output. When cities intensify, energy and infrastructure costs grow more slowly than population, but the rate of innovation grows more quickly.

That adds up to more of the net economic growth and employment opportunities that Hamilton desperately needs.

Unfortunately, the business owners quoted lately seem to think LRT is merely about moving people around. I wish someone would organize a fact-finding trip with these business owners to the King-Spadina area in Toronto.

Once a manufacturing centre, King-Spadina was a mess of empty factories and warehouses in the 1990s, when a group of visionary urbanists came together to develop a new plan for the district.

Through a combination of planning rules that encourage mixed-use investment and an anchoring streetcar line, King-Spadina has experienced an impressive influx of new condominium developments, offices and entertainment businesses created through both adaptive reuse and new construction.

When the plan was unveiled, skeptics scoffed. "How will people get there?" they sneered. The answer, of course, is that people moved there in droves.

In just a 45-hectare area, King-Spadina attracted $55.6 million in new investment between 2000 and 2007, creating 700 new jobs and 230,000 square feet of property.

The population has quadrupled since 1996, and the biggest cohort has been young professionals looking for an urban lifestyle close to employment and social amenities.

One would be forgiven for assuming business owners would be head-over-heels about such an opportunity coming to Hamilton.

Surprisingly, business owners may not be the best people to talk to when determining whether and how to transform a neighbourhood. When the economic system is failing most people, does it really make sense to base planning decisions around the few who manage to survive?

The risk of owning a business and the fact that the current system works for them makes such business owners inherently risk-averse. Transformation may well bring huge benefits - particularly for property owners who will enjoy the windfall of rising property values - but it also means that the rules for success change.

The business strategies that work in a poor, failing economic and social environment might not transfer to a booming, thriving environment.

As a result, business owners feel they have a lot to lose.

Compound the fact that many owners don't actually seem to understand that LRT is qualitatively different from buses, and it's a recipe for fear and doubt.

But for every business owner worried that LRT might adversely affect their business, how many potential businesses locate elsewhere or simply never start up at all? How many potential customers never materialize because they chose to live elsewhere?

In the case of a transformative initiative like LRT, there's a real danger that survivorship bias will lead us horribly astray.

Survivorship bias is the error of studying only entities that survived some kind of selection process, and ignoring those entities that did not survive.

Business writer Jason Cohen shares the anecdote of British engineers during the Second World War trying to determine how best to armour planes undertaking aerial raids in Germany.

Studying the patterns of bullet holes on planes that returned from aerial raids, they noticed that the holes were mainly on the wings and tail, with few near the cockpits or fuel tanks.

Survivorship bias means concluding that the planes need more armour on the wings and tail because that's where the returning planes have the most bullet holes.

Of course, the planes that were hit in the cockpits and fuel lines were not available for study.

The business owners along the B-Line are survivors of the economic battle downtown Hamilton has suffered over the past half-century.

How many businesses were "shot down" because of Hamilton's low-quality transit, low population densities and pedestrian-repellent one-way streets?

Who will speak for those failed businesses and lost opportunities? How can we incorporate them into our public discourse so the survivorship bias of existing businesses doesn't lead us astray?

* Note: under the phased approach, the route east from Ottawa Street would be served by the current BRT-like B-Line service, not by BRT on dedicated lanes.

Hamilton Spectator PDF Downloader

2010-02-15T12:00:00Z

Review: Mark Pilgrim's Dive Into Python 3

2010-02-08T12:00:00Z

TL;DR Summary

If you're an experienced programmer, want to learn Python 3, and don't have a lot of time to waste, skip this review and just go straight to Mark Pilgrim's Dive Into Python 3.

DIP3 starts with practical, working code, takes it apart piece by piece, puts it back together, and leaves you with a solid understanding of the concepts and their applications.

DIP3 is opinionated but well-informed. It hammers home the important stuff and skips the blather. It's also very well written: terse, witty, and expressive (like Python itself).

Introduction

Mark Pilgrim's Dive Into Python 3 came out late last year, but I didn't receive a review copy until late January - complete with autograph and a friendly note from the author apologizing for the delay. (Disclosure: I felt warm and fuzzy after receiving the personal note. Also, the review copy I received was free.)

If you're familiar with Mark Pilgrim, you'll know that he's an opinionated writer. DIP3 is no exception; this is Python 3 the way Mark Pilgrim wants you to understand it.

The good news is that Pilgrim is a reliable narrator. That is, he really knows his stuff; and his opinions, while strong, are deeply knowledgeable, ethically and philosophically consistent, and shared in a spirit of cooperation and stewardship.

Purpose and Relation to Dive Into Python

Comparisons to Pilgrim's original Python book, Dive Into Python (also published by Apress), are inevitable. The good news is that this edition improves on the original in most of the ways that matter. The book design is more elegant and stylish, with better use of white space and contrast, cleaner fonts (with one notable exception, about which more below), and clearer layout.

The book is written primarily for experienced programmers coming to Python for the first time, but Pilgrim recognizes that a lot of its readers will have a background in Python 2.

Since Python 3 breaks compatibility with the previous trajectory of versions, the book contains specially formatted bullet notes at points where version 3 breaks with 2.x; for instance, the merging of int and long datatypes, or the fact that the / division operator now triggers floating point division by default, rather than integer division.

By focusing mainly on Python 3 itself and then highlighting the diffs from Python 2, Pilgrim generally gets the best of both worlds.

Still, there are a few ways in which the book isn't entirely sure whether it wants to be a resource for programmers learning Python for the first time, or for Python 2 programmers who want to upgrade. It would be more helpful for experienced Python 2 programmers if Pilgrim noted the diffs consistently rather than sporadically.

Another significant change from DIP is the way it uses programs to introduce the language. In the original book. the introductory program was an 11 line script to build an ODBC connection string from a dictionary. It demonstrated function declaration, datatypes, everything-is-an-object, code blocks (and significant whitespace), and the if __name__ == '__main__' trick.

The next chapter covered dictionaries, lists, tuples, declaring and returning variables, string formatting (old style, with % () notation), and basic list comprehensions. The program was dissected exhaustively over two chapters and concluded with an explicit acknowledgment:

The odbchelper.py program and its output should now make perfect sense.

In DIP3, by contrast, the introductory program spans four chapters - and there's never a discrete aha moment where you realize that you understand the program fully.

Still, overall the book is a commendable accomplishment: a concise, accessible, and above all fun introduction to a language also known for concision, accessibility and fun.

Why Python 3?

Python 3 has not yet enjoyed wide adoption among programmers - existing Python programmers or otherwise - and part of the reason is that the language is caught in the chicken-egg problem whereby the lack of ported libraries makes it less appealing for coders, while the lack of existing coders makes it less compelling for developers to port their libraries.

My hunch is that Dive Into Python 3 will carry us closer to the tipping point in which coders and library supporters start to upgrade en masse.

For one thing, Python 2 programmers will come away from this book with a much clearer understanding of how many problems and pain points Python 3 solves. In addition, version 3 sweetens the deal with powerful new data structures that already make me envious when I go back to work on projects written in 2.x.

Diving In

If you've read programming books before, you'll know that the general format is to start with lots of theoretical background and history, then introduce the syntax, data types and common code blocks; and finally, after a few chapters, start putting together a program.

That was my original experience trying to learn Python a few years ago by reading through How to Think Like a Computer Scientist: Learning with Python by Allen Downey, Jeff Elkner and Chris Meyers.

It wasn't until I read Mark Pilgrim's original Dive Into Python that I really started to get an internalized sense of the language. He raised 'learn by doing' to the level of an art form, and it clicked with me in an immediate and sustained way.

It becomes clear that Dive Into Python 3 follows the same aggressively practical, hands-on approach as soon as you read the opening line of the introduction:

Welcome to Python 3. Let's dive in.

The introduction covers installing Python 3, and it's a significant improvement on the equivalent chapter in the original DIP. If you run Windows, Mac OSX or Debian/Ubuntu, the book takes you step by step through the installation process. (If you run a more exotic operating system, you can probably figure out how to install Python all by yourself.)

Basics: Functions, Datatypes and Comprehensions

Chapter 1 covers function declaration and arguments, doc strings, sys.path, objects (and the oft-repeated fact that everything, in Python, is an object), denoting code blocks through indentation (i.e. significant whitespace), exceptions, variable declarations, and the if __name__ == '__main__' trick.

These are the absolute basics, and it's noteworthy that Pilgrim includes an introduction to exceptions. He's an opinionated programmer, and he wants you to understand that exception handling is fundamental to good Python code, not an exotic extra to be mentioned in passing once you're already proficient.

Chapter 2 dives into Python's datatypes: booleans and boolean contexts, numbers, type coercion, operations, fractions, lists, tuples (immutable lists), dictionaries, sets (new in Python 3, sets are unordered lists with a syntax similar to dictionaries), and the special None type.

Chapter 3 introduces comprehensions, a delightful set of syntactic sweeteners that allow you to map and filter an iterable collection using a terse one-liner:

newlist = [func(item) for item in oldlist if test(item)]

New in version 3, Python adds dictionary and set comprehensions to the list comprehensions it already supported.

Aside: this is one of the places where Pilgrim doesn't specifically mention the difference between Python 2 and 3. I found myself going back to the Python 2 documentation for a sanity check just to make sure these structures weren't there and I just somehow missed them.

Bytes vs. Characters

Chapter 4 breaks form. Instead of diving straight into code, Pilgrim opens with three pages of exposition - the only such indulgence in the book, and with good reason. Possibly the most beautiful and simultaneously despairing chapter in the book - indeed, this achieves a level of pathos befitting a novel, let alone a technical manual - Pilgrim recounts the tragedy of text on an international data network. And it is a tragedy.

In what might be the most significant break from verson 2, Python 3 consistently and explicitly treats strings as streams of bytes, not streams of characters.

Because a string is a stream of bytes, it must be encoded using a string encoding that maps the bytes to specific characters.

In Python before version 3, the default encoding was ASCII, the American Standard Code for Information Interchange, a seven-bit encoding that handles all conventional English characters - the lowercase and uppercase letters, numerals, and punctuation symbols - plus various control characters including tabs, spaces, newlines and so on.

Of course, not everyone speaks or writes English, and many other languages use additional characters (like the French e-accent-aigu é or the German eszett ß) or even collections of characters with little or no overlap to English (like Japanese Kanji). For these languages, ASCII is wholly inadequate, and different languages have independently developed various encoding systems that often map the same character to different bytes.

The potential for chaos is huge, and indeed a number of incompatible encodings have already caused plenty of grief for people trying to process text on computers.

We've all seen web pages with jumbles of nonsense characters where you would expect, say, quotation marks to go. This is caused by a mismatch between the encoding used to produce the text - say, ANSI CP-1252 in Microsoft Word) and the encoding used to render the text later (say, ISO/IEC 8859-1).

Pilgrim sets the scene eloquently and then, with a seemingly innocuous "Enter Unicode", leads us on a harrowing emotional tennis match, swinging back and forth between optimistic proposed solutions - let's agree to put every single character into one big encoding! - and new problems - each character now takes up 4 bytes! - to follow-up solutions - use agreed-upon subsets! - to still more problems.

Now cry a lot because everything you thought you knew about strings is wrong, and there ain't no such thing as plain text.

Pilgrim doesn't offer simple solutions to the problems he raises, because they don't have simple solutions. Character encoding may be the Great Granddaddy of more-or-less insoluble technical problems. They can't be solved, as such, but if you understand the problems well enough they can be addressed more or less safely.

But as Pilgrim points out in regards to the necessary shortcuts that programs take in mapping out solutions:

[It's] a good assumption right up until the moment that it's not.

String processing is further complicated by the matter of big-endian vs. little-endian byte ordering - which is only a problem when people try to share files from different computers, "perhaps on a worldwide web of some sort".

With all this in mind, you can't help but conclude, despairingly: Character encoding is hard!

But rather than just giving up and going shopping, Pilgrim leads you through the maelstrom and delivers you - shaken but intact - on the mostly-safe harbour of UTF-8. Not that UTF-8 isn't also beset with traps and gotchas.

Chapter 4 demonstrates formatting strings (using Python's powerful new string formatting syntax, first introduces in version 2.6), common string methods, string slicing (a string, after all, is a list of ~~characters~~ bytes), the difference between strings and bytes (lots of gotchas exposed here), and encoding of source code (Python 2 files were ASCII by default, whereas Python 3 files are UTF-8 by default).

Digression on the Book's Text Formatting

This is as good a place as any to mention a formatting bug in this edition: the peppering of the text with artifacts - hollow vertically aligned rectangles - in place of special characters like em-dashes. For example:

The first line imports the humansize program as a module▯a chunk of code that you use interactively or from a larger Python program.

Given the attention Pilgrim gives to character encoding (I still remember the reddit thread on which he posted an early draft of that section for review and feedback - some of it unbearably pedantic), it seems bizarre that his own book would be bitten by an encoding gotcha!

I contacted Pilgrim to ask if he knew what happened but did not receive a response in time for publication. It may be related to the fact that he had to convert the book from HTML to MS Word before submitting it to the publisher.

Regular Expressions

Chapter 5 delves into regular expressions, a powerful and pragmatic DSL for solving real-world data extraction problems. Yet in what is turning into a common theme, regex is complicated and contains some gnarly gotchas and edge cases.

Pilgrim takes us through real-world use scenarios in much the same way that real workaday programmers approach them: by iterating through a sequence of progressive changes and enhancements, until we're satisfied that the solution is Good Enough.

One of the nicest touches of Python's support for regex is verbose mode, which ignores whitespace and allows inline comments so that the regex still makes sense when you have to look at it again in six months. Unless you're parsing text on a daily basis and know regex inside and out, this will be helpful to you.

Iterating Through Iteration

Chapter 6 tackles closures and generators. While Python is not as purely functional as a Lisp or even Ruby, it still provides some powerful functional tools. Pilgrim takes a thorny problem - matching pluralization rules for a variety of English nouns - and combines regular expressions with generators (functions that yield their values lazily and iteratively) and meta-functions (functions that take functions as arguments and return functions).

Pilgrim combines these tools to build generators that act as closures and yield functions based on regex pattern matching. It's heady stuff, but by the time he's finished taking it apart and putting it back together, you can't help but understand it. Along the way, we learn about files and file-like objects, and the powerful with context (about which more below).

Chapter 7 dives deeper, tackling the same problem using classes and iterators. Everything in Python is an object, with properties and methods. In this chapter, Pilgrim explains who you can define your own classes and instantiate them as objects - this case, an iterator class.

He covers Python's special reserved word pass, which does nothing but is necessary for those occasions when Python syntactically requires something - what Pilgrim calls "a Python reserved word that just means, 'move along, nothing to see here'."

He covers Python's special __init__() method, distinguishing it semantically from the C++ constructor method; the special self argument in Python clases; instantiating classes; instance variables; passing in parameters and calling methods.

Then he shows how to create an iterator class and revisits the pluralization code from the previous chapter to produce the same results as a more terse, abstract form.

Aside: when I first discovered Python, I was delighted to discover that it appeared to be lists all the way down. Pilgrim argues by way of contrast that it's actually "iterators all the way down."

Chapter 8 indulges in a bit of whimsy by way of introducing the itertools library in the context of an extremely terse (just 14 lines!) Alphametics solver. Pilgrim dives a little deeper into regular expressions, introducing the findall() method and highlighting another pattern matching gotcha (it doesn't return overlapping matches). He also shows how you can use the new set datatype to extract the unique items in a list.

We also learn how to make assertions and catch AssertErrors with the terse, one-line syntax you may have come to expect from Python by now.

Speaking of which, Pilgrim revisits the generator paradigm with a one-line generator expression. If you prefer not to iterate, you can pass the generator right into a tuple(), list() or set() function.

An important paradigm of functional programming is the use of lazy evaluation, in which the values in an iterable object are calculated on the fly as needed rather than all at once.

Python comes equipped with the powerful itertools module, which includes methods for generating permuatations, products, combinations, chains, groups, and more.

Along the same lines, Pilgrim shows how to treat a file as an iterable list (most things in Python are iterable), with all the methods and functions available to manipulate lists.

The chapter closes with a vintage Pilgrim dive when he iterates through the chain of problems / solutions / new problems related to Python's eval() function. eval() is powerful - and dangerous, since it can run arbitrary code that does anything Python can do (subprocess.getoutput('rm-rf') anyone?).

Say it with me: "eval() is evil!"

It turns out that there is no truly reliable, safe way to expose eval() to untrusted third parties.

Unit Testing

Chapter 9 introduces unit testing. If you unit test, you'll appreciate the value of a powerful, simple testing framework; if you don't unit test, you may well close this paragraph a convert.

Python, of course, comes with a unittest module that lets you subclass simple, robust test suites to prove, empirically, that your code does what it's supposed to do (at least on a micro level).

Using unittest and the code from Chapter 5 to generate Roman Numerals, Pilgrim takes you through testing, debugging and (in Chapter 10), safe refactoring.

His philosophy is simple and elegant:

Write tests.
Write code.
Test code.
Debug code.
When all the tests pass, stop coding.

Over the course of two chapters, you'd be hard pressed not to come away persuaded that Pilgrim's onto something here.

File I/O

Chapter 11 dives deep into files. I/O is central to computing, and Pilgrim has touched on files and file-like objects several times so far. In this chapter, files are the main attraction.

In Python 3, the open() function takes an encoding argument so that Python knows how to encode the string of bytes that is a file:

Bytes are bytes; characters are an abstraction. A string is a sequence of Unicode characters. But a file on disk is not a sequence of Unicode characters; a file on disk is a sequence of bytes. So if you read a "text file" from disk, how does Python convert that sequence of bytes into a sequence of characters?

If you don't specify an encoding, Python 3 uses your system's default encoding (e.g. Windows, where the default encoding is CP-1252). That might be the file's encoding, but it might not - assume at your peril! This is especially true given that Python is cross-platform. Code written on, say, a Linux machine that makes assumptions about file encoding might suddenly fail on a Windows machine.

Pilgrim then dives into opened files as stream objects, with useful properties and methods, including readline(), seek() and tell(). Stream objects also behave like iterators, yielding a line at a time.

In addition to reading files, Python can write files and append new lines to files. But not all files are text files. Python can also open, read, write and append to binary files, byte by byte.

Further, Pilgrim explains how to use the io library to create and manipulate file-like objects that may not correspond to actual files on disk but still possess file-like properties and methods.

Of course, Python can also handle compressed files via the gzip module. (The standard Python library is phenomenal in its breadth and usefulness.)

Python is thoroughly cross-platform, but it allows you to use and redirect the stdin, stdout and stderr pipes indigenous to *nix systems. Python shows you how to hook into these pipes.

Pilgrim advocates using the with context as the safest way to handle files and file-like objects. He notes that Python 3.1 supports multiple nested with contexts (if you have Python 3 installed, take a few minutes to upgrade to 3.1).

Serialization via XML, Pickle and JSON

Chapter 12 wrestles XML. Yes, XML is a special flavour of hell, but you will have to deal with it from time to time, and Python has easy, powerful tools to help you. He sketches out a crash course on XML, concluding, "Now you know just enough XML to be dangerous."

The standard library offers xml.etree.ElementTree to parse, walk, search, and iterate (it really is iterators all the way down) through, create, and modify your XML object.

Pilgrim pauses to draw our attention to the third lxml module, a third-party library that replicates the ElementTree API but extends it with more powerful search methods.

With matching APIs, your code can prefer the more powerful third party library but fall back on the standard library if the former is not installed.

Pilrim also shows how you can dig into an XML object with bad syntax and recover its data using lxml.

Of course, XML is evil (albeit an often necessary evil), and Chapter 13 introduces other methods of serializing Python objects.

Sometimes you want to store more than just strings in your data persistence routine, and Python has (wait for it) powerful tools for serializing even complex objects and datatypes - like the standard pickle module, which has dump() and load() methods for storing object,s dicts, lists, and so on.

But pickle is only good for Python programs, and data often has to pass between different applications written in different languages. Fortunately, Python comes equipped to handle JSON, or JavaScript Object Notation, a data format developed by Douglas Crockford and inspired by javascript object literal syntax (in fact, JSON is valid javascript).

JSON can handle objects, lists and dicts, and the standard json library can handle JSON. You can also extend JSON with custom serializers for unsupported data types, like tuples.

Pilgrim really knows how to hammer home the gotchas. In his umpteenth reminder to define the encoding of an opened file:

You'll forget! I forget sometimes! And everything will work right up until the moment that it fails, and then it will fail most spectacularly.

Take a REST

Chapter 14 takes the reader through creating RESTful, HTTP-based web services. (Pilgrim wisely decided not to update the chapter on SOAP web services from the orginal DIP.)

Python does HTTP via the urllib and httplib modules (standard library), but Pilgrim recommends the third-party httplib2 module instead. It's more complete and clean than the former two. In particular, it supports HTTP caching, last-modified testing, ETags, compression, and 30x redirects.

Pilgrim notes:

urllib speaks HTTP like I speak Spanish - enough to get by in a jam but not enough to hold a conversation. HTTP is a conversation. It's time to upgrade to a library that speaks HTTP fluently.

Pilgrim shows how to fetch data from a web resources without being rude or inefficient, by accepting compressed data, caching files locally, remembering permanent redirects, and so on.

He also demonstrates RESTful interactions with HTTP GET, POST, UPDATE and DELETE requests to a web API (in his example, identi.ca, an open source, twitter-like microblogging service).

Important note: httplib2 returns bytes, not strings. Yes, the character demon rears its head again. You need to specify an encoding.

Porting and Packaging

I've mentioned that DIP3 doesn't always explicitly highlight the differences between Python 2 and 3. Chapter 15 comes as something of a remedy.

The bytes/characters dichotomy is paramount throughout this book, since it's a huge gotcha that will burn you sooner or later. (It's one of the reasons the Python development team decided to break compatibility to create Python 3 in the first place.)

It's not surprising that Pilgrim ported a library (from Mozilla) that guesses the character encoding of byte sequences into Python. In this chapter, Pilgrim walks the reader through the sometimes painful exercise of porting the library, called chardet, from Python 2 to Python 3.

While the book has noted difference between the versions along the way, it has not served as a systematic guide to the changes. As Chapter 15 shows, there are some important but subtle differences under the hood that will break your code and may be infuriating to track down.

Python includes the 2to3.py tool, which automates the automatable aspects of porting, and highlights those issues it couldn't handle.

Of course, every chapter does double- and triple-duty, and this chapter also introduces the structure and internal arrangement of multi-file libraries.

Speaking of which, Chapter 16 covers packaging Python libraries for distribution. If nothing else, Pilgrim gets an A+++++ Would Definitely Read Again for demystifying the distutils library and unlocking the PyPi online catalogue.

I've always found the documentation on packaging to be arcane and inaccessible, and this chapter is a much-appreciated remedy.

Conclusion

DIP3 is not suitable as an introduction to programming, since Pilgrim assumes you bring a background in programming concepts to the table; but it's perfect for busy programmers looking to gain mastery in Python 3.

Over the course of a witty, compelling narration, the book repeatedly hammers both Python 3's most important strengths (terse syntax, rich datatypes, everything-is-an-object, abundant iteration, powerful libraries) and most dangerous pitfalls (bytes vs. characters).

Pilgrim wastes no more time than is strictly necessary on exposition. We program to solve problems, and DIP3 keeps problem solving - practical, real-world problems - in the driver's seat throughout.

Notwithstanding a few minor quibbles, I can heartily recommend this book to anyone who wants to tackle Python 3. Since reading it, I find myself thinking about my own modest Python applications - sooner or later I'm going to have to cut the cord and make the jump to Python 3.

After reading Dive Into Python 3, that point seems closer than ever.

Reference

Mark Pilgrim, Dive Into Python 3, Apress, 2009.

DIP3 is available for download in HTML or PDF; or you can clone the document repository:

you@localhost:~$ hg clone http://hg.diveintopython3.org/ diveintopython3

Top 10 Programming Lessons in 10 Years

2010-01-25T12:00:00Z

After reading Top 20 Programming Lessons I've Learned in 20 Years by Jonathan Danylko, a recent article featured on Hacker News, I was inspired to come up with my own list of programming lessons.

These lessons are pretty basic, but they have served me well over the years. When I find myself getting lost in a project, it helps to go back to the basics and get some perspective.

I have learned most of these lessons the hard way, i.e. by not following them and getting burned. In some cases, I have been burned more than once. (What can I say? I'm a slow learner.)

On the other hand, following every lesson consistently always leads to positive outcomes. Many of these lessons seem like additional work, but my experience has always been that they save time and aggravation over the life of the project. A little investment of effort up front can prevent a lot of catch-up work down the road.

So here's my top 10 (plus one bonus) list of programming lessons I've learned over the last ten years.

Break it down.

A whole project - even an ostensibly simple one - is overwhelming to contemplate and leads to defensive procrastination. Always take the time at the beginning to break the project into manageable components, even if it seems like a waste of time. You'll be glad you did when a week or two goes by, the project scope has expanded, and you're starting to lose site of the big picture.

With a list of components, you don't have to worry about the big picture - you just work through the steps until complete. Of course, this means you must be sure to review and revise the list of steps as your requirements change.

Your requirements will change.

Accept it. Embrace it. Understand that the final product will be better if you're willing to change your mind when the facts change.

It's generally impossible to know, in advance, exactly what the end product is going to look like. The harder you try to detail the end product in absolute, unchanging terms, the more wrong you will end up being.

Obey Gall's Law.

When coding a project, take the shortest path to a simple system that works, even if it's only a fractional subset of the total project requirements. Then incrementally add functionality, testing as you go, until the project is completed.

Document as you go.

This includes both code comments and external documentation for application and/or API users. Every time you add or change a feature, update the documentation to reflect the change. Just make it a core part of your workflow.

If you wait until you've finished coding to start documenting, it will end up looking forced, rushed, and incomplete. Ideally, include your documentation right inside your version control.

Use version control.

Disk space is cheap and abundant, so commit early and commit often. Make sure your commit comments reflect what you're capturing in each snapshot.

Maintain separate development and production environments.

Never you mind that 'quick fix' that will just take a monent to type out. Put it in development, test it, commit the change, and then push to production. Do this every time, and you won't end up looking like an asshole when something goes wrong.

Backup and restore.

This should be a no-brainer, so don't be one of those people who has to learn the hard way. You need a proper, reliable, redundancy-tested offsite backup, and you need to test regularly that you can restore from your backups.

No, a RAID array is not a backup solution. Nor is blindly trusting your hosting provider to do their job.

Leave your code in a working state at the end of every day.

Don't walk out on code that won't run. It's surprisingly demoralizing to come to work the next day knowing that a broken build is waiting for you. If you have to roll back or comment out a half-finished code block, do it.

In addition, it's helpful to draft up a quick todo list so you can quickly pick up where you left off the next day. (Thanks to bsaunder on Hacker News for this suggestion.)

If you can't figure out a problem, walk away.

Cognitive science tells us that our brains protect us from the stress of not knowing how to solve a problem by ... hiding key information about the problem from consciousness. Thanks, brain.

For small to medium-sized problems, I find a good brisk walk is enough to break the logjam in my mind and see through to a solution. For big problems, I may have to pull out the big guns: a good night's sleep.

I'm not being silly or even exaggerating: sleep on it is an essential tool of my problem solving strategy, a tool on which I rely regularly, and which has yet to fail me.

Fix bugs first.

Don't add new features while any identified bugs are still outstanding. When part of your system doesn't work, the last thing you want to do is introduce additional complexity.

When troubleshooting bugs, try to be scientific about it. Don't just randomly make changes and hope the bug will go away - this is a recipe for introducing more bugs. Make sure you come to understand what's causing the bug so that you can be confident your fix really is a fix. (Thanks to bendtheblock on Hacker News for this suggestion.)

Communicate, communicate, communicate.

When you're developing a project for someone, keep the channels open at all times. Clearly communicate your timelines, progress and difficulties and provide regular status reports.

If you realize something is going to take longer than you expected, resist the urge to go into hiding! Instead, notify them as early as possible. Likewise, if the client comes back to you with changes that impact your time lines, be very clear in communicating what those impacts will be.

Finally, though it may seem like a hassle, make yourself available to answer questions and provide demonstrations. If you followed #3 and #8, you will always have something to show.

PyToc

2010-01-19T12:00:00Z

Introduction

PyToc generates a table of contents for an HTML document based on headings, with anchor links from the TOC to specific headings.

Download

You can download the latest version of PyToc from its github repository:

http://github.com/quandyfactory/PyToc

Requirements

Python 2.5 or 2.6
BeautifulSoup 3.0.8

Using PyToc

You can see the code in action on my website's About page.

It's pretty simple to use. Download pytoc.py and save it somewhere in your PATH.

Here's a demonstration:

>>> import urllib
>>> import pytoc
>>> url = 'http://quandyfactory.com/projects/40/pytoc'
>>> page = urllib.urlopen(url)
>>> html = page.read()
>>> toc = pytoc.Toc(html_in=html)
>>> toc.make_toc()
True
>>> toc.html_toc # returns an HTML table of contents
>>> toc.html_out # returns the html with anchors and numbering in headings
>>> toc.toc_list # returns a list of tuples in the form (section number, title)

Input Properties

The following are input properties you enter to generate the table of contents.

html_in - The HTML document for which you want to generate a table of contents.

This is the only necessary property to assign. The rest have default values that may meet your needs.
levels - A list of numbers corresponding to the heading levels you want to include in your TOC.

E.g. [3, 4] would include <h3> and <h4> headings.

Default is [3, 4].
id - The base id of the HTML table of contents to be generated.

Default is "toc".
title - The title of the generated table of contents.

Default is "Contents".

Methods

make_toc() - this generates the table of contents and populates the output properties.

Returns True when complete.

Output Properties

After calling the make_toc() method, the following output properties are populated with values.

htmlout - The same as htmlin except with the TOC anchors and numbering included in the headings.
html_toc - The generated HTML table of contents.
toc_list - A list of tuples containing the anchors and headings, in case you would rather roll your own HTML table of contents.

That's it, really.

History

This library started out as one-off code to generate a table of contents for a long document that I'm converting from MS Word format over to HTML.

The existing Word document has had many contributors and editors over the years and the format is a shambles. The table of contents is a mess and the headings are all over the place. (Thanks, WYSIWYG.)

I converted the whole thing into Markdown, a simple plain-text formatting syntax that converts to clean, structural HTML.

I still wanted the final document to have a table of contents, but I didn't want to have to go through the bother of maintaining the thing - especially if I ever wanted to add a new section in the middle, which would require a re-numbering of all the subsequent sections and subsections.

I whipped up a simple parser that walked the document and dynamically generated a table of contents, plus anchors in the document so the section headings listed in the contents could straight down to the sections themselves.

Need for Flexibility

It worked, but the code was brittle. It required the HTML formatting to be very strict, e.g. the following would work:

<h3>Some subheading title</h3>

but the following would not work:

<h3>
  Some subheading title
</h3>

Documents generated using python-markdown2 would work pretty consistently, but documents generated using, say, python-markdown would not, since the latter produces messier HTML output.

Of course, with documents produced using other means, all bets were off.

Anyway, I decided that if this was going to be at all useful as a general-purpose tool, it needed to be more flexible and forgiving. Of course, if you're programming in python, parsing HTML and want to be flexible and forgiving, there's no better tool than the mighty BeautifulSoup parsing library.

So I re-wrote it using BeautifulSoup.

About

2010-01-11T12:00:00Z

Introduction

My name is Ryan McGreal, and I live in Hamilton, Ontario, Canada with my family. I work as a web programmer, writer, editor and troublemaker, though it's the programming that pays the bills.

Raise the Hammer

My principal activity that doesn't pay the bills is my role as editor of Raise the Hammer, an online magazine dedicated to sustainable urban revitalization in Hamilton. I also work as the city editor at H Magazine.

Hamilton Light Rail

I am also a proud founding member of Hamilton Light Rail, a community group dedicated to bringing light rail transit to Hamilton.

Published Essays

I have written several essays on urban issues that have been published in the Hamilton Spectator and elsewhere over the past five years.

Contact

You can reach me via email at ryan@quandyfactory.com.

This Site

This is my personal website, repository of essays and projects, and playground for new ideas.

Quandy is a portmanteau of "Quick and Dirty", which can be a useful method of approaching problems. "Quick and dirty" has the benefit of being, well, quick, as well as flexible for those cases when initial requirements end up changing (i.e. just about every nontrivial project).

It suggests an iterative approach, on the reasoning that it's easier to build something simple and then make it better than it is to try and spring a fully-formed application from your forehead.

As John Gall famously stated:

A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work.

Eye Quandy

Quandy Logo

The red Quandy logo in the top left corner is courtesy of Trevor Shaw, a great local graphics designer and the creative director of Juice Creative.

Footer Image

The awesome cityscape panorama in the footer was taken by the talented photographer and amateur urbanist Aaron Segaert, and is used with permission.

Interests

In recent years I have been particularly interested in: the nature of city economies and urban development; the role of public participation and community engagement in creating and sustaining a healthy society; and ways to increase the openness, transparency and responsiveness of organizational governance and policy making.

Conceptual Overlap

I admit that my ideas about openness in government and policy making reflect my experience using and developing software: an open, information-sharing approach with peer review results in better results than a closed, proprietary approach based on blind trust.

Jack of All Trades

My interests take me all over the place, figuratively, from land use patterns and transportation modes to the global energy situation, geopolitics, social policy, economics and political economy, democratic structures and traditions, broad-based community organizing, local politics and current affairs, architecture, city life, ecology, sustainability, cognitive psychology, and more.

I don't claim expertise in any of these areas, but I am committed to studying the experts and following empirical best practices in these domains.

Benefit from Shared Expertise

The great thing about living in an open, knowledge-based culture is that you can benefit from the expertise of others. Once you establish the credibility of expertise, you can use it as a kind of knowledge API that allows you to take advantage of the expertise without necessarily knowing everything about the internals.

If not for this ability for non-experts to access expertise, there would be no way for the benefits of that expertise to disseminate into the broader society and inform our policy decisions.

Programming

I enjoy programming and have benefited immensely from the vast, rich ecosystem of free and open source software available to programmers today (see "Technical Notes", below).

Great Time for OK Coders

I know enough about great programmers and their remarkable contributions to understand that I am not a great programmer. Nevertheless, the rich ecosystem of programming languages, libraries, frameworks and tools means even a duffer like me can be creative and productive - and that's a Good Thing.

Productive Modern Languages

One of the great things about modern programming languages is how highly expressive they are. You can create working code very efficiently, with a minimum of boilerplate.

That means it's easy to develop simple tools that do exactly what you want them to do and no more - and to do them quickly.

Shared Open Source Software

Recently I have begun releasing a few such handy tools under a free software / open source licence. You can find my shared resources hosted on GitHub.

The code isn't beautiful, but I'm no genius.

Website Development

I do a bit of freelance web application development. Feel free to contact me via email at ryan@quandyfactory.com to inquire about services and rates.

Outsourced Graphic Design

I am not a graphic designer, and my own website design tends to the very minimal. However, I do have a good working relationship with a talented graphic designer who can design the layout and colour scheme to reflect your organization.

I am also happy to work with a design that you provide.

Technical Notes

There's no particularly good reason why I didn't simply use WordPress or Drupal or some other off-the-shelf blogging software for this site; except that I enjoy building things (also, PHP makes the baby Jesus cry).

Anyway, it's not like I built the site from scratch.

It runs on an Apache web server on a Linux machine hosted by the awesome admins at WebFaction.
It is written in the Python programming language and uses the lightweight web.py application development framework.
Web.py talks to Apache via the mod_wsgi server module, which implements the standard Web Services Gateway Interface (WSGI) specification for Python applications to communicate with web servers.
The site stores its documents in a MySQL database, to which it connects via the ingenious SQLAlchemy database toolkit and object-relational mapper (ORM).
Documents are saved in Markdown syntax and converted to HTML for display using the python-markdown2 library (which is itself a re-implementation of the original python-markdown library).
It also uses Quandy, a library of handy classes and functions that I use frequently in writing web code.

In other words, I'm sitting here on the shoulders of giants - and the view is grand!

The Virtue of Forgiving HTML Parsers

2010-01-08T12:00:00Z

Today Hacker News featured an essay by Alex Russell of the Dojo javscript toolkit in which he mused on the virtues of view-source as they apply to the internet.

Since browsers render HTML, javascript and CSS from plain text, it's possible not only to see the finished product - a rendered web page - but also its underlying source code.

Every browser I've ever seen includes an option to view source - to see the underlying HTML markup, javascript code and style sheet formatting that the browser uses to render a web page.

Russell makes the important point that it was the ability to view source - especially in the early days of the internet - that made web content easily accessible to anyone who took the time to read the code and to experiment with it to discover how it works.

View-source provides a powerful catalyst to creating a culture of shared learning and learning-by-doing, which in turn helps formulate a mental model of the relationship between input and output faster. Web developers get started by taking some code, pasting it into a file, loading it in a browser and switching between editor and browser between even the most minor changes.

This is a stark contrast with other types of development, notably those that impose a compilation step on development, in which the process of seeing what what done requires an intermediate action.

In other words, immediacy of output helps build an understanding of how the system will behave, and ctrl-r becomes a seductive and productive way for developers to accelerate their learning in the copy-paste-tweak loop.

The only required equipment is a text editor and a web browser, tools that are free and work together instantly. That is to say, there's no waiting between when you save the file to disk and when you can view the results. It's just a ctrl-r away. [paragraph breaks added]

On reading this, a parallel observation occurred to me: The power of view-source is multiplied when generous parsers forgive errors and render anyway.

The value of a big, broad internet is far greater than the value of a clean, pure internet.

Most programmers will agree that programs or data streams with malformed syntax should fail, and fail fast. The worst thing that can happen with a software application - particularly a complex one - is for errors to pass silently while malformed data flows into data storage, only to produce gibberish - or worse, subtly wrong results - on export.

Passing a string into a function that expects an integer should throw an exception; passing a set of four values into a method that expects three parameters should raise a red flag; and so on.

By contast, the HTML parsers in every browser are extremely forgiving. If an HTML parser encounters an opening tag for an element that is missing a corresponding closing tag, it just accepts the missing close tag as implied and renders as if it was there.

Likewise, if the parser encounters a malformed or non-standard element, it will either try to render it somehow by guessing at the code's intention (naturally, different parsers approach such matters differently) or will just ignore it completely and move onto the next line.

As a result, even badly malformed HTML can still produce a readable web page.

Purists bristle at this, insisting that it makes the web a worse place by forcing browser makers to code for errors, encouraging sloppy coding practices, causing the same content to be rendered differently on different browsers, and so on.

They're missing the point. The virtue of forgiving parsers is that they vastly increase the pool of people able and willing to create web content.

If you're already a programmer, HTML syntax is easy enough to understand and produce in valid form.

However, most people who create web content aren't programmers - particularly during the early days of the internet when it grew exponentially and established the virtuous cycle of positive network externalities that ultimately dragged more professional developers onto that platform.

Rather, they were amateur enthusiasts exploring a new technological domain. Thanks to HTML, view-source and forgiving parsers, the number of people who could create web pages was vastly higher than the number of people who could write computer programs.

It was the rapid democratization of HTML made possible by view-source and forgiving parsers that accounts for much of its success as a language - and of the success of the internet as a platform.

When an HTML parser finds code so bad that it can't render it, the parser just skips it and moves to the next line, in the manner of VB's on error resume next (programmers are welcome to cringe here). Contrast the stricture of XML parsers, which are obliged to fail on encountering malformed code and produce no output at all.

Since HTML rendering in response to an HTTP GET request is essentially idempotent, there's no real harm in continuing to parse code after encountering an error - but the positive network effects from this are huge.

If only programmers possessed the arcane ability to produce well-formed, valid markup, it would never have experienced the early growth that transformed it into the reigning standard of a huge and growing public network.

One more thing: for many people, HTML provided a gently sloping pathway into programming for people who might otherwise never have managed to overcome the steep barriers to entry of, say, C, with its verbose syntax and elaborate requirements. (Compiler? What's a compiler? And why does it hate me?)

Many programmers today (myself included) found their way into programming from a start in HTML, after bumping into the limitations of static content, exploring event handling in javascript, and then making the jump over to PHP or classic ASP and thence to SQL - and ultimately over to more modern languages with more robust, structured software design principles baked into them than the spaghetti code that powered a lot of early dynamic websites.

Again, there are some people who consider this to be a terrible thing - a watering down of an industry that ought to have high barriers to entry. Setting aside my own interest in the matter, I disagree. Getting more computing and networking capability directly into the hands of more people can only increase the rate of technical innovation by deploying more expressive power more widely.

Stack Trace for hnsh.py on proxy

2010-01-07T12:00:00Z

The Stack Trace

Here's the original stack trace (Python 2.6 on a Windows XP machine behind a proxy server) for Hacker News Shell.

C:\Python26>python hnsh.py
Traceback (most recent call last):
  File "hnsh.py", line 558, in <module>
    hnsh = HackerNewsShell()
  File "hnsh.py", line 314, in __init__
    self.stories = self.h.getLatestStories(self.alreadyReadList)
  File "hnsh.py", line 194, in getLatestStories
    source = self.getSource("http://news.ycombinator.com")
  File "hnsh.py", line 35, in getSource
    f = urllib2.urlopen(url)
  File "C:\Python26\lib\urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python26\lib\urllib2.py", line 389, in open
    response = self._open(req, data)
  File "C:\Python26\lib\urllib2.py", line 407, in _open
    '_open', req)
  File "C:\Python26\lib\urllib2.py", line 367, in _call_chain
    result = func(*args)
  File "C:\Python26\lib\urllib2.py", line 1140, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Python26\lib\urllib2.py", line 1115, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

Update 1: Proxies With Urllib2

After prompting the user for their proxy server, you'll probably want to do something like this:

# Example: proxies is a dict
proxies = { 'http': 'http://proxy.domain.com:80' } 
proxy_support = urllib2.ProxyHandler(proxies)
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)
urllib2.urlopen(url)

This is untested, but it should work more or less as written.

Update 2: If No Proxy

It may be helpful to note further that you can submit an empty proxies dict in the above code if the internet connection is direct:

proxies = {}

This way, you can use the same code to connect to the URL whether or not the user is behind a proxy server.

Update 3: hnsh.py on GitHub

Scott Jackson, author of hnsh.py, moved the code into git and posted the repository on GitHub:

http://github.com/scottjacksonx/hnsh

Check this out from the source code:

Special thanks to Ryan McGreal (http://github.com/quandyfactory) for the code that makes hnsh work from behind a proxy.

flushes with pride