Software as a Model for Government
The way software developers approach software may be instructive in helping us to think more constructively about politics and governance.
By Ryan McGreal
Posted October 31, 2009 in Blog (Last Updated November 19, 2009)
I make my living as a programmer, meaning I get paid to create and maintain software applications. At the same time I have a keen interest (my close friends might say an excessive interest) in politics and governance.
Between the two domains I've observed what appear to be some very strong parallels and analogies, and it may be that some principles that apply profitably to software could also be made to apply profitably to governance.
Of course it's possible that I've merely fallen prey to the Hammer Fallacy, but I really do think that the way software developers approach software may be instructive in helping us to think more constructively about politics and governance.
The first part of this essay attempts to explain, in lay terms, what programming actually entails. I'm adding it because no one among my non-programming friends seems to have a clue what I actually do for a living, and the analogy I'm trying to articulate kind of falls flat without this context.
If you're already a programmer and want to skip past your job description, please feel free to jump directly to the second part.
There is quite a lot to the process of developing and maintaining software, but I'd argue that the most important tasks are:
Fixing bugs - Certain assumptions go into developing features for a web application, and those assumptions tend to break down in edge cases, especially when software must work in a wide variety of different systems and environments. Fixing bugs generally entails adding logic to a function to handle those exceptional cases once they manifest.
Improving the feature set - After using software for a while, it often becomes clear that it would be easier to use and/or more powerful if it had additional useful features or didn't have existing useless features. This is why it's a good idea for software developers to eat their own dogfood by using the tools they build and maintain.
Improving the user interface - It doesn't matter how many great features an application has if users can't find them. A good user interface is clean, straightforward, easy to use, and visually atractive. Known features are easy to find, unknown features are easy discover, and superfluous features don't get in the user's way (I'm talking to you, bossy modal dialog windows!).
Optimizing performance - Applications process data, and processing data uses both CPU capacity and RAM. An application that uses too much capacity starts to run slowly, can become unstable and can even interfere with the smooth functioning of other applications and processes that also use CPU cycles and RAM. A significant part of application development involves finding more efficient ways of processing data so that a given function requires less processing power and memory to run.
There's room for debate over how best to prioritize these tasks - for example, does adding a really important feature take precedence over fixing a relatively unimportant bug - but taken as a whole, they make up the bulk of a programmer's workload.
Maintaining the software code in response to all these pressures - bugs, feature requests, difficulties in using the software, performance slowdowns, and so on - tends to cause the code to grow bigger and more complex over time.
This is especially true for applications with lots of users that must work in a wide variety of different computing environments, since there are a lot more exceptional cases that need to be taken into account.
As a function becomes more and more patched with extra logic to handle exceptions, it becomes harder and harder for developers to understand just what the code does.
Once the code is too complicated to understand, a patch to fix a new bug might break an old patch or introduce yet another bug - often in subtle ways that don't become apparent until later.
The best way to address this is to refactor the code on an ongoing basis. Refactoring changes the internal logic of a software function or method without changing what the code does.
This can take the form of replacing several similar patches scattered across the codebase with a single function that handles all of them. That way, there is only one point of failure and future bugs are easier to fix.
It might also entail removing old patches that no longer do anything - but this must be done with care and extensive testing to ensure that removing an obscure patch doesn't re-introduce an old bug.
There's also a significant danger of what's called feature creep when adding features: while a given feature on its own can be useful, each additional feature marginally increases the cognitive overhead involved in using an application.
Taken as a whole, an excess of features can be an impediment to usability as it becomes difficult for users to find the individual features they're looking for among the set of all features.
As a result, improving the feature set sometimes involves reducing the complexity of the application by merging similar features to reduce overlap and duplication, or else deprecating or removing features that turn out to be unnecessary.
This can be a real challenge because, as Joel Spolsky reminds us, "80% of the people use 20% of the features. ... Unfortunately, it's never the same 20%."
It's a truism that programmers love to write code but hate to read it. Put differently, programmers love to create new applications but hate to fix existing ones.
That's because code is a lot harder to read than it is to write - especially code that was originally written by someone else. There isn't much by way of universal best practices for programming, so individual programmers and programming departments come up with their own best practices so they can at least standardize what they do.
Sometimes the community around a given programming language will develop a style guide for that language (e.g. this style guide for the Python programming language). Some languages themselves are designed to encourage standardized coding techniques, with varying levels of success.
Still, it's tempting to stare in dismay at a legacy application that looks like a big ball of mud and spit and decide that it's better just to wipe the slate clean and start from scratch.
This is generally a bad idea. As Spolsky explains:
The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. There's nothing wrong with it. [...]
Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.
This is broadly true, but it overstates the case for old code to claim there's nothing wrong with it. In addition to the battle-tested patches and bug fixes, software can also acquire cruft: encrustations of useless stuff that takes up space, obscures what the code does, and slows down execution.
Maybe a piece of code once did something important but is no longer needed. Maybe it's there only because some other piece of code expects it to be there an will throw an error if it is removed. Maybe the programmers can't figure out why the code is there but are afraid to remove it in case it breaks something.
It doesn't help that bugs are time-sensitive, and bug fixes are often written more in the interest of expediency than long-term maintainability. As a result, they tend to acquire an ad-hoc, slapped together quality to them that can later lead to obscurity and instability.
Of course, the answer to cruft is still not to throw out the code and start from scratch. Instead, it's generally better to fix the code a piece at a time, by refactoring slow algorithms, encapsulating similar actions, fixing sloppy architecture, standardizing naming conventions, adding helpful code comments, and so on.
It may be worth rebuilding if the benefits of a far more expressive language and a better repository of well-tested libraries and frameworks outweigh the pain of the rebuild.
My decision to rewrite the Raise the Hammer codebase from scratch in Python was based on a combination of principle and practical need.
The legacy codebase was written in a proprietary language that is no longer in active development but needs to run on a proprietary platform that may not continue to support it.
Moving over to an open source language that runs on open source platforms is a painful, one-time shift, but the advantages of powerful frameworks like web.py and SQLAlchemy effectively address many of the bugs my old code fixed - and they benefit from a larger, more active user pool.
In effect, I'm rebuilding but not from scratch, since I now get to use a variety of well-designed, well-tested open source tools. If I were rebuilding the site but only using code I'd written myself (the dreaded Not Invented Here syndrome), Spolsky's warning would apply more forcefully.
If you've read through Part 1, you may already be starting to see the analogy between software and government.
The Hamilton Spectator reported last month that the City of Hamilton has "probably over 40,000 bylaws" on the books, the combined legacy of the city's long history and its recent amalgamation with five neighbouring municipalities.
That's a crufty codebase just crying out for a major refactoring.
The important thing, as Spolsky reminds us, is to optimize the performance and feature set without throwing out the accumulated knowledge and experience that is embodied in that corpus of legislation.
But politicians, like programmers, enjoy creating legislation more than they enjoy maintaining it. They enjoy writing laws more than they enjoy reading them. They enjoy adding features more than they enjoy fixing bugs.
Their general response to the relentless clamour of their user base (citizens) for more functionality is to succumb to a paralyzing feature creep that renders the legal system too complicated for almost anyone to understand.
Ideological politicians love to wipe the slate clean and start fresh. Parties in opposition promise that if elected, they will fix the problems that plague the current government in power.
When they actually get into power, they discover, like their predecessors, that the system is so complex as to be intractable and that the details are enough to bog down even the most action-oriented developer.
Some programmers adhere so rigidly to a given ideology (Agile development! Waterfall development! Functional Programming! Object-oriented programming!) that they force-map every problem to fit the constraints of their ideology.
Of course politicians and political parties are no different. Some are so dogmatic in their support of a given ideology, be it small-government conservatism or libertarianism or socialism, that they ignore the complexity in existing legislation and wipe the slate anyway to replace it with their own doctrinaire solutions.
The result is usually an excruciating failure.
To this day, for example, the Province of Ontario is still recovering from the epic fustercluck that followed from the Harris government's thoughtless political decisions, from slashing features across the board (Municipal water inspection? No one uses that) to ill-considered architecture changes (Let's push social services out to the city and upload education, how bad can it be?).
If the Harris government had been committed to a responsible process of law refactoring instead of their scorched earth policy, they might have been able to eliminate real waste without throwing out valuable code that caused all kinds of crises once it was removed.
As I've tried to argue, reading, fixing and refactoring code is hard. It's easier not to do it than to do it.
Governments sweep into power with grandiose plans to fix up the codebase, add in those features everyone's been asking for, fix its performance problems (Eight weeks for a passport? What gives?), and make government services easier to understand and use.
After a few years at the helm, however, that motivation starts to run out. Instead of transitioning to a slower, more methodical approach that respects history and carefully tests changes to make sure other features don't break, the government simply reverts to standby mode.
It all stops: feature development, bug fixes, performance improvements. The code maintainers are getting paid to do nothing, and the users start to get frustrated.
In the meantime, the party in opposition has forgotten its own prior stints in power and insists that it could do a better job of cutting out the waste, fixing the problems and introducing those great new features everyone's been asking for.
Eventually voters get so tired of the party in power that they toss them out and give the opposition a chance - and the cycle starts again when the new government's rhetoric hits the reality of the legacy codebase they're stuck working with.
Good software development entails: listening closely to your user base; fixing bugs in a timely fashion; refactoring code to work more effectively every time you update it; exhaustively testing your changes to make sure you don't accidentally introduce new bugs; making sure the user interface is clean, attractive and easy to use; and following demonstrated best practices where they're available - including making use of frameworks and libraries of proven tools.
I propose that good governance entails the same approach: listening closely to voters; fixing problems when they present themselves; incrementally improving the effectiveness and efficiency of existing government processes; introducing new features where they add real value; merging similar government functions where it makes sense to do so; changing or removing government programs that don't do anything and redeploying those resources elsewhere; ensuring that government services are accessible and easy to use; and following established best practices where available to take advantage of the accumulated knowledge embodied in shared code.
Thanks to Adrian Duyzer for reviewing a draft of this essay