Quandy Factory Newsfeed - All

Solitaire

2013-05-08T12:00:00Z

This is a version of solitaire I hacked up using HTML, CSS, javascript with jQuery, jQUery-UI draggable, the Noisy jQuery plugin, and only one small image. It's pretty nasty, but I wanted to get a feel for the draggable jquery-UI functionality.

Demo: http://quandyfactory.com/static/solitaire/solitaire.html

Source: https://github.com/quandyfactory/Solitaire

Update: You can play this game live, courtesy of PythonAnywhere:

https://www.pythonanywhere.com/gists/459573/rps.py/ipython2/

Why Bloomberg is Wrong About Terrorism Response

2013-04-23T12:00:00Z

There's quite a bit to like about New York City Mayor Mike Bloomberg, especially if you like walking, cycling and living in a lively city. But at the same time, Bloomberg's inclination to technocratic paternalism has a dark side that surfaces from time to time.

In the wake of the Boston Marathon bombings, Bloomberg warned that the country's "interpretation of the Constitution" will "have to change".

"Look, we live in a very dangerous world. We know there are people who want to take away our freedoms. New Yorkers probably know that as much if not more than anybody else after the terrible tragedy of 9/11," he said.

"We have to understand that in the world going forward, we're going to have more cameras and that kind of stuff. That's good in some sense, but it's different from what we are used to," he said.

There's a glaring problem with Bloomberg's assessment of both the problem and the solution: the world is no more dangerous today than it was when the US Constitution was written.

In fact, by every conceivable measure - including violence of all kinds - it is vastly less dangerous today than just about any time in history.

More to the point, most of the danger we do face today does not come from terrorists. Terrorists are not as interested in taking lives as they are in capturing the public attention - more specifically, the public's panicky attention.

Humans are notoriously subject to a cognitive bias called the availability heuristic, by which we evaluate how likely or common something is by how easily we can call examples of it to memory.

Terrorism plays on this cognitive bias because terrorist attacks are extremely dramatic and high-profile, and thus imprint themselves strongly on our memory.

When the news media, politicians and commentators pile on by focusing disproportionately on such events, they can trigger an availability cascade - a self-reinforcing cycle in which more focus on an event motivates people to think more about it, which in turn leads people to focus more on it.

Reactions like Bloomberg's, in which he argues that Americans need to reinterpret their Constitution to accept a loss of rights in the face of the threat of terrorism, are precisely the wrong way to approach the matter.

There are a few important facts about terrorism that should put our fear in context:

Muslim-American terrorism has been in steady decline for the past decade. The risk is getting smaller, not bigger.
More generally, terrorist organizations tend to collapse over time, through various combinations of frustration, attrition and overreach.
Terrorists themselves tend to be inept, amateurish and incompetent.
The preferred methods of terrorism - like homemade IEDs built from pressure cookers - tend to be highly ineffective at killing people.

On the same day that three people were killed in the Boston Marathon bombings, approximately 40 Americans country-wide were murdered for various non-terrorist reasons.

Also on the same day, approximately 140 Americans died via guns (this includes suicides, which make up two-thirds of the total, and accidental deaths).

We can keep going. On an average day in the United States, approximately 85 people are killed in automobile accidents, 16 people are killed in falls, 14 people are killed from accidental poisoning, and ten people are killed in home fires.

My point is that the fear that we feel about terrorism is vastly disproportionate to the risk that we will experience it - especially compared to the kinds of things we rarely think about but really should worry about.

Bruce Schneier put it best in a 2005 essay when he wrote:

If a risk is in the news, then it's probably not worth worrying about. When something is no longer reported -- automobile deaths, domestic violence -- when it's so common that it's not news, then you should start worrying.

By no means do I mean to dismiss or demean or make light of the trauma, injury and death experienced by the people who experienced the Boston Marathon terror attack.

However, it will only compound the tragedy if we allow such an event to transform our civilization from one that esteems liberty, autonomy and privacy into one that sacrifices those things for needless and counterproductive expedience.

The appropriate way to respond to terrorism is the same as it has always been: through effective, targeted law enforcement measures that build on good community relationships and do not violate the civil liberties of innocent citizens.

Adria Richards and the Allocation of Outrage

2013-03-28T12:00:00Z

One of the most glaring observations in light of the Adria Richards fiasco has been the allocation of outrage by the tech community:

The so-called "discussion" about whether Richards did the right thing when she tweeted a photo of two Python Conference attendees telling sexual jokes in violation of the event's Code of Conduct hasn't been much of a debate.

Rather, it has been a general piling-on with a generous helping of condescension and chauvinism. There has been only a smattering of disagreement and it has been summarily dismissed with a combination of the most perfunctory hand-waving and sheer shouting-down.

Meanwhile, the avalanche of rape threats, death threats, insults, intimidation, mockery and so on get by with, at best, a passing and semi-apologetic mention: Of course death threats are wrong, but...

Worse, the tech community immediately and decisively latched onto a convenient portrait of Richards as an underhanded, incompetent, cowardly, race-baiting, gender-baiting fraud with a manipulative agenda to aggrandize herself at the expense of everyone around her. This, despite almost none of the participants to this condemnation knowing anything about her.

It was a classic availability cascade, with each iteration of outraged commentary boiling the portrait into a cruder and uglier pastiche.

No amount of counter-evidence could make a dent in this depiction - the very same depiction, incidentally, that is immediately slapped onto every woman who dares to speak up about gender issues in the tech industry:

A woman in the tech community identified people violating the stated Code of Conduct of the group. She was summarily run out of the community. Oh, wait, that wasn't just this week, that was six years ago.

A woman in the tech community takes her blog offline and stops speaking publicly after receiving death threats for a month. That was also six years ago.

A man attending a festival for the tech community harassed and attempted sexual assault on multiple women in attendance. That was three years ago.

A man attending a high-profile invite-only tech event groped and harassed multiple women in attendance. That was also three years ago.

A very high-profile man in the tech community is arrested for multiple counts of sexual assault. The tech community assumes loudly and repeatedly that the women reporting the assaults are lying. Again, this was three years ago.

A woman representing her employer at a large tech event was physically assaulted by a man attending the event. That was two years ago.

A sponsored hackathon lists "friendly (female) event staff" delivering beer to participants as a "great perk" of participating. That was last year.

A prominent man in the tech community was hired by a large computer manufacturer to be its master of ceremonies at a customer summit, where he said things like, "Men have invented everything worthwhile. All we can thank women for is the rolling pin." That was also last year.

A woman who produces online feminist educational content ran a Kickstarter campaign to examine tropes about women in video games. In response, avid gamers sent her rape and death threats, vandalized her Wikipedia page, and created a game that allowed the player to "beat up" the woman's image. Again, this was only last year.

Everyone circulated and referred to a post by Amanda Blum that started with the statement, "I don't like Adria Richards," called her a "bully" and suggested that her action was part of a pattern of creating drama about gender issues to drive traffic to her blog.

The same people mostly ignored Gayle Laakmann McDowell's detailed review of the instances Blum cited that cast them in a far more reasonable and less sinister light. But that detailed analysis got little play because it didn't fit into the convenient caricature.

Also ignored was an article by Sarah Milstein in which she called Richards a "model speaker, seeking and taking feedback on her presentation, showing flexibility in the logistics around her talks, and on one occasion, giving a high-pressure talk as she was still recovering from a bout of food poisoning".

No matter how you try to parse it, it's unambiguously clear that our community whipped itself into an apoplectic firestorm of outrage when a woman complained about behaviour in a conference that violated the conference's code of conduct, but at best shrugged and at worst said she deserved it when that very firestorm of outrage overflowed into a forward panic of the most vile, vitriolic and disgusting abuse, insults, threats and mockery against her.

That says something most disturbing about our community, and the collective unwillingness even to acknowledge it, let alone come to terms with it, should be all the evidence we need that this industry's blind spot on gender issues is as pervasive as ever.

Misogyny in the Tech Industry

2013-03-25T12:00:00Z

The tech industry has spoken loud and clear:

If a woman sees, hears or experiences inappropriate and/or sexist behaviour from her colleagues and/or peers, she should keep her mouth shut and not make such a big deal out of it.

But if a man is called out for inappropriate and/or sexist behaviour, everyone will rise up in a vast eruption of righteous indignation - against the woman who had the nerve to speak up.

She will be insulted, abused and threatened with assault and rape. Her motives will be subjected to the ugliest, most damning judgments. Her history will be dragged over for anything that makes her look hypocritical or hysterical.

Her employer will readily sacrifice her to assuage the angry mob that launched a denial of service attack on its servers.

And far too many people who may not actually be deliberately misogynist will nevertheless pile on, blaming her for 'getting one of the developers fired' and providing cover for the vicious personal attacks against her.

Today I am ashamed to work in this industry.

https://www.facebook.com/SendGrid/posts/10151502570463967

http://blog.sendgrid.com/a-difficult-situation/

Background:

http://butyoureagirl.com/14015/forking-and-dongle-jokes-dont-belong-at-tech-conferences/

Update 2013-03-21 9:07 PM:

In case you're wondering what I'm talking about, here is just a recent sample of tweets that come up in a realtime search. I didn't have to dig far - at all, really - to find them. These are literally from the past 20 minutes.

Warning: these are extremely ugly, mean-spirited, dismissive and offensive. Some of them use the most degrading words the English language can hurl at a woman.

Update 2013-03-25 6:34 AM:

Some excellent write-ups around the web:

Courtney Stanton, A Woman Walks Into A Tech Conference
Sarah Milstein, I Have a Few Things to Say About Adria
Gayle Laakmann McDowell, Digging Beneath the Surface: That Amanda Blum Article on Adria Richards is Not What It Seems
Matt LeMay, On PyCon

On Crimes and Punishments: The Cruel and Unusual Threat against Aaron Swartz

2013-01-18T12:00:00Z

In the aftermath of Aaron Swartz' tragic suicide, I've been finding myself thinking that it's time for American lawmakers and prosecutors to re-read Cesare Beccaria's Of Crimes and Punishments.

After its publication in 1764, the framers of the US Constitution were persuaded by its reasoning to amend the Constitution to include a prohibition on "excessive fines" and "cruel and unusual punishments".

It's important to note that "cruel and unusual" did not mean that such punishments were uncommon. At the time the book was written, every European civilization still routinely used the most barbaric, sadistic and horrific methods for the most appalling ends: torturing suspected criminals until they confessed their crimes, and inflicting appalling physical and psychological torments on the convicted.

It's easy to forget, today, that Europe once made widespread, unremarked use of flogging, stocks, breaking on the wheel, burning at the stake, the heretic's fork, knee splitters, the rack, the thumbscrew, and various other sadistic forms of punishment for a staggering array of crimes, including both victimless offenses like blasphemy and imaginary offenses like witchcraft.

When Beccaria decried such practices, he argued not that they were uncommon - after all, it was their very commonness to which he objected - but rather that they were offensive to contemporary Enlightenment ideas about fairness, civility and efficacy in the administration of justice.

In the case of Aaron Swartz, the threat of 50 years in federal prison and a million dollars in fines for downloading a bunch of academic papers cannot reasonably be considered to be anything other than "excessive" or "cruel and unusual".

Apologists for the US prosecutors who brought the case against him are arguing that he was offered a plea bargain that would likely result in only six months of prison time.

Carmen M. Ortiz, the US Attorney for Massachusetts who brought the case against Swartz, issued a public statement yesterday that defended her office's actions on the grounds that the plea bargain offered "sought an appropriate sentence that matched the alleged conduct."

Ortiz insisted, "At no time did this office ever seek - or ever tell Mr. Swartz's attorneys that it intended to seek - maximum penalties under the law."

However, this spectacularly misses the point: Swartz would only be entitled to a reasonable expectation of leniancy in exchange for pleading guilty.

I will set aside the contention that even a day of time in a federal prison is disproportionate for what amounts to a TOS violation, and instead focus on the fact that a short prison term would only come with a guilty plea.

The prosecutor used the threat of 35-50 years in prison to get Aaron to accept a guilty plea. If he chose to plead not guilty and attempt to defend himself, it would cost well over a million dollars in legal fees, and if he were found guilty anyway, the judge would be strongly predisposed to make an example of him by giving him a very long, punitive sentence.

A prosecutor forcing someone to admit guilt by threatening a life-destroying cruel and unusual punishment if they do not admit it - why, that is the very essence of using torture to extract a confession!

Bill C-38

2012-06-29T12:00:00Z

On June 19, 2012, I sent the following email to the members of the Canadian Senate:

Dear Senators,

If ever a piece of legislation cried out for sober second thought, it is Bill C-38: a sprawling omnibus bill that makes so many changes at once, it's almost impossible for Parliament, let alone the electorate, to get their heads around it.

It is not sound governance to replace clear, consistent rules with discretionary ministerial regulations that are susceptible to political interference and regulatory capture.

It is not prudent stewardship to cut back the Fisheries Act so that it only pertains to commercial fish stocks. If the collapse of the Atlantic cod industry has taught us anything, it is that a commercial stock cannot survive without the broader ecosystem that supports it.

It is profoundly anti-democratic for a government to attempt to impose a chill on non-profit organizations that challenge its agenda. It is only through vigorous, open debate that the best public policy can prevail.

It sets a dangerous precedent to grant the Immigration Minister the power to overrule national immigration policy and reject applicants to Canada whose applications meet the rules. Will we be governed by the rule of law or by the caprice of de facto letters patent?

Similarly, it is dangerous and capricious to grant the government the power to overrule the rulings of the National Energy Board in order to expedite energy and resource projects. Combined with the narrowing of the mandate of environmental assessments and the exclusion of non-business stakeholders, this all but guarantees that resource extraction projects will be determined on the basis of narrow private interest, not the broader public interest.

Even this broad swath of policy implications barely scratches the surface of Bill C-38, which crams so many changes into one package that those changes cannot be properly evaluated. Certainly they do not belong in a bill that is supposed to implement the annual budget!

As one member of the House of Commons noted:

We can agree with some of the measures but oppose others. How do we express our views and the views of our constituents when the matters are so diverse? Dividing the bill into several components would allow members to represent views of their constituents on each of the different components in the bill.

The bill contains many distinct proposals and principles and asking members to provide simple answers to such complex questions is in contradiction to the conventions and practices of the House.

That member was Stephen Harper, and his prudent appraisal of a 2005 omnibus bill is as true today.

Just today, I received a reply from Senator James S. Cowan, Leader of the Opposition in the Senate:

Dear Mr. McGreal:

Thank you for your email about Bill C-38, the Harper Government's omnibus Budget Bill.

I agree with you - the bill is highly objectionable. I too have concerns both about the substance of the bill, and the abuse of our democratic traditions in the way the bill was rushed through Parliament.

The Bill is over 420 pages long and contains over 720 clauses. It amends or repeals some 70 statutes, including as you mentioned, Canada's environmental assessment laws and the laws that protected fish habitat. It dismantles the National Round Table on the Environment and the Economy, the National Council on Welfare, Rights and Democracy, the Public Appointments Commission, the Inspector-General that acts as a check on CSIS - the list goes on and on. It makes extensive changes to Canada's social safety net, including Employment Insurance and Old Age Security.

These - and many other provisions in the bill - go far beyond what should be contained in a single bill, and are flatly bad public policy.

The Conservatives have rammed it through Parliament, with as little possibility for scrutiny or input from Canadians as possible. In the House of Commons, the entire bill was sent to a single committee, the Standing Committee on Finance - and it had to deal with the entire package in 10 sitting days. Frankly, the changes to Old Age Security or Employment Insurance alone would have deserved that attention.

Here in the Senate, the Liberal opposition was able to get the government to agree to send the subject matter of the bill to six different Senate committees for "pre-study", but even this, while better than the cursory study possible in the House of Commons, was simply inadequate to properly examine this mammoth bill.

The Conservatives have repeatedly shut down debate and study of this bill, and this is continuing now that the Bill is before the Senate.

My Liberal colleagues and I have tried to place on the record our concerns with both the substance and process. Our debate on the Bill - which was again cut short by the Conservatives - can be found at the following link:

http://www.parl.gc.ca/Content/Sen/Chamber/411/Debates/095db_2012-06-21-e.htm

Although the process in the Senate is still on-going as I write, it is clear that the Harper government has no intention of allowing real debate and study. It will continue to force the bill through to final passage, over our strong objections and those of many Canadians like yourself.

Once again, thank you for taking the time to write to me on this important issue.

If you want to send a letter to the Senate, here is a list of their email addresses.

Update: I also received a reply from Senator Elaine McCoy.

Hello, and thank you for your email about the omnibus budget bill. I just wanted to let you know that I voted against C-38, as a matter of principle, last Thursday night. A copy of my speech is attached for information. Sadly, the vote did not succeed in stopping the bill as the Conservatives overwhelmingly supported it.

You can see the results of the standing vote (as well as other speeches against the bill) by clicking here and scrolling to the section titled Jobs, Growth and Prosperity Bill.

Although you and I have not succeeded in persuading the government to follow a different course of action in this instance, I feel it is important we continue to speak out about issues of importance to Canada. I know I will. I hope you do too.

Warm regards, Elaine

Alberta Senator

The Appification of Computing

2012-06-28T12:00:00Z

The historian and philosopher Walter J. Ong drew a distinction between what he called "craft literacy" and "social literacy". The former is the kind of literacy you find in cultures that have systems of writing but do not have cheap, ubiquitous publishing - like Europe before Gutenberg's invention of movable type.

In a culture with craft literacy, most people can't read and write and those who can - scribes - do so as a vocation.

When a society moves toward full literacy, made possible by universal access to both reading and writing material, a number of huge social transformations occur - but one such change is that the mere ability to read and write stops being a rare skill, possessed only by professionals.

For some time I've believed that computer literacy - by which I mean the ability to read and write computer programs - is currently at the level of craft literacy in our society.

I've expressed the hope that we will eventually move toward social literacy in computing, in which most people routinely learn to read and write computer programs in the same way we currently learn to read and write text.

Computer Illiteracy

So a recent article in the Globe and Mail gave me serious pause. Titled, "Are we breeding a generation of app-living, web-addicted digital illiterates?", the article quotes Sang-Jin Bae, a computer animation technical director and instructor at New York University's Tisch School of the Arts:

"When kids come into my class they divide into three groups," he says. There are the pure geeks who love technology. There are those trying to understand. And then there is the biggest group: "Those who couldn't care less."

As remarkable as it is to consider, this hip, articulate 36-year old computer whiz makes a heck of an argument that the computer age is entering a dark new era: the age of the digital illiterate.

Today's teens grew up on SMS and Facebook. Everything is being presented to them all the time. Web companies love it, since kids are addicted to their products. But, he says, "They expect less and less from the Web and the software they use."

Mr. Bae is not just talking about obscure, high-end animation tools. Instead, he sees an essential dumbing down of bedrock computing skills.

"The kids I have, and that is roughly two dozen of the brightest young digital artists a semester, often have no idea what Microsoft Word is. They can't tell a Mac from a PC. And forget Excel," he says. He struggles to get his students to use basic computing etiquette.

It appears that the relentless drive to simplicity in user interface has had the-side effect of serving as a disincentive for students to bother learning more about how computers work.

Knowledge-Destroying Idea

Lately I've been reading Edward Glaeser's book Triumph of the City, a sprawling tour-de-force in support of the thesis that cities are our most important engines of creativity, innovation and growth and that we would do well to understand how cities work so that we can run our cities more effectively.

Writing about the spectacular decline of Detroit, formerly one of America's greatest cities, Glaeser argues that the seed of Detroit's failure was the invention of the assembly line:

By turning a human being into a cog in a vast industrial enterprise, [Henry] Ford made it possible to be highly productive without having to know all that much. But if people need to know less, they also have less need for cities that spread knowledge. When a city creates a powerful enough knowledge-destroying idea, it sets itself up for self-destruction.

I wonder if the 'app-ification' of computing is turning out to be the same thing: a powerful, knowledge-destroying idea that is actually crippling our collective ability to use computers as tools of creation rather than merely as vectors of consumption.

Update: interesting discussion on Hacker News.

My Second Baguette

2012-04-30T12:00:00Z

Last summer, during a grueling heat wave, I decided to try my hand at making baguettes. The results were decent for a first try but left plenty of room for improvement, but life conspired to get in the way of a follow-up effort.

Finally, this past weekend, I prepared a second batch and applied the lessons I took from the first attempt. I'm happy to report that the results were definitely more impressive.

What You'll Need

Every decent baguette recipe I found measured the ingredients by weight, not by volume, so make sure you have a decent kitchen scale. Here's a full list of equipment and tools:

Kitchen scale
Medium-size bowl
Large bowl
Wooden spoon
Sharp knife
Baker's couche or, failing that, a big, thick tea-towel and some wax paper
Cellophane
Large baking sheet
Kettle for boiling water
Medium-sized baking pan (e.g. for brownies or lasagne)

Poolish

A day before you intend to make your baguette, you first need to prepare a poolish, or a pre-ferment.

Ingredients

300 g. flour
300 g. warm water
1/8 tsp dry active yeast

Directions

Combine the ingredients thoroughly in a medium-sized bowl.
Cover with a cloth and let sit overnight in a cool room (20-22 degrees Celsius is optimal).

Combine flour, warm water and yeast to make a poolish

After fermenting, the poolish should look bubbly and almost liquid.

After fermenting overnight, the poolish should look wet and bubbly

Here's a closeup of the fermented poolish:

Closeup of fermented poolish

Baguette

Once your poolish is mature, you can proceed to making the baguette dough.

Ingredients

The poolish you just made
600 g. flour
300 g. warm water
1 Tbsp. salt
1 tsp dry active yeast

Directions

Combine all the ingredients in a large bowl using the wooden spoon. Don't worry - there's enough water even if it doesn't seem that way at first.
Knead the dough for 15 minutes. You might need to add bits of flour to stop it from sticking to your hands.
Form the dough into a ball. Rub it with oil and cover it with a cloth.
Let it rise for around 2 hours, or until doubled in size.

Dough in the process of rising

Once the dough is ready, cut it into four equal-sized pieces and shape each piece into a rough circle.
Cover each piece with cellophane and let them rest for 20-30 minutes.

Dough quartered, shaped into balls and wrapped in cellophane

Next, shape the four dough pieces into baguette shapes.
Fold the couche or cloth into four folds, one for each baguette, so that it holds its elongated shape during the final rising period. Note: if you're using a cloth in place of a proper couche, cover the cloth in wax paper so the dough does not stick.

Dough shaped into baguettes and folded into a makeshift couche

Cover and let rise for another around 1.5 hours.
Now preheat the oven to 450 degrees fahrenheit.
While the oven is heating, boil some water in a kettle. When the oven is ready, pour most of the water into your pan and place it on the bottom tray of the oven.
Transfer the baguettes from the couche onto a large baking sheet.
Using a sharp knife, cut several shallow, slightly diagonal scores across the top of the baguettes.

Baguettes placed on the baking sheet and scored with a knife

By now, the oven should be back up to 450 degrees and the water in the pan should be boiling energetically. Place the baking sheet with the baguettes onto the top tray of the oven.
With the rest of the boiling water, splash it around the interior bottom and sides of the oven to get some good steam going, and then quickly close the door.
Bake for around 20 minutes, and remove from the oven once the baguettes are golden brown.

Out of the oven: slightly overdone but still tasty

Closeup of the baked baguettes

As you can see, my baguettes turned out slightly overcooked (I should have taken them out a minute or so sooner), but they were delicious: crunchy on the outside, soft and chewy on the inside, with a nice baguetty flavour. We served them as part of a French-themed dinner on Saturday night and they were well received.

Services-First: A Better Way to Build a Web Application

2012-02-22T12:00:00Z

Introduction

The best way to find yourself is to lose yourself in the service of others.

-- Mahatma Gandhi

When you're building a web application, it's a powerful design heuristic to develop your functionality as a web service and then build your application on top of your service.

The functionality your application needs to perform should all be accessible or at least exposable as web services. This enforces a responsible design approach that will pay real dividends in maintainability, extendability and reduced technical debt.

It may also pay unexpected dividends in generating value over and above the principal goal of your web application. By thinking about your application design as a web service, you open up the possibility that the real value you deliver is not what you thought your service was going to be, but some subsidiary or extraneous solution you develop that turns out to be a) useful to others and b) scalable/profitable.

The Infamous Yegge Platform Rant

Last October, Google engineer Steve Yegge posted a long, thoughtful rant on his public Google+ page in which he argued that Google "does everything right" - except platforms, which he argues Google still doesn't really understand. Rather, Google provides products, not services.

Yegge's essay was directed internally at his fellow Googlers, and he took it down after realizing he had accidentally posted it publicly. Luckily for us, copies are still available with Yegge's blessing.

Yegge contrasted his former employer, Amazon, which overwhelmingly adopted a services-first approach after founder and CEO Jeff Bezos ordered every business unit in the company to start exposing its functions to the rest of the company as a web service.

Service-Oriented Architecture (SOA)

SOA is a methodology for developing software as a collection of services. Every functional unit exposes its functionality via API calls, and each unit consumes other units' services by accessing those API calls. The services are modular, exposable and 'mashable'.

Platforms

According to Yegge, Amazon recognized that a services-first approach would enable Amazon to provide a platform on which other companies can build their businesses.

[T]he first big thing Bezos realized is that the infrastructure they'd built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform.

A platform is a system on which others can build applications and products. A product can be used, but a platform can be programmed and *extended. Users can create, access, modify and share resources programmatically.

Outside developers can build applications on top of your platform. The more accessible your platform is, the more people will use it and the easier it will be to achieve critical mass and positive network externalities.

Metcalf's Law states: The value of a network is proportional to the square of the number of connected nodes. As the number of people using your platform increases linearly, the overall value can increase geometrically.

Applications

Applications are programs built on top of platforms that are meant to be used directly.

A good platform is hackable, in the sense that it is general and flexible enough to allow third parties to build applications that the platform designer might never have imagined.

The more powerful, reliable, accessible and flexible a platform is, the more likely it is that someone will build a "killer app" - a program so useful that it makes the platform indispensable. Think of VisiCalc on the Apple II, Lotus 1-2-3 on the PC, or Microsoft Office on Windows.

Electricity Grid

Here's another platform that attracted killer apps: the electricity grid. When the grid was first built, it was dedicated to powering lights. There weren't even outlets: electrical lines were hard-wired to light bulbs.

But it didn't take long for innovators to recognize that there were lots of other uses for electricity. Within a few years, electric fans, irons and toasters became extremely popular.

The first appliances didn't have plugs: they had connectors that screwed right into light bulbs. When outlets were introduced, they provided a much safer, more flexible user interface for the power grid.

To this day, innovators are still finding new and creative ways to build on the electricity grid. It's a beautiful platform: about as powerful, reliable, general and flexible as it gets.

Bootstrapping

When you build a robust platform, you make it easier and more attractive for developers to build on it. The more developers you have building on your platform, the more likely it is that one of those developers will build a killer app. More, better and more popular apps, in turn, make your platform more attractive to developers.

It's a virtuous cycle if you can manage to set it in motion.

Dogfooding.

Yegge stresses the vital importance of building on your own platform:

The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything."

When you dogfood your platform, you have a direct interest in improving it, and that improvement makes it more attractive to developers. You also get to see your platform from your customers' perspective, helping you to understand their needs better. Finally, you demonstrate confidence in your own offerings.

Microsoft built an extremely successful software business through religious dogfooding. The original Windows NT team actually developed NT using computers that ran NT - talk about an incentive to build a stable, functional platform!

A word of warning: taken to the extreme, dogfooding leads to Not Invented Here syndrome.

Products v. Platforms

Products	Platforms
Targeted	Broad
Differentiated	Integrated
Fast to Market	Long-term Strategy
Maximize profit	Maximize Market Share

The danger of a business model based on a product offering is that a powerful platform can surround and crowd out a product if the platform allows equivalent functionality. Since computers became widespread, the market for typewriters and calculators has collapsed.

In general, an extensible platform will tend to win over a monolithic product. iPhone and Android crowded out Blackberry in large part because they made it easy for third party developers to extend their functionality.

Similarly, a more open platform will tend to win over a more closed platform. As much as iPhone was more open and accessible than BB, Android is still more open and flexible - even to the point that the operating system can be installed on a variety of third party devices. Its market share has shot ahead of the iPhone.

This is the same story that played out in the early 1980s: Microsoft won the PC operating system market mainly because of its cheap, permissive licencing model. They extended this model to Windows and the Win32 API, which was similarly more open than Mac, Amiga and other systems that were actually technically superior but more closed.

In the 1990s, the World Wide Web crowded out and assimilated Compuserve, AOL, The Source, Prodigy, Minitel and other proprietary networks because it was an open standard with a low barrier to entry.

The story of web servers is the same: the LAMP stack (Linux, Apache, MySQL, Perl/PHP) came to dominate web applications against competition from the proprietary IIS/ASP and .NET alternatives.

Products to Platforms

Remember, however, that the line between products and platforms is fuzzy. Products use standardized, modular components; design/manufacturing/logistic expertise transfers to new products; and products can generalize into platforms.

Amazon and Facebook both transformed themselves from product companies to platform companies. Today, the killer app on Amazon Web Services is Amazon, and the killer app on the Facebook platform is Facebook.

Yegge points out that it might be painful to maintain the discipline to eat your own dogfood and build services-first, but it is far more painful and expensive to turn a monolithic application into a platform after the fact.

If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

According to Yegge, Amazon went through the painful exercise of transforming itself from a product company into a platform company because of necessity:

[I]t took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers... if only they could be monetized somehow... you can see how he arrived at AWS, in hindsight.

Amazon Web Services

Here is how Bezos himself explained the strategy in his latest annual letter to shareholders:

Our technologies are almost exclusively implemented as services: bits of logic that encapsulate the data they operate on and provide hardened interfaces as the only way to access their functionality. This approach reduces side effects and allows services to evolve at their own pace without impacting the other components of the overall system. Service-oriented architecture - or SOA - is the fundamental building abstraction for Amazon technologies.

Amazon launched its first web service - Elastic Compute Cloud (EC2) - in 2006. Since then, it has followed up with a long list of additional services:

CloudFront - low latency global CDN
DevPay - accounts receivable service
DynamoDB and SimpleDB - schemaless nosql key/value store
ElastiCache - scalable in-memory cache
Elastic MapReduce - distributed mapreduce processing for huge datasets
Flexible Payment Service (FPS) - digital payment service
Mechanical Turk - on-demand human intelligence market
Relational Database Service (RDS) - relational database
Simple Email Service (SES) - bulk and transactional email service
Simple Notification Service (SNS) - send notifications
Simple Storage Service (S3) - data storage and retrieval
Route 53 - Domain Name System service

None of these service are central to Amazon's core business, which is selling books and other consumer products directly to consumers. However, they all began life internally as essential peripheral or supporting functions.

Today, the company can sell these web services publicly because Bezos had the foresight to insist that the company conduct its internal affairs via web services.

The company is secretive about how much revenue it earns directly from its suite of web services, but the best estimates are that revenue reached $500 million in 2010 and $750 million in 2011. Web service revenue is expected to reach $1 billion in 2012 and $2.6 billion in 2015.

Platforms Express Value

No matter what you build, you will develop expertise in a number of related, peripheral functions that support your primary product. If you develop your product in a services-first fashion, you allow for the possibility to expose those peripheral functions to third parties.

No initial product idea survives first contact with the market. Every successful business is successful because it was able to adjust - sometimes wildly - its product and business strategy in response to real-world feedback. Startup gurus call this "pivoting", and a founder's ability to pull it off is decisive in the survival of the business.

Pivoting doesn't mean throwing away what you've built and starting over. Rather, it means re-purposing what you've built to reach a more promising market.

Many successful startup products started out as components of larger, less focused products. In his book The Lean Startup, entrepreneur Eric Ries calls this the "zoom-in pivot". Other businesses started out assuming they would be selling to one market and ended up attracting interest from a different sector altogether.

A services-first approach decouples your product from the platform it's built on, allowing you more flexibility to change it as required. It also allows you to provide the platform itself to clients as an additional source of value.

Designing Your Web Application

So when building your web application, it's a powerful design heuristic to build a web service first, and then build the application on top of it.

Why a Heuristic?

Application design is not linear or deterministic, it's open-ended, incremental and adaptive. Small changes early on can push an application in vastly different directions.

Most important, there isn't one right way to do things. Ten different developers given the same problem will create ten different solutions - and any number of them can be 'good enough'.

When you can't reason your way to a proof, you have to fall back on a less certain, more exploratory approach. A heuristic is a method of approaching the problem that tends to help you produce good solutions but falls short of being a solution itself.

Think of a heuristic as a general rule of thumb rather than a specific prescription.

Design Goals

For the end user, a web service should be:

Useful - The most elegant web service in the world won't interest anyone if it doesn't solve real problems.
Usable - Similarly, the most useful web service won't be adopted if people can't figure out how to use it.
Explorable - The easier it is to navigate and discover what a service does and how to use it, the more it will be adopted.
Accessible - In the case of a web service, accessibility means a variety of clients can use and interact with it.
Combinable - The purpose of web APIs is so clients can combine data and functionality from a variety of sources to build new things.
Flexible - Your own product is not the only thing that can be built with your platform. Your API should be open enough to allow for use cases you couldn't imagine.

For the web service developer, a web service should offer and/or encourage:

Clean, consistent organization of data and functionality
Separation of different layers in the application stack
Flexibility to change the application built on it
Clarity of abstractions

Finally, the developer is also an end user and will also benefit from usefulness, usability, discoverability and so on.

Design Considerations

With these goals in mind, I propose the following design considerations. Again, since design is non-deterministic, these considerations can help you to make decisions in the open-ended design process that will tend to point you toward a better final product.

Design for Adoption

The entire purpose of an API is for people to use it. If you're not sure how to design something, use this question as a guide: What will make it easier for users to understand how this works and how to use it?

If you force yourself to be a user, you will be more inclined to look at the API from the user's perspective.

Design for Maintainability

Developers spend more time fixing, refactoring and modifying existing code than they spend creating new code. Your API should be designed in such a way that it is easy to dive in and work with existing code. One way to achieve this is through what the Rails developers call convention over configuration. An architecture that builds on established standards will be easier to maintain.

Web Service APIs

There are two basic kinds of web service:

Arbitrary Remote Procedure Calls (RPC)
Hyper-Text Transfer Protocol (HTTP)

Remote Procedure Calls

RPC is a client/server model that allows a client to execute a remote procedure as if it was a local procedure. It uses HTTP as a dumb transport tunnel, and all the objects, methods and data required to call the procedure are transferred inside the message body.

RPC is arbitrary in that the method of transferring data and methods over HTTP varies from one type to another. A number of competing RPC protocols emerged in the early days of the internet: CORBA, XML-RPC, MS-RPC, and so on.

SOAP

By the end of the 1990s, one format had emerged as a de facto standard: Simple Object Access Protocol (SOAP), which was a formal attempt to standardize XML-RPC.

A SOAP web service, all methods are accessed via a single endpoint URL. Methods and data are transferred inside an XML payload, and all requests use the HTTP POST method.

The functionality in a SOAP web service is defined in a Web Services Description List (WSDL).

SOAP Benefits

SOAP was an improvement over previous RPC approaches. Because it was more or less standardized, it allowed for push-button tooling and deployment. The WSDL allowed properly configured clients to introspect the web service's functionality and abstract away the "web" part of the transaction.

SOAP Problems

However, these benefits are not unalloyed. The XML payloads are extremely verbose and not very human-readable. The standards are arbitrary, poorly defined and have changed more or less constantly for over a decade.

Worse, the system is designed to be interoperable, but a client in, say, C# often cannot consume a WSDL built in Java. At bottom, SOAP is a leaky abstraction that promises easy tooling and wide accessibility but often fails to deliver.

But the worst problem with SOAP is that it re-invents the functionality of HTTP on top of HTTP. It tunnels through the web, but does not work like the web.

Hyper-Text Transfer Protocol (HTTP)

Often called "Representational State Transfer" (REST), an HTTP web service is a client/server model that works like the web. The client makes an HTTP request to the server, and the server sends a response back to the client.

HTTP web services are stateless: each request contains all the information needed to process it. HTTP web services are also cacheable and can define rules around which responses can be stored locally or in an intermediate server.

The concept was formalized in 2000 by Roy Fielding, one of the architects of Hypertext Transfer Protocol (HTTP), in his doctoral dissertation. It is not surprising, then, that REST and HTTP mesh very smoothly.

A RESTful client-server system is stateless, meaning each request against the server contains all the information the server needs to process it; and cacheable, in that the server can specify whether and for how long resource representations can be cached either locally on the client or on intermediate servers between the client and the server.

HTTP

Hypertext Transfer Protocol (HTTP) is a stateless protocol based on a client requesting a resource across a network and the server providing a response. As such, an HTTP transaction entails a request and a response. The request goes from the client to the server, and the response goes from the server back to the client.

HTTP Requests and Responses

In HTTP, the client makes an HTTP request to the server and the server issues an HTTP response.

Requests

An HTTP request has three parts:

The request line, which includes the HTTP method, the URL and the HTTP version: GET /users/1 HTTP/1.1
One or more optional HTTP headers, which are key/value pairs with metadata about the data being requested and/or provided.
An optional message body, which is data being sent from the client to the server as part of the request. E.g. in a POST request, the message body will include the data that the server should use to create a new resource.

Response

An HTTP response also has three parts:

The HTTP Status Code, which indicates the status of the requested resource: HTTP/1.1 200 OK
One or more optional HTTP headers, which are key/value pairs with metadata about the response being provided.
An optional message body, which is a representation of the resource that was requested (hence the name "Representational State Transfer").

URLs and Resources

In a RESTful architecture, each URL represents a resource. This is vitally important: a resource is a noun, an object, and not the action performed on it.

A RESTful web service has multiple endpoints - one for each resource. (Contrast SOAP, which has only one endpoint and puts everything else - objects, methods, parameters, etc. - into the XML payload.)

If you find yourself creating URLs like /create_user, you're doing it wrong. Instead, create a URL like /users and map your user object to that URL.

Remember: a resource is a noun, not a verb.

HTTP Methods

If a resource is a noun, the HTTP Method is the verb. What you do to the resource with your request depends on what method you use. If a URL is an object, the HTTP method is the action you execute on that object.

HTTP Methods
Method	Action
GET	Retrieve a resource
POST	Create a new resource
PUT	Update an existing resource
DELETE	Delete an existing resource

In our user example, execute an HTTP POST request on /users to create a new user. The server should respond with the specific URL of the user you created: /users/1.

To view a representation of that resource, issue an HTTP GET request on the user's URL: /users/1.

To update the user, issue an HTTP POST request on the user's URL with the new user data in the request body.

To delete the user, issue an HTTP DELETE request on the user's URL.

GET Method

To retrieve a resource, issue an HTTP GET request. GET requests are idempotent (see below), which means making a GET request multiple times does not cause any change in the resource that is requested.

GET requests do not include a message body, but GET responses usually do.

POST Method

To submit data to be processed, issue an HTTP POST request. POST requests require a message body, i.e. the data to be processed.

For example, if there is a resource called /articles and you want to add a new article, issue a POST request to /articles with the content. The server should create a new subsidiary URI under /articles - for example, /articles/1 - and assign that URL to the content you sent with your POST request.

Important note: POST requests are not idempotent, meaning multiple POST requests will create multiple resources with unique identifiers.

PUT Method

To update the resource at an existing URL, issue an HTTP PUT request on that URL. For example, if there is a URL /articles/1 and you want to replace the content served at that URL, issue a PUT request to that URL with the new content.

Important note: PUT requests are idempotent, so issuing 2 or 5 or 50 identical PUT requests will have the same effect on the resource as issuing just one PUT request. Like POST requests, PUT requests include a message body (the resource to be placed at the URL).

DELETE Method

To remove a resource (and remove its accompanying URL), issue an HTTP DELETE request. DELETE requests should be idempotent, i.e. issuing 1 or 2 or 5 or 50 identical DELETE requests will delete exactly one resource. DELETE requests do not require a message body.

Idempotence

This funny-looking word is really important: a request is idempotent if making it multiple times has the same effect as making it just once. Read that again if you have to.

A request is not idempotent if issuing it more than once has a different effect than issuing it once.

For example, a POST request to add a comment to a document is not idempotent: issuing the POST request twice adds the comment twice, so that the document contains two comments with unique URLs.

However, a PUT request to update an existing comment is idempotent: whether you execute the request once, twice or twenty times, the result will be exactly the same.

HTTP Headers

Both HTTP requests and responses can include optional HTTP Headers, or key/value pairs that supply meta-data about the request and the response.

Accept Header

An HTTP request can include an Accept header that specifies what media types the client will accept in a response.

In a REST web service, the request should include an Accept header with the preferred media type - e.g. application/json - and the server should attempt to fulfill the client's preference in its response, given its capabilities.

If you are willing or required support multiple formats (e.g. JSON, XML, YAML), the best way for the client to specify what format they prefer is via the Accept header.

Here are some examples of Accept headers:

JSON: Accept: application/json
XML: Accept: application/xml
YAML: Accept: application/x-yaml

Note: YAML has an x- prefix because it is not a formalized MIME type. For thoroughness, you could accept all four possibilities:

application/x-yaml, application/yaml, text/x-yaml, text/yaml

HTTP Status Codes

HTTP has a great way of telling the client whether the request was successful or an error has occurred: HTTP status codes. Every HTTP response includes a status header with an HTTP status code that indicates the status of the request.

This tells the client whether the request was successful or not, and what to expect by way of a response. It's important to send the right status code.

401 Unauthorized (Image Source: Flickr)

I also include HTTP status codes and error descriptions in the response body. A REST purist might call this overkill, but remember that you want to make your API as self-documenting and easy to use as possible.

For the purposes of your web service, there are three categories of HTTP status codes you need to worry about: success codes, client error codes, and server error codes.

Success Codes

HTTP success status codes provide useful information about successful requests:

200 OK - A GET request was successful at retrieving a resource.
201 Created - A POST request was successful at creating a resource.
202 Accepted - This means the request was accepted but the response is not ready. Useful for queued or otherwise deferred processes.

Client Error Codes

400 Bad Request - The request data could not be parsed properly, or a value was missing or invalid.
401 Unauthorized - The user requested an access-restricted resource but authentication either failed or was not provided.
404 Not Found - The user tried to request a resource that does not exist.
405 Method Not Allowed - User tried to execute an HTTP method on a resource that does not allow it. E.g. a POST request against /users/charlie instead of /users.
406 Not Acceptable - The request came with an Accept header for a media type that the server cannot provide in the response.
410 Gone - A previously available resource has been removed permanently.
429 Too Many Requests - Useful if you've rate-limited your API for a given user.

Server Error Codes

500 Internal Server Error - Try to avoid this one. It generally means your web service is broken.
501 Not Implemented - User has tried to send a request method that your server doesn't recognize.
503 Service Unavailable - This tells the client the service disruption is temporary.

Media Types

In an industry known for bad acronyms, this may be the worst: Hypertext As The Engine Of Application State (HATEOAS) is a mandatory requirement for a web service to meet Roy Fielding's criteria of a "RESTful" design.

Here's how Fielding explained it in a blog post that took issue with the profusion of web services that called themselves "RESTful" but clearly weren't:

A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

There, isn't that clear?

In short, Fielding's requirement is that a client should not need any out-of-band information to be able to access, navigate, use and execute a web service. All it should need to know are the base URL endpoint and the media type (identified as the Content-Type header in the response) of the web service so it knows how to interpret the responses.

In other words, if your web service has a custom way of identifying hypertext URLs in its responses, your web service must also have a dedicated media type that corresponds to how your service works and includes a description of how it works.

In other words, if the client has to read your documentation to discover how your web service works, it is not RESTful because it depends on out-of-band information (i.e. the documentation); but if your web service has a custom media type and the client needs only to read a bunch of documentation to understand how your media type works, it is RESTful because it doesn't depend on out-of-band information.

Designing a RESTful Web Service

Don't Reinvent the Wheel

HTTP was invented to allow clients to request resources from servers across the internet and for those servers to respond with representations of those resources. The entire internet runs on it. It's flexible, powerful and offers a clean abstraction model based on resources and methods.

Don't re-implement what HTTP does in an ad-hoc way on top of HTTP. Just use HTTP.

Many web services only use HTTP as a network tunneling protocol, and then pass objects, methods and parameters inside an 'envelope' that contains all the information. This type of web service architecture includes SOAP and other Remote Procedure Call (RPC) formats.

Web services that actually use HTTP the way it was designed are called Representative State Transfer (REST) web services. Unfortunately, REST is widely misunderstood and a lot of web services that call themselves "RESTful" aren't.

Discoverable API

In a RESTful web service, a GET request on a base API URL returns a list of resources available in the API. Critically, each resource has a direct URL. This makes it very easy for developers to step into your API and figure out what is available.

Now, you may be thinking you've seen this pattern before. You're right: it's exactly how every single website works. Go to the homepage, and what do you find? Links to the other resources on the website!

A RESTful approach makes your URLs hackable. If your API user is looking at a URL like: /cats/miss_mew, they should be able to lop off the /miss_mew and do a GET request against /cats to return the base collection of cats (each with its own URL, of course).

REST Resource/Method Matrix

At a conceptual level, a RESTful web service API is a matrix of resources and methods that exposes the functionality of the service to third party applications. Below is an example of what that matrix might look like.

Again, I will repeat that actions are not mapped to URLs. A resource is an object - a noun - and the action inheres to the HTTP method - a verb - not to the URL.

As a result, the same resource URL can serve different responses (corresponding with different actions) depending on the HTTP method used.

REST Resource/Method Matrix
Request				Response			Idempotent
Resource	URL	Method	Request Data	Server Action	Response Body	Status Code	Idempotent
Users	/users	GET		retrieves list of users	list of users and associated URLs	200 OK	Yes
Users	/users	POST	user details	creates a new user	new URL and user representation	201 Created	No
User 9001	/users/9001	GET		retrieves details for user 9001	user representation	200 OK	Yes
User 9001	/users/9001	PUT	new user details	updates user details	updated user representation	200 OK	Yes
User 9001	/users/9001	DELETE		deletes user 9001	returns status	200 OK	Yes
User 9001	/users/9001	GET		checks that user 9001 does not exist	Not found error message	404 Not Found	Yes

Use JSON

JavaScript Object Notation (JSON) is a data serialization format that delivers an optimal combination of the following criteria:

Lightweight - less boilerplate means less data transfered across the network
Readable - Humans can easily see and understand the content of a JSON object
Flexible - allows various nestable data types: object (dictionary), array (ordered list), string, integer, float, boolean
Interoperable - every programming language and framework under the sun can generate and consume JSON.
Convertable - JSON can be converted into a native object more easily than XML.

Here's an example of a response to an API base URL GET request, represented in JSON:

{
    "ok": true,
    "status_code": 200,
    "resources": [
        {
            "url": "/authors",
            "resource": "authors"
        },
        {
            "url": "/articles",
            "resource": "articles"
        },
        {
            "url": "/blogs",
            "resource": "blogs"
        },
        {
            "url": "/comments",
            "resource": "comments"
        },
        {
            "url": "/events",
            "resource": "events"
        }
    ]
}

Unless you've got a compelling business or technical reason for using a different media type, stick with JSON.

Just remember: if you want to satisfy HATEOAS, you need to give your web service a dedicated media type, e.g.:

application/vnd.my-ad-hoc-web-service+json

Note that the vnd prefix refers to a vendor-specific media type, e.g. application/vnd.google-earth.kml+xml for Google Earth KML files.

Search

Since grammatically, "search" can be a noun as well as a verb, it's okay to have a URL called /search. Arguably, the best way to filter the response a server delivers to a GET request on /search is to use a querystring:

/search?doctype=article&keyword=hello%20world

It is possible to do this without a querystring - something like:

/search/doctype/article/keyword/hello%20world

But that's silly and pedantic - and arguably wrong, anyway. It's harder for your users to guess and it's harder for you to parse.

Pagination

When your user does a GET request on /api/cats, do they really want a list of 7 million cats? Do you really want to have to serve that and support the processing and bandwidth involved?

Again, I don't think there's anything wrong with using querystrings here:

/api/cats?offset=101&limit=50

Similarly, I don't think there's anything wrong with using terminology that is familiar to developers who work with databases.

Versioning

Your API is a contract you have made with your users. If you plan to introduce a change to your API that breaks backwards compatibility, you need to do so in such a way that it doesn't break existing clients.

There is no easy, obvious way to do this that will make everyone happy, satisfy REST and satisfy the goal of making your API usable and discoverable.

There are a few different ways you can version your API, all of which have pros and cons.

Versioning in URL

Many popular web APIs do this: /api/v1/cats. If you roll out a version 2 of your API that would break clients using version 1, you expose it at api/v2/cats.

Pro: it's easy for users to understand and doesn't break existing clients.
Con: results in multiple URLs for the same resource; locks versioning into the URL; multiplies API maintenance issues.

Versioning in Querystring

Some APIs do this: api/cats?v=1.

Pro: default URL doesn't need to include version (defaults to newest version); one URL for each resource.
Con: clients that don't explicitly specify a version will break when the API is updated.

Versioning in Accept header

REST purists recommend putting the version in a custom Accept header: Accept: application/vnd.company.app-v1+json

Pro: default URL doesn't need to include version; one URL for each resource; changes do not break clients; version is a formatting issue so it's the "right" place.
Con: more esoteric and difficult for developers to discover how it works.

Use Secure HTTP

I don't want to say too much about security (though I do recommend the REST Security Cheat Sheet - Draft document from the Open Web Application Security Project (OWASP) as a handy reference), but I will say a quick word in support of Secure HTTP - or HTTPS - rather than plaintext HTTP.

HTTPS is a bit slower, but on a web service, you're not making multiple requests to pull down javascripts, CSS, image files and so on, so you can afford to take a small hit for an encrypted connection.

The payback in client security is worth it.

Summary of Benefits

It's extremely easy to lose yourself in theory and get sucked into the purity movement (remember XHTML?). If REST doesn't deliver practical results, it doesn't matter how elegant, pure or canonical it claims to be.

In short, a RESTful API is discoverable and navigable. With no background knowledge, a client can do a GET request on the root URL to get a list of resources with URLS. It's easy to dive right in and quickly figure out what is available.

Better URLs

Stable, persistent URLs allow for resource bookmarking and better search indexing.

Sane, well-named URLs are more descriptive than arcane or arbitrarily named URLs.

Hackable URLs allow for better resource discovery.

Developers Understand HTTP

REST is essentially just HTTP, and we already understand HTTP (with an important caveat, below).

The API developer doesn't need to waste time and energy (badly) reinventing/re-implementing what HTTP does on top of HTTP.

The client doesn't need to waste time and energy learning a bunch of ad hoc RPC protocols.

Our Tools Understand HTTP

Just about every programming language can natively perform an HTTP request and parse the response (with an important caveat, below).

Problems with REST

REST comes with its own issues, of course, and it would be disingenuous to gloss over them.

Requires Proper Understanding of HTTP

Much as I wrote earlier that REST is just HTTP and we all understand HTTP, the honest fact is that we don't understand HTTP all that well.

It's true we've all been using it for 20 years now, but for various historical, cultural and educational reasons, there's a constant tension between understanding and using our protocols the way they were designed and knowing just enough to build something that gets the job done.

That latter imperative lowers the barrier to entry into web development, but it also leads to the same avoidable mistakes being made over and over again in applications that require a more professional approach: SQL injection, XSS, hacked cleartext passwords, and so on.

We've all heard horror stories about web applications that didn't understand HTTP and blew up when a search engine bot innocently crawled a GET request to /delete_user.php?username=Ryan and thereby wiped out the entire database.

For a newbie developer who doesn't really understand HTTP very well, a web service with URLs like /api/create_user is more immediately obvious than a web service with URLs like /api/users that require a POST request to create a user.

However, this is changing as the benefits of REST become more widely known and more developers start to take advantage of its defining characteristics.

One thing is for certain: once you've developed and built on a REST web service, you'll finally understand HTTP inside and out.

Tooling

SOAP advocates argue that mature tooling makes it easy for a service provider to deploy a WSDL and for clients to consume it and execute web service methods as if they were local function calls.

Of course, the reality is less rosy: leaky abstractions, incompatible data types, lack of interoperability between, say, a Java WSDL and a C# SOAP client, and so on.

In far too many cases, I've had to hand-roll an ad-hoc SOAP client and build my own XML templates when a supposedly push-button WSDL turned out to be platform-dependent. Instead of saving time by consuming a WSDL, I was stuck painstakingly reverse-engineering the RPC formatting details that the interface tried unsuccessfully to hide.

Still, SOAP has that enterprise-friendly turnkey appeal, even if the delivery doesn't fulfill the promise.

REST, on the other hand, is sorely lacking in push-button tooling - on both the framework development and client side.

Even tools that ought to know better tend to abstract away the HTTP methods from the developer and generically treat every request as a GET request - unless it comes with form data or hits a kludgy /process_form kind of an URL.

We've come a long way from the cgi-bin form processing scripts of the '90s, but we still have a way to go. The good news is that this changing as developers come to recognize the benefits that come from understanding and using HTTP the way it was designed.

Frameworks

Most web application frameworks weren't developed with HTTP in mind, but rather with the minimal subset of HTTP that that is comprised of viewing web pages and filling out forms. Data transfer and processing, even in web applications designed for human users, tends unconsciously to follow an RPC model.

There are already a few that have embraced the full functionality of HTTP more deeply.

I'm not that familiar with Ruby, but I understand Rails 3 is designed to work RESTfully right out of the box with its routing DSL.

Similarly, the lightweight Sinatra framework understands resources and methods natively and simply:

require 'sinatra'

get '/users' do
  # show users
end

get '/users/1337' do
  # show a user
end

post '/users' do
  # create a user
end

put '/users/1337' do
  # update a user
end

delete '/users/1337' do
  # delete a user
end

In Python, Django is not RESTful by design has some good REST plugins.

The lightweight Python framework I use, web.py, is REST-friendly at its core: resources map to classes and HTTP methods map to class methods.

import web

app = web.application(urls, globals())

urls = ('/users/(.*)', 'User')

class User(object):
    def GET(self, name):
        # get a resource

    def POST(self, name):
        # create a resource

    def PUT(self, name):
        # update a resource

    def DELETE(self, name):
        # delete a resource

In PHP, you've got Zend_REST, Recess, Tonic and others that understand and support RESTful API design.

The .Net framework has Windows Communications Foundation (WCF). Java has Play RESTlet, Jersey and so on.

Client Tools

Nearly every language understands HTTP, but too many default HTTP modules understand HTTP in a crippled, semi-literate way that reflects how most browsers use HTTP: fetch pages or fill out forms.

In Python, there are no less than three HTTP libraries that ship with the language: urllib, urllib2 and httplib - and none of them make the full expressiveness of HTTP particularly accessible.

Fortunately, third party developers have stepped in with libraries like httplib2 and requests, which do a much better job of exposing all the HTTP methods in a sane, consistent manner.

Similarly, REST-aware clients are appearing in other languages as well.

If you're a .net shop, you've got tools like RestSharp.

Of course, JavaScript understands HTTP natively, perhaps better than some other, more "mature" and "full-featured" languages - possibly because JavaScript lives in browsers and breathes HTTP.

var xmlhttp=new XMLHttpRequest();

// GET request
xmlhttp.open("GET", "/users", true);
xmlhttp.send();
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('GET request successful'); 
        // response is in xmlhttp.responseText
    }
}

// POST request
var jsonObject = {"username": "ryan", "fullName": "Ryan McGreal"};
var jsonString = JSON.stringify(jsonObject);
xmlhttp.open("POST", "/users", true);
xmlhttp.setRequestHeader("Content-Type","application/json");
xmlhttp.send(jsonString);
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('POST request successful'); 
        // response is in xmlhttp.responseText
    }
}

// PUT request
var jsonObject = {"username": "ryan", "fullName": "Ryan G. McGreal"};
var jsonString = JSON.stringify(jsonObject);
xmlhttp.open("PUT", "/users/ryan", true);
xmlhttp.setRequestHeader("Content-Type","application/json");
xmlhttp.send(jsonString);
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('PUT request successful'); 
        // response is in xmlhttp.responseText
    }
}

// DELETE request
xmlhttp.open("DELETE", "/users/ryan", true);
xmlhttp.send();
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('DELETE request successful'); 
        // response is in xmlhttp.responseText
    }
}

JavaScript also has a panoply of frameworks and libraries that make HTTP requests even easier:

// yay jQuery!

$.get("/users", function(response){ alert('GET request successful'); });

$.post("/users", data, function(response){ alert('POST request successful'); });

$.put("/users/ryan", data, function(response){ alert('PUT request successful'); });

$.delete("/users/ryan", function(response){ alert('PUT request successful'); });

So the tools for creating and consuming REST web services are steadily improving, while the tools for creating and consuming SOAP web services are over-engineered and IMHO over-rated.

In the meantime, REST makes up in discoverability and navigability what it lacks in automated tooling.

Browsers

Of all our tools for working with HTTP, the most universal is the browser. It's the easiest way to browse a web service API, particularly if it works like the web itself. After all, the URL is just a resource on a server accessible via HTTP, and browsers are specifically designed to make HTTP requests and present the responses to users.

However, like other established HTTP-based tools, most browsers are only good at GET and POST requests, not PUT or DELETE requests. Even then, POST requests generally require an HTML form that the user can fill in and submit.

In addition, most browsers are not good at rendering representations in formats other than HTML or XML.

I use Firefox as my main browser (yes, I'm old-fashioned), and I've discovered a few add-ons that make exploring web APIs a lot easier:

HttpFox - an HTTP analyzer.
JSONView - pretty-prints JSON response objects in the browser window.
Modify Headers - Add, modify and filter the HTTP request headers sent to a web server.
RESTClient - Visit and test REST/WebDav services.

REST is a Spectrum

Ultimately, it makes sense to talk about an API being more or less RESTful and you probably shouldn't kill yourself to reach 100.00000%.

Like anything else, REST purity is subject to diminishing returns. If you get the important stuff correct - URLs are resources, methods are actions, representations include URLs - you'll get to enjoy the main benefits of a REST design approach without tearing your hair out.

For each choice, ask yourself whether the benefit of going more RESTful justifies the cost. Keep in mind that your goal is to make your API as usable and discoverable as possible.

References

Classification of HTTP-based APIs

http://nordsc.com/ext/classification_of_http_based_apis.html
Richardson Maturity Model - A model (developed by Leonard Richardson) that breaks down the principal elements of a REST approach into three steps. These introduce resources, http verbs, and hypermedia controls.

http://martinfowler.com/articles/richardsonMaturityModel.html
REST APIs must be hypertext-driven - Roy Fielding's attempt to get people to understand what he was talking about when he defined REST.

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
RESTful API Design - Pragmatic REST - Webinar by Apigee.

https://www.youtube.com/apigee#p/c/5/R8SIxZVaai4
REST Security Cheat Sheet - Draft document from the Open Web Application Security Project (OWASP)
RESTful Web Services: The Basics - IBM developerWorks article

https://www.ibm.com/developerworks/webservices/library/ws-restful/
General Principles for Good URI Design - StackOverflow answer

http://stackoverflow.com/a/1619677/682547

Video

Here is a video of my talk at Hamilton DemoCamp #5.

Python String Formatting With Dictionaries

2012-01-04T12:00:00Z

My mind is duly blown. I never realized that the traditional printf-style string formatting in Python - the kind that uses the % operator - supports the use of a dictionary as well as a tuple.

The following works:

>>> data = {
    'title': 'Python String Formatting With Dictionaries',
    'author': 'Ryan McGreal',
    'summary': 'In Python, you can use dictionaries instead of tuples to populate values via classic string formatting.',
    'content': '<p>My mind is duly blown...',
    'author_bio': 'Ryan McGreal lives in Hamilton with his family and works as a programmer and writer.',
}

>>> template = """<!DOCTYPE html>
<html lang="en">
<head>
<title>%(title)s</title>
</head>
<body>
<article>
<header>
<hgroup>
<h1>%(title)s</h1>
<h2>By %(author)s</h2>
<h3>%(summary)s</h3>
<hgroup>
</header>
%(content)s
<footer>
%(author_bio)s
</footer>
</article>
</body>
</html>"""

>>> print template % data

As an ultra-lightweight templating engine, this is pretty sweet: it's fast, simple, supports Unicode, and has no third-party dependencies.