Quandy Factory Newsfeed - Projects

MakeChart

2010-04-27T12:00:00Z

Check out the Github repository for source and documentation.

Here's an example use, assuming you save makechart.py somewhere in your PATH:

try: import json
except: import simplejson as json
import urllib
import makechart

# chart with world petroleum production data by month from EIA
url = 'http://quandyfactory.com/json/makechart' 
output = urllib.urlopen(url)
contents = output.read()

dataset = json.loads(contents)
caption = 'World Oil Production by Month, 2001-2010<br>(Source: EIA)'
unit = 'mbpd'
chart = makechart.make_chart(dataset, caption, unit)
html = makechart.make_html(chart)

file = open('makechart_example.html', 'w')
file.write(html)
file.close

Note: for the sake of convenience, this example uses a sample dataset in JSON format that is hosted on this website.

PyToc

2010-04-15T12:00:00Z

Introduction

PyToc generates a table of contents for an HTML document based on headings, with anchor links from the TOC to specific headings.

Download

You can download the latest version of PyToc from its github repository:

http://github.com/quandyfactory/PyToc

Requirements

Python 2.5 or 2.6
BeautifulSoup 3.0.8

Using PyToc

You can see the code in action on this very page!

It's pretty simple to use. Download pytoc.py and save it somewhere in your PATH.

Here's a demonstration:

>>> import urllib
>>> import pytoc
>>> url = 'http://quandyfactory.com/projects/40/pytoc'
>>> page = urllib.urlopen(url)
>>> html = page.read()
>>> toc = pytoc.Toc(html_in=html)
>>> toc.make_toc()
True
>>> toc.html_toc # returns an HTML table of contents
>>> toc.html_out # returns the html with anchors and numbering in headings
>>> toc.toc_list # returns a list of tuples in the form (section number, title)

Input Properties

The following are input properties you enter to generate the table of contents.

html_in - The HTML document for which you want to generate a table of contents.

This is the only necessary property to assign. The rest have default values that may meet your needs.
levels - A list of numbers corresponding to the heading levels you want to include in your TOC.

E.g. [3, 4] would include <h3> and <h4> headings. Default is [3, 4].
id - The base id of the HTML table of contents to be generated. Default is "toc".
title - The title of the generated table of contents. Default is "Contents".

Methods

make_toc() - this generates the table of contents and populates the output properties. Returns True when complete.

Output Properties

After calling the make_toc() method, the following output properties are populated with values.

html_out - The same as html_in except with the TOC anchors and numbering included in the headings.
html_toc - The generated HTML table of contents.
toc_list - A list of tuples containing the anchors and headings, in case you would rather roll your own HTML table of contents.

That's it, really.

History

This library started out as one-off code to generate a table of contents for a long document that I'm converting from MS Word format over to HTML.

The existing Word document has had many contributors and editors over the years and the format is a shambles. The table of contents is a mess and the headings are all over the place. (Thanks, WYSIWYG.)

I converted the whole thing into Markdown, a simple plain-text formatting syntax that converts to clean, structural HTML.

I still wanted the final document to have a table of contents, but I didn't want to have to go through the bother of maintaining the thing - especially if I ever wanted to add a new section in the middle, which would require a re-numbering of all the subsequent sections and subsections.

I whipped up a simple parser that walked the document and dynamically generated a table of contents, plus anchors in the document so the section headings listed in the contents could straight down to the sections themselves.

Need for Flexibility

It worked, but the code was brittle. It required the HTML formatting to be very strict, e.g. the following would work:

<h3>Some subheading title</h3>

but the following would not work:

<h3>
  Some subheading title
</h3>

Documents generated using python-markdown2 would work pretty consistently, but documents generated using, say, python-markdown would not, since the latter produces messier HTML output.

Of course, with documents produced using other means, all bets were off.

Anyway, I decided that if this was going to be at all useful as a general-purpose tool, it needed to be more flexible and forgiving. Of course, if you're programming in python, parsing HTML and want to be flexible and forgiving, there's no better tool than the mighty BeautifulSoup parsing library.

So I re-wrote it using BeautifulSoup.

PyCouchContacts

2010-04-12T12:00:00Z

Coming soon.

Gitiot

2010-04-07T12:00:00Z

Gitiot is a really simple cross-platform GUI wrapper for the most minimal useful subset of git's awesome power; i.e. one-button commit and push-to-master for people who want revision control but don't want to learn the command line.

Download the repository on github.

Download Tweets

2010-04-01T12:00:00Z

Introduction

A simple python script to download all the tweets for a given Twitter username.

Download the script and documentation from its github repository.

Notes

The twitter API will only let you download the most recent 3,200 tweets. (Don't worry - all your tweets are still in their database. They eventually plan to make them all available.)
The Twitter API also limits the number of data requests to 150 per hour. At 20 tweets per page, that means you're actually limited to 3,000 tweets.

Exciting Update

My first fork! On a request from John Fink, Jay Parlar was nice enough to fork Download Tweets and add optional command line username and filemane arguments.

As soon as I get a chance, I'll merge his additions into my branch.

Hamilton Spectator PDF Downloader

2010-03-23T12:00:00Z

Stack Trace for hnsh.py on proxy

2010-01-07T12:00:00Z

The Stack Trace

Here's the original stack trace (Python 2.6 on a Windows XP machine behind a proxy server) for Hacker News Shell.

C:\Python26>python hnsh.py
Traceback (most recent call last):
  File "hnsh.py", line 558, in <module>
    hnsh = HackerNewsShell()
  File "hnsh.py", line 314, in __init__
    self.stories = self.h.getLatestStories(self.alreadyReadList)
  File "hnsh.py", line 194, in getLatestStories
    source = self.getSource("http://news.ycombinator.com")
  File "hnsh.py", line 35, in getSource
    f = urllib2.urlopen(url)
  File "C:\Python26\lib\urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python26\lib\urllib2.py", line 389, in open
    response = self._open(req, data)
  File "C:\Python26\lib\urllib2.py", line 407, in _open
    '_open', req)
  File "C:\Python26\lib\urllib2.py", line 367, in _call_chain
    result = func(*args)
  File "C:\Python26\lib\urllib2.py", line 1140, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Python26\lib\urllib2.py", line 1115, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

Update 1: Proxies With Urllib2

After prompting the user for their proxy server, you'll probably want to do something like this:

# Example: proxies is a dict
proxies = { 'http': 'http://proxy.domain.com:80' } 
proxy_support = urllib2.ProxyHandler(proxies)
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)
urllib2.urlopen(url)

This is untested, but it should work more or less as written.

Update 2: If No Proxy

It may be helpful to note further that you can submit an empty proxies dict in the above code if the internet connection is direct:

proxies = {}

This way, you can use the same code to connect to the URL whether or not the user is behind a proxy server.

Update 3: hnsh.py on GitHub

Scott Jackson, author of hnsh.py, moved the code into git and posted the repository on GitHub:

http://github.com/scottjacksonx/hnsh

Check this out from the source code:

Special thanks to Ryan McGreal (http://github.com/quandyfactory) for the code that makes hnsh work from behind a proxy.

flushes with pride

PyCalendar

2010-01-05T12:00:00Z

This class has since been rolled into Quandy.

Wayne MacPhail's URL Lengthener

2009-12-22T12:00:00Z

Wayne MacPhail sent out a few tweets today in which he proposed an "URL lengthener". In a fit of whimsy (leavened by the promise of procrastination from present-wrapping), I decided to build it.

You can find it here:

http://quandyfactory.com/longurl

RTH Codebase Redesign

2009-12-20T12:00:00Z

The code is now running live on raisethehammer.org. The code is published to this project's GitHub repository.

If you encounter any bugs, please report them to the bug tracker.