Planet Python

William's Journal: Fapws3-0.10 is out

2012-02-12T11:41:33+00:00

Despite lot of progress on the python3 branch, I propose a new release
of Fapws3 for python2.x.

Indeed, since 1 year (date of the last stable release), lot of new
features have been added into Fapws :-).
And several are coming from, you, the community !!! Thus, I first
thank all contributors (code, tests, ...).
This is a great achievement.
More specifically, I will thank Stiletto, Liu Qishuai and Keith for
their active contributions.

To make it short, this release comes with the following new features:
- support of tuple for callback output.
- support of socket. Thanks to this, Fapws3 is serving webpages via a
socket instead of a port.
- better support for OSX

You can download the tarfile from the github website:
https://github.com/william-os4y/fapws3
Or directly via the following link:
https://github.com/william-os4y/fapws3/tarball/v0.10
Pypi users can grab it there too: pip install fapws3

As said in the mailinglist, the python3 version of Fapws3 is nearly
out (this release will be backward compatible with python2.7).
Thus I plan to have 2 parallel branches so that there is a need for
python2.5 or 2.6.
This will change in the following months, but currently Fapws3 for
python2.x is in the master branch.

Ruslan Spivak: Crammit – Yet Another Asset Packaging Library

2012-02-12T07:34:56+00:00

Crammit is really simple. It’s a small tool that uses YAML for configuration and other libraries to do the heavy lifting.

Here is a config file assets.yaml:

output: assets    # directory path relative to  current directory
fingerprint: true # add sha1 hash to the output file name

javascript:
  # 'common' is a bundle name
  # output file will have prefix 'common'
  common:
    - static/js/application.js
    - static/js/vendor/*.js
  utils:
    # paths are relative to the current directory
    - static/js/utils.js

css:
  base:
    # you can use Unix shell-style wildcards in file names
    - static/css/*.css

Let’s put it through its paces:

$ crammit -c assets.yaml

In addition to generated asset bundles it outputs an information file that sums up all what’s been done.

css:
  base:
    fingerprint: 71fe4cba05a1a51023c6af4c4abf9c47ab21e357
    output:
      gz: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.min.css.gz
      min: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.min.css
      raw: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.css
    size:
      gz: 108
      min: 235
      raw: 277
javascript:
  common:
    fingerprint: 6493b619c73c49ce1f4dfe2c31d41902e98acaee
    output:
      gz: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.min.js.gz
      min: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.min.js
      raw: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.js
    size:
      gz: 56
      min: 41
      raw: 50
  utils:
    fingerprint: c3ef63280b954d99e8b13fc11ea3031caee77f1a
    output:
      gz: utils-c3ef63280b954d99e8b13fc11ea3031caee77f1a.min.js.gz
      min: utils-c3ef63280b954d99e8b13fc11ea3031caee77f1a.min.js
      raw: utils-c3ef63280b954d99e8b13fc11ea3031caee77f1a.js
    size:
      gz: 42
      min: 22
      raw: 24

Check it out here for more information.

Godson Gera: Whitney Houston and Python!

2012-02-12T02:12:36+00:00

Back in 2004 or 05 (can't remember exactly) I have used "Heart break hotel" song of Whitney to test out the small mp3 music player I have built using Python (g2Dlite and NeverHeard). During testing this song was played out countless times to add features and fix bugs in my program. Apart from this I have never really listened to any of her songs. I was saddened to learn about her death this morning, news reminded me of those special days.

Cormoran Project: Cormoran 0.1 Roadmap

2012-02-11T19:19:21+00:00

This week we had been working very hard to have as soon as possible an usable version of Cormoran. There are only a few pending issues to release next version. You can find the roadmap here.

Hopefully next week we will announce the Cormoran 0.1 release :)

Twisted Matrix Labs: Announcing Twisted 12.0

2012-02-11T17:58:37+00:00

Thanks to Thomas Hervé, we are proud to announce the release of Twisted 12.0.

47 tickets are closed by this release, among them:

A fix to the GTK2 reactor preventing unnecessary wake-ups
Preliminary support of IPV6 on the server side
Several fixes to the new protocol-based TLS implementation
Improved core documentation's main page

Twisted no longer supports Python 2.4, the latest supported version is 2.5.

For more information, see the NEWS file here: http://twistedmatrix.com/Releases/Twisted/12.0/NEWS.txt

Download it now from:
http://pypi.python.org/packages/source/T/Twisted/Twisted-12.0.0.tar.bz2 or
http://pypi.python.org/packages/2.5/T/Twisted/Twisted-12.0.0.win32-py2.5.msi or
http://pypi.python.org/packages/2.6/T/Twisted/Twisted-12.0.0.win32-py2.6.msi or
http://pypi.python.org/packages/2.7/T/Twisted/Twisted-12.0.0.win32-py2.7.msi

Thanks to the supporters of Twisted via the Software Freedom Conservancy and to the many contributors for this release.

Ralph Bean: Hacking tw2 resource injection (in pyramid)

2012-02-11T14:58:10+00:00

In #toscawidgets this morning, zykes- was asking about how to do the jquery removal hack I wrote about a month back but in Pyramid instead of TurboGears2. First I’ll summarize the problem, then show you our solution. toscawidgets2 tw2.* provides lots of handy web ui widgets including a wrapper around jquery-ui, jqgrid, and lots of [...]

Catherine Devlin: HTSQL-powered Flask

2012-02-11T14:14:59+00:00

With seven lines of code, you can plug HTSQL into a web application platform like Flask. Yes, I know that you know SQLAlchemy already; the point is that then your Flask apps can support HTSQL's rich in-the-URL filtering language.

For instance, this app serves reports from HTSQL's sample "University" database. Each report can run as-is


  http://localhost:5000/hardcourses

or can accept arbitrary HTSQL filters


  http://localhost:5000/hardcourses?title~'ogy'

The advantage over plain HTSQL is that you can use full Flask power. In this case, the reports come through a template that includes functioning links in each row, so it's a drill-down report:


<td><a href="/departments?code='{{ row.department_code }}'">
        {{ row.dept }}</td>

Here are the magic seven lines:


htsql = HTSQL('pgsql:uni')   # customize me

def data(qry):
    filters = flask.request.url.replace(flask.request.base_url, '', 1)
    if filters and ('?' in qry):    # this can be fooled by a filtered subquery
        filters = '&%s' % filters.strip()[1:]
    qry = '%s%s' % (qry, filters or '')
    return htsql.produce(qry)

Now each of your report pages asks for data in the form of an HTSQL query, and the ``data`` function appends any user-supplied filters to that:


    return flask.render_template('departments.html', data=data(qry))

full code for the sample app

Mike C. Fletcher: Planning to use Numpy Structured Data Types

2012-02-11T13:18:06+00:00

One of the things I'd like to have for a revised OpenGLContext/scenegraph API is a nice, efficient, friendly, mechanism for processing buffer data. I currently tend to follow VRML97's OpenGL 1.1-style array model, which is very dated these days. Each component of a vertex is separated out into position, normal, and textureCoordinate arrays and the drawing operation indexes into those arrays in lock-step. Modern OpenGL (shaders) pretty much work best when you have "interleaved" data-types for your vertices, that is, you pack (position, normal, textureCoordinate1, textureCoordinate2, someOtherValue) into a single VBO and then just use offsets into the VBO for the actual rendering.

The rendering loop (the part most likely to be coded in C/Cython/C++ eventually) doesn't really have to "deal with" the arrays other than as opaque blobs, as it is the shader which interprets what is inside them. So, only the "client" side needs to model them. Numpy structured data-types should provide a very nice way to do the modelling:

dtype=[
    ('position', [('x','1f'),('y','1f'),('z','1f')]),
    ('normal', [('x','1f'),('y','1f'),('z','1f')]),
    ('texCoord',[('s','1f'),('t','1f')])
]

should create a VBO compatible friendly interface to N data-points, so that a['position'] is an N*3 array of 3-float vectors and a['texCoord'] is an N*2 array of 2-float vectors, while a['position']['x'] is a simple linear array of x coordinates. At that point I can just stop worrying about array representations. I can use numpy's fast array manipulation routines (which I already do in OpenGLContext).

The only real downside (assuming it all works as expected) IMO is the large dependency (does anyone care about this anymore), and having the numpy implementation detail "leak" into my code-base (mostly an annoyance when I look at porting to Javascript, and really, one way or another there needs to be an interface, that interface will leak into the code using it).

Another approach would be to use a custom-coded C-ish extension providing just the basics of a Vertex object (with configurable fields), some dot and cross product operations and some other basic math... I would control the API, sure, but that doesn't really convince me it would be worthwhile. Similarly, I could pick up a 3D math-focused library and use that, but then I'm still using someone else' API, so why not use the "standard" python one.

Doug Hellmann: rst2blogger 1.0

2012-02-11T12:42:47+00:00

rst2blogger is a command line program for converting reStructuredText
documents to HTML suitable for posting to blogger.com. It takes as
input a single filename and an optional blog title. The input file is
parsed with docutils to create HTML, and the HTML is uploaded as a
draft to the specified blog. If the blogger account only has one
blog, the name does not need to be specified.

See the project documentation for installation and setup instructions.

Fredrik Håård's Blaag: Why Python is important for you

2012-02-11T10:26:03+00:00

Why Python is important for you

I believe that Python is important for software development. While there are more powerful languages (e.g. Lisp), faster languages (e.g. C), more used languages (e.g. Java), and weirder languages (e.g. Haskell), Python gets a lot of different things right, and right in a combination that no other language I know of has done so far.

It recognises that you’ll spend a lot more time reading code than writing it, and focuses on guiding developers to write readable code. It’s possible to write obfuscated code in Python, but the easiest way to write the code (assuming you know Python) is almost always a way that is reasonable terse, and more importantly: code that clearly signals intent. If you know Python, you can work with almost any Python with little effort. Even libraries that add “magic” functionality can be written in perfectly readable Python (compare this to understanding the implementation of a framework such as Spring in Java).

Python also acknowledges that speed of development is important. Readable and terse code is part of this, and so is access to powerful constructs that avoid tedious repetition of code. Maintainability also ties into this - LoC may be a all but useless metric, but it does say something about how much code you have to scan, read and/or understand to troubleshoot problems or tweak behaviours.

This speed of development, the ease with which a programmer of other languages can pick up basic Python skills, and the huge standard library is key to another area where Python excels - toolmaking. Any project of size will have tasks to automate, and automating them in Python is in my experience orders of magnitude faster than using more mainstream languages - in fact, that was how I started out with Python, creating a tool to automate configuring Rational Purify for a project where it before was such a chore that it was never run (and memory leaks were not fixed). I’ve since created tools to extract information from ticket systems and presenting them in a way useful to the team, tools to check poms in a Maven project, Trac integration, custom monitoring tools... and a whole lot more. All of those tools have been quick to implement, saved a lot of time, and several of them has later been patched and updated by people with no Python background - without breaking.

That building custom tools is easy hints at another strength - building and maintaining custom software is easy, period. This is why, while the quite huge Django framework might be the most famous Python web framework, there is also a host of successful small and micro-frameworks. When working in a powerful programming language with a wide array of standard and third-party libraries, you often don’t need to accept the trade-offs that are necessary when using any large off-the-shelf framework. This means that you can build exactly the software your customers want, rather than telling them that ”this is how it’s done, sorry”. To me, this is a huge difference. I feel ashamed when I have to tell a customer that no, sorry, this seems like a simple requirement, but the framework we use makes it impossible or prohibitively expensive to implement. Whenever this happens, you have failed. Writing software that fits into the customer’s model rather than into a framework is important, and I for one feel that a lot of developers today has lost sight of that simple fact. A lot of programmers now spend more time being configurators of frameworks and makíng excuses for their shortcomings, rather than actual programming.

Finally, if you’re a boss-wo/man or general manager, using Python has a final benefit - Python programmers run into less frustration*, which makes them happier, and even more productive!

(*may not be true when installing source-distributed C extensions on Windows)

Tomer Filiba: Wizard Dialog Toolkit

2012-02-11T08:00:00+00:00

Following my Deducible UI post, and following some of the criticism it had received, I'd like to share something I've been working on (read: experimenting with) at my work place. You see, we have some "interactive wizards" that storage admins use to connect storage arrays to their hosts (say, a DB server). These wizards prompt you with questions like your what's your username, the name of the pool/volume, whether it's an iSCSI or a Fiber Channel connection, etc., and then they go and perform what you've asked for.

These wizards operate in a terminal environment, but we've had thoughts to make GUI/web versions of them. This would be a considerable effort with the current design. Another issue they currently have is the mixing of "business logic" and presentation together. For instance, the code that scans the devices attached to your host also prints ANSI-colored messages or reports its progress. All in all it works fine, but there's lots of room for improvement.

I began to investigate this corner a month or two ago. The initial observation was that such wizards have a pretty rigid and repetitive structure, thus we can find some abstraction or a "toolkit" for "expressing" wizards more compactly. This has also led to the realization that once the business logic and presentation are separate, there's no reason to limit ourselves to terminal-based interaction: our wizard-toolkit could do the plumbing and work with terminals, ncurses, GUIs, web-browsers, etc. The business logic would remain oblivious, and we could have a nice GUI at zero-cost!

There was also a second issue of styling, i.e., printing text in color, that I wanted to get r id of. This part was easy: I thought, why not employ the model of HTML and CSS? Let's separate the structure (semantics) of the text from its styling. Instead of printing a banner for titles, we'll display a Title object, whose exact appearance is determined by a "style sheet" (a class, of course, not actually a text document).

For instance, when we're using a color-enabled terminal, the title would be printed in bold and followed by an empty line; but if our terminal is color-blind, we'll render the text centered and surrounded by = marks. Another example is error-handling: instead of printing error message in red every time, we'll display an Error object; on a terminal, this would be rendered as red text, but when running in a GUI, rendering this object would pop up a message box. I'm going to ignore this for the rest of this post, as this is really a side issue.

Now let's get to expressing wizards, or more generally, dialogs. Following some earlier iterations, I came to the model where a dialog is a "container object" that's made of dialog elements. These elements can be output-only (such as a welcome message), or input-output (such as a message telling you to choose one of the available options). A dialog is "executed" by a DialogRunner that renders it and returns the results gotten from the user. It's quite important to note that dialog elements within a single dialog cannot be interdependent -- that is, if you want to ask the user for his name and then show "Hi there %s" with the user's name, this has to be done as two, serial dialogs.

That was quite a lot of babble -- let's see this in action:

class MyApp(WizardApp):
    def main(self):
        iscsi = Option("iSCSI")
        fc = Option("FC")
        d = Dialog(
            Text(Title("hello world")),
            Input("un", "Username"),
            Password("pw", "Password"),
            Choice("conf", "What do you want to configure?", [iscsi, fc]),
        )
        res = self.ui.run(d)
        if res["conf"] == fc:
            self.config_fc(res)
        else:
            self.config_iscsi(res)
    ...

if len(sys.argv) == 2 and sys.argv[1] == "--gtk":
    MyApp.run(GtkDialogRunner("My App"))
else:
    MyApp.run(TerminalDialogRunner(ANSIRenderer))

It's a short and incomplete snippet of course, as I'm only going to cover the big picture. The main function creates a dialog object d and passes it to ui.run, which "runs" the dialog and returns the results, as a dictionary. Notice that the dialog elements Input, Password and Choice all take a first parameter -- this is the key under which the result would be placed in the returned dictionary, e.g., res["un"] would hold the user-provided user name, and res["pw"] would hold the password. Text, on the other hand, is an output-only element, so it doesn't return anything and doesn't take a key. Long story short, we're asking the user to enter some information and choose one of two options, and then continue processing based on the selected option. At the bottom, we determine how to run the application based on a command-line switch: if --gtk is given, we'll run the dialogs through the GtkDialogRunner; otherwise, we'll use the TerminalDialogRunner.

And how does it look like? When running on a terminal:

And with a single command-line switch, we run as a GTK application:

So of course it's far from perfect, but then again, it's a small research project I've only put ~15 hours into. It suffers from some of the problems I've listed in the deducible UI post, for instance, the GUI hangs when the business logic performs blocking tasks. This could be solved by moving to a reactor-based model, but I've tried to keep the existing wizard code in tact as much as possible. A hanging GUI is not nice, but it's not the end of the world either, and there are numerous ways to overcome this.

Another benefit this design brings along is the ability to automate testing by using mock dialog runners. Since our business logic is only exposed to the returned dictionary, we can use a dialog runner that actually displays nothing and returns a scripted scenario each time. We can even go further: because our business logic "talks" in high-level primitives like Choice, we can compute the Cartesian product of all choices and run through each of them. We can show that we've covered all paths! And we can do this automatically... without people hitting buttons and keeping logs of their progress.

Anyway, I just wanted to show that it's feasible. I'm not releasing any code as this project is currently in very early stages, and it's something I do at work. Perhaps we'll open-source it in the future, if it proves useful enough.

Python Diary: Tutorials section having issues

2012-02-11T06:40:38+00:00

The Tutorials section is having some technical difficulties, as I have previously added 2 more tutorials. The tutorials are not showing up despite the published date being set correctly. I find this odd as the other 2 tutorials work just fine, it's just these 2 new ones which are very reluctant to show up and be accessible.

I altered the publishing date to today, so hopefully they show up this time. If not, I will need to dive into the Python source code to find out why they are not showing up.

Overall, I am rather happy with the rest of the sites sections, although the template needs to be redone. I plan on looking for another similar template, but one which scrolls better to support multiple screen sizes. A mobile template is also planned for those out there who read the site on a mobile device. I assume most of my subscribers are using the provided Atom feeds, as it contains the entire posting, and should be compatible with most popular news readers.

The Reviews section will be built shortly, as this is one section I hope will bring plenty of traffic. The review section is a bit ambitious, as it will differ from other sites which mostly do comparisons between packages. It will be formatted something like a CNet review. It will have the same general information, such as the The Good, The Bad, and The Bottom Line. The Review will also list any dependencies the package would require. The dependency system will be similar to how the Tutorials section dependencies work, a simple clickable link to each dependency, full with description and project website.

What are your thoughts on this type of review system for Python packages? Reviews will take time, as I will need to spend a good amount of time working with the package and it's various features to determine it's ability to deliver. I've seen both great and not so great python packages out in the wild. The not so great ones are very difficult to use in existing projects, and some have issues with subclassing, or don't properly support it. Integration into existing projects is a valuable resource for any Python package. If it cannot be easily integrated into your existing project, then why use it in the first place? The time it would take you to integrate it, you could have built a similar package from scratch that is more fine tuned to your projects specific needs.

Robin Dunn: OS X Lion Was My Idea

2012-02-11T00:35:38+00:00

Several months ago Microsoft had an ad campaign where they would have an average Joe or Jane talk about some cool wizbang feature of Windows 7 and how it was all their idea. It was one of Microsoft's better ad campaigns, and was actually understandable by the average person without requiring the viewer to know industry inside jokes, or to understand Jerry Seinfeld, to be able to "get" the gist of the commercial. It also employed some cute gimmicks, such as when the person was remembering back to how they thought up the wonderful new idea the directors used a different actor who was younger, thinner, cuter, handsomer, and had way better hair than the real person.

A few weeks ago I realized that I had the same experience with Mac OS X Lion. Way back in 2007 I wrote a post in this blog about some of the things that annoyed me about OSX. Two of them were finally fixed in Lion, and it only took 4 years! Here is the first one.

Can only resize from 1 corner

It bugs me that I can only resize windows from the lower right corner. I often find myself wishing that I could resize from any edge or corner as I've been doing for nearly 20 years on every Windows, OS/2 and X-Windows system I've ever used.

We can now grab any corner or side of most windows and resize the window by dragging the mouse cursor. Yay! For you non-Mac users out there, until you're not able to do that you may not actually realize how much you do it. For example, if you have a window that you would like to make wider and have it grow to the left, then on Macs running an earlier version of the OS you would first have to move the window to the left, and then you would have to move the mouse cursor down to the bottom right corner and drag to the right. That's lots more effort than it should require, and now, thanks to me, you don't have to go through all of that! <wink!>

Can't change mouse cursor

You'd think that with how important the visual appeal of the OS seems to be with Apple that they would not have let something this ugly slide through their QA department for so long. Since I usually have multiple computers with multiple screens active at once, I like to increase the size of the mouse cursor a bit, or replace it with alternate cursors that are more easily visible. I've tried options for things like giving the cursor a comet trail of fading arrows behind it, or what not, and I hate them all after a few minutes. For me, just increasing the size and/or using a colored arrow that is easily visible does the job and doesn't get in the way of my work. However on Mac OS X you can't use custom cursors and all you can do is scale the image up in the Universal Access section of System Preferences. And being Apple you might assume that it would be a nice scaled up anti-aliased rendering of a vector image, but NO! They simply take the original 16x16 pixel image and stretch it to fit the new 64x64 or whatever size. Anyone who has used a photo editing application more than a few times knows you can't take a small image and enlarge it very much without it looking like crap! It embarrasses me too much to run a Mac with a cursor stretched up to a monstrosity like that, so I suffer with the itty-bitty one that occasionally gets lost among the dozens of windows on my desktops.

Next up is the lack of ability to have a nice looking mouse cursor that is larger than normal. At least that's the way it was prior to Lion. Now when you change the cursor size it scales up very nicely, probably because they are using some scalable image format now instead of a small 16x16 bitmap image. Unfortunately if an application uses some non-standard cursor images they are still poorly scaled up into a jagged edged ugly blob. Also unfortunately I haven't been able to find a screenshot program yet that will capture the new enlarged cursor so I can show you the difference. They all end up with the small default cursor in the screenshot instead. Weird. Anyway, I'm no longer embarrassed to have a larger than normal cursor on my Macs because they are no longer so dang ugly. Thanks to me you can also have a purdee big cursor too! <wink!>

Now the only question remaining is who should Apple get to play the younger/fitter/handsomer version of me while filming the flashback parts of my "OS X Lion Was My Idea" commercial, what do you think?

Armin Ronacher: Appreciating Design

2012-02-11T00:00:00+00:00

I think no piece of software had a bigger impact on my work than Trac. Nowadays people often look down on Trac for various different reasons but if you keep in mind how bug management and revision control web interfaces in the open source world looked before Trac you can see why it initially had that much hype around it.

Learning from Trac

When Trac was released tons of Open Source projects used it. It had a slick design and did the few things it provided quite well. The wiki was not great but useable, the integrated subversion support was very well implemented and the bug Tracker was simple but friendly. It was the first bug Tracker that I had used that by default could have been presented to a non technical user without having to tweak it.

Trac had a reasonable user interface design compared to the competition. But the beauty of Trac has always been the code that runs it. For the German ubuntu community I deployed a phpBB installation a few years prior to when I used Trac first. One of the great features of phpBB was that people wrote extensions for it that provided extra functionality to the base bulletin board system. The way these plugins worked however was by actually modifying the code. Called “mods” they came with long installation instructions that showed the specific places where you had to copy/paste code in. Not even diff files were provided in those days but it still sortof worked because people took the time to hand patch their installations.

Trac on the other hand always had a real plugin system. The base system was littered with interfaces and hook points that plugins could utilize to add extra functionality. Trac's code taught me proper software and API design. Now it seems obvious that this is how you create libraries and applications but the perfection that Trac had in that regard is still largely unmatched. At least in the Python world.

For instance Trac always supported running more than one Trac instance per Python interpreter. It was from the ground up designed for that. Django as a framework still does not do that. Which is especially sad considering that Django is a framework and not an application. You could not reimplement Trac in Django without losing this functionality. Trac also had the concept of an instance folder where the instance specific files are dropped (design, static files, uploads, configuration, the sqlite db if used, per instance plugins etc.) which more people should consider.

Trac also explicitly did not adopt hyped technologies if they went against their design principles. They wanted to use a template engine that was extensible and built on XML to properly support XHTML (which was hip and cool at the time) but Kid did not serve that purpose. It did not scale properly to the large pages that were generated and it did some funky things with the Python import systems. Christopher Lenz wrote Genshi as a result of that which was a reimplementation of the Kid principles but with a largely improved implementation. Genshi is still (from the design perspective) the coolest piece of template engine you can get in Python land. It's also the reason why I no longer hate XML all that much because properly used it's actually very clever.

However all of that is completely irrelevant because Trac and Genshi failed in other areas. They were amazingly well written but all that came at huge costs. Trac was a memory hog and some early design mistakes that everybody relied on were only slowly phased out. For instance before Trac had Genshi they used a template engine called quicksilver. It was implemented in C and used a special syntax similar to JSON to specify the template context. When that was replaced with Genshi it broke existing plugins and provided users with a good excuse to switch from Trac to alternatives that had the functionality they required built in. Trac was also built around the subversion idea of having one huge repository for everything. This became a problem when decentralized version control systems popularized the concept of having more than one repository.

Users don't care, or do they?

Ultimately all of that would have been great however if the users would have appreciated the elegance of the design. But that's not how it seems to work. A user expects his software to work and the internals don't matter at all. And this is not just true for Trac. Battlefield 3, which I wrote about last year, moved the main menu into a browser. There are a bunch of reasons why the whole concept in generally is technically amazing, but for the user it was not understandable why it was cool or useful.

It's not that users are dumb and would not understand it, but it was never communicated in the first place. I don't know if it's because the developers did not take their pride far enough in showing everybody how cool it actually is or if they think users would not understand it.

Let's compare this to cars for a moment. If I look around for a car, the design is only half the reason. I look for the technical details when I shell out that much money. I want to know why it's cool, I was to see images of the engine and internals. I want to read about the design ideas that went into every part of the product. True, there will be some people that just want a car, but those are not selling your car anyways. If a friend of mine shows up with a brand new BMW I can promise you he will start telling me about the cool details, he wants me to actually try it and ultimately he wants me to get this car.

Apple seems to be doing very much the same. Many Apple computer users are as loyal to their product as buyers of cars. They will point how cool the case design is, how nicely the lid closes, how cute the machine looks when the sleep light is “breathing” or how few problems they have with it. I know many flaws in my Macs but that does not stop me from continue supporting the company and getting their products. That's because I appreciate the work that went into every part of it.

Head over to the Apple website and you see the product in the center like you see a car. Apple not only sells you the product there, they also provide videos of the assembling process. They let designers share their thoughts that went into every aspect of the product and they show you that they love what they created.

Why don't we do that about software products? Trac probably would not have become better by itself just because users would know what goes on behind the scenes, but I can almost promise you that more developers would have had a look at the internals if they were communicated properly and that might have helped improving it. As it stands right now, only a small community remains around Trac and only one active developer remains.

With Battlefield 3 it's very similar. The advert said “the easiest way to play with friends” but it did not say why Battlelog (their online community and also main menu) was cool. Why it's better for the user than the main menu being in the game. Combined with the fact that the game did not deliver in some other parts (such as missing VOIP support) people suddenly start blaming battlelog for parts it does not have anything to do with it. And an unhappy fanbase is the very last you want to have when it comes to multiplayer games because those games are sold in a very similar way: if my friends play I want to play it too. Selling technical excellence to computer gamers would not have been hard either because they take pride in “knowing more” about computers and game then their console playing friends.

Make them appreciate it

I guess to a large degree the idea of “the best design is the one you don't notice” plays into that but that does not mean it wouldn't be great to point it out nonetheless. And by that I don't mean putting a red arrow on the product saying “this is a cool design element” but by providing a nicely designed website or folder that shows the thought process that went into it.

I know nothing about cars but damn do I love looking at brochures and reading about them.

Roberto Alsina: PyQt Quickie: Don't Get Garbage Collected

2012-02-10T22:57:35+00:00

There is one area where Qt and Python (and in consequence PyQt) have major disagreements. That area is memory management.

While Qt has its own mechanisms to handle object allocation and disposal (the hierarchical QObject trees, smart pointers, etc.), PyQt runs on Python, so it has garbage collection.

Let's consider a simple example:

from PyQt4 import QtCore

def finished():
    print "The process is done!"
    # Quit the app
    QtCore.QCoreApplication.instance().quit()

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    proc.start("/bin/sleep 3")
    # After it finishes, call finished
    proc.finished.connect(finished)

def main():
    app = QtCore.QCoreApplication([])
    # Launch the process
    launch_process()
    app.exec_()

main()

If you run this, this is what will happen:

QProcess: Destroyed while process is still running.
The process is done!

Plus, the script never ends. Fun! The problem is that proc is being deleted at the end of launch_process because there are no more references to it.

Here is a better way to do it:

from PyQt4 import QtCore

processes = set([])

def finished():
    print "The process is done!"
    # Quit the app
    QtCore.QCoreApplication.instance().quit()

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    processes.add(proc)
    proc.start("/bin/sleep 3")
    # After it finishes, call finished
    proc.finished.connect(finished)

def main():
    app = QtCore.QCoreApplication([])
    # Launch the process
    launch_process()
    app.exec_()

main()

Here, we add a global processes set and add proc there so we always keep a reference to it. Now, the program works as intended. However, it still has an issue: we are leaking QProcess objects.

While in this case the leak is very short-lived, since we are ending the program right after the process ends, in a real program this is not a good idea.

So, we would need to add a way to remove proc from processes in finished. This is not as easy as it may seem. Here is an idea that will not work as you expect:

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    processes.add(proc)
    proc.start("/bin/sleep 3")
    # Remove the process from the global set when done
    proc.finished.connect(lambda: processes.remove(proc))
    # After it finishes, call finished
    proc.finished.connect(finished)

In this version, we will still leak proc, even though processes is empty! Why? Because we are keeping a reference to proc in the lambda!

I don't really have a good answer for that that doesn't involve turning everything into members of a QObject and using sender to figure out what process is ending, or using QSignalMapper. That version is left as an exercise.

Mike C. Fletcher: Phenomenal Enterprise-Grade Messaging: Itty Bitty Message Size

2012-02-10T21:35:37+00:00

(Read that in Robin Williams' Genie Voice) A project I used to work on has been informed by certain rabbit-mq types that they should not send messages > 15KB on a RabbitMQ server because it would "chew up all the subscription threads and limit throughput"... which... wow, dropped off a robustness/scalability/credibility cliff there.

Invent with Python: New Book: “Making Games with Python & Pygame”

2012-02-10T21:13:03+00:00

I’ve completed my next book, which focuses on the Pygame library and making graphical games in Python. It assumes you have a little bit of Python programming knowledge. The book is free to read online from http://inventwithpython.com/pygame and can also be bought on Amazon.com for $25.

Thanks to everyone who helped me out with this book over the last year and a half.

Tarek Ziade: Defining a wsgi app deployment standard

2012-02-10T19:28:48+00:00

Next month at Pycon, we’ll have a web summit and I’m invited there to talk about how I deploy web applications. This is not a new topic, as it was already discussed a bit last year — see Ian Bicking’s thought on the topic.

My presentation at the summit will be in two parts. I want to 1/ explain how I organized our Python deployments at Mozilla (using RPMs) 2/ make an initial proposal for a deployment standard that would work for the community at large – I intend to work on this during Pycon and later on the dedicated SIG.

Here’s an overview of the deployment standard idea…

How we deploy usually

If I want to roughly summarize how people deploy their web applications these days, from my knowledge I’d say that there are two main categories.

Deployments that need to be done in the context of an existing packaging system — like RPM or DPKG
Deployments that are done in no particular context, where we want it to just work. — like a directory containing a virtualenv and all the dependencies needed.

In both cases, preparing a deployment usually consists of fetching Python packages at PyPI and maybe compile some of them. These steps are usually done using tools like zc.buildout or virtualenv + pip, and in the case of Mozilla Services, a custom tool that transforms all dependencies into RPMs.

In one case we end up with a directory filled with everything needed to run the application, except the system dependencies, and in the other case with a collection of RPMs that can be deployed on the target system.

But in both cases, we end up using the same thing: a complete list of Python dependencies.

The trick with using tools like zc.buildout or pip is that from an initial list of dependencies, you end up pulling indirect dependencies. For instance, the Pyramid package will pull the Mako package and so on. A good practice is to have them listed in a single place and to pin each package to a specific version before releasing the app. Both pip and zc.buildout have tools to do this.

Deployments practices I have seen so far:

a collection of rpms/debian packages/etc are built using tools like bdist_rpms etc.
a virtualenv-based directory is created in-place in production or as a pre-build binary release that’s archived and copied in production
a zc-buildout-based directory is created in-place in production or as a pre-build binary release that’s archived and copied in production

The part that’s still fuzzy for everyone that is not using RPMs or Debian packages is how to list system-level dependencies. We introduced in PEP 345 the notion of hint where you can define system level dependencies which name may not be the actual name on the target system. So if you say you need libxml-dev, which is valid under Debian, people that deploy your system will know they’ll need libxml-devel under Fedora. Yeah no magic here, it’s a tough issue. see Requires-External.

The Standard

EDIT : Ian has a much more rich standard proposal here. (see the comments)

The standard I have in mind is a very lightweight standard that could be useful in all our deployment practices – it’s a thin layer on the top of the WSGIstandard.

A wsgi application is a directory containing:

a text file located in the directory at dependencies.txt, listing all dependencies – possibly reusing Pip’s requirements format
a text file located in the directory at external-dependencies.txt, listing all system dependencies – possibly reusing PEP 345 format
a Python script located it the directory at bin/wsgiapp with an “application” variable. The shebang line of the Python script might also point to a local Python interpreter (a virtualenv version)

From there we have all kind of possible scenarios where the application can be built and/or run with the usual set of tools

Here’s one example of a deployment from scratch :

The repository of the project is cloned
A virtualenv is created in the repository clone
pip, which gets installed with virtualenv, is used to install all dependencies describes in dependencies.txt
gunicorn is used to run the app locally using “cd bin; gunicorn wsgiapp:application”
the directory is zipped and sent in production
the directory is unzipped
virtualenv is run again in the directory
the app is hooked to Apache+mod_wsgi

Another scenario I’d use in our RPM environment:

The repository of the project is cloned
a RPM is built for each package in dependencies.txt
if possible, external-dependencies.txt is used to feed a spec file.
the app is deployed using the RPM collection

That’s the idea, roughly — a light standard to point a wsgi app and a list of dependencies.

Filed under: mozilla, python

PyCon: Interview with PyCon speaker Brian Jones

2012-02-10T11:11:02+00:00

After a successful tutorial session in 2011, Brian Jones is back this year with a talk on a booming topic: mocking. Testing has always been fairly popular in the Python community, but it seems like more developers than ever are really on board with testing. The difference between unit testing and functional testing is getting more coverage. Test-driven development is becoming a more popular methodology. Continuous integration is getting easier to setup and use. It seems like every corner of the testing world is improving at a nice pace. Mocking in particular has been blowing up with more widespread use and more options than ever. Brian's talk, Fake It Til You Make It: Unit Testing Patterns With Mocks and Fakes, aims to help developers get to the next level of testing by sharing best practices in the test design.

While Brian has long been a fan of mocking, the last several years have seen enough mocking libraries produced to get him to switch from his home-grown tools to try what the community has to offer. Although he’s a former user of the Mox project, the choice of library for this talk will be the aptly named Mock. However, the choice of Mock is less about the library and more about being able to easily convey ideas behind mocking. "I suspect that Mock's 'action->assertion' approach to mocking will be more intuitive to pick up 'on the fly', in the course of a talk, than the 'record->replay' approach taken by other modules," says Brian.

"I want people to think of testing as being more like a fun little puzzle than...calculus," he said when asked what caused him to submit the proposal for this talk. Mocking, he feels, is a huge help in getting testing to fit inside the minds of newcomers to the arena. With the talk he hopes to introduce the techniques and the tools, then put it all together in practice. The 45-minute session includes several types of example applications, from a datetime abstraction library to a command shell program, to show how Brian’s best practices with test design and mocking can be used.

Speaking is just one of the fun parts of PyCon for Brian. In his second year on PyCon's program committee, he helped review a record number of talk proposals. Sometimes meeting two times a day, the program committee had a busy fall between online reviews and voting, then the various IRC-based discussions on later rounds of reviews. Being a part of the program committee for a second year allowed him to "further [his] conviction that the Python community is the most welcoming, open and inclusive group [he's] ever had the pleasure of being involved with."

When it comes to getting on-site at the conference, last year Brian volunteered some of his time to help out around the event. The evenings may be one of his favorite parts: he's a huge fan of the BoFs, or Birds of a Feather meetings. “I've been to conferences where I'm really just not that interested in *any* of the BoF sessions, but at PyCon '11 I actually had conflicts: I wanted to attend multiple BoFs in the same time slot," he says.

One of the best benefits he says of the conference is the ability meet up with the people he had been interacting with online for years. Now that he’s been to a PyCon in the past, meeting up with those same friends and meeting many more in 2012 is an incredible experience. In closing, he remarked that "The community around Python is unbelievable, and attending PyCon is a great way to be bombarded by that fact on a continual basis for a few days, which is good for the soul."

Be sure to check out Brian's talk on Friday March 9 at 2:55!

PyPy Development: Introductory Article About RPython

2012-02-10T11:06:38+00:00

Laurence Tratt from King's College London has written a long and detailed introduction to the goals and significance of RPython over on his blog. Laurie has been implementing his Converge Language in RPython in the last months. He is one of the first people external to the PyPy team who have pushed a sizeable RPython-based VM quite far, adding and tuning JIT hints. The post describes some of that work and his impressions of RPython and PyPy.

"RPython, to my mind, is an astonishing project. It has, almost single-handedly, opened up an entirely new approach to VM implementation. As my experience shows, creating a decent RPython VM is not a huge amount of work (despite some frustrations). In short: never again do new languages need come with unusably slow VMs. That the the PyPy / RPython team have shown that these ideas scale up to a fast implementation of a large, real-world language (Python) is another feather in their cap."

PyPy Development: PyPy 1.8 - business as usual

2012-02-10T10:43:24+00:00

We're pleased to announce the 1.8 release of PyPy. As habitual this release brings a lot of bugfixes, together with performance and memory improvements over the 1.7 release. The main highlight of the release is the introduction of list strategies which makes homogenous lists more efficient both in terms of performance and memory. This release also upgrades us from Python 2.7.1 compatibility to 2.7.2. Otherwise it's "business as usual" in the sense that performance improved roughly 10% on average since the previous release.

you can download the PyPy 1.8 release here:

http://pypy.org/download.html

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (pypy 1.8 and cpython 2.7.1 performance comparison) due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or Windows 32. Windows 64 work has been stalled, we would welcome a volunteer to handle that.

Highlights

List strategies. Now lists that contain only ints or only floats should be as efficient as storing them in a binary-packed array. It also improves the JIT performance in places that use such lists. There are also special strategies for unicode and string lists.
As usual, numerous performance improvements. There are many examples of python constructs that now should be faster; too many to list them.
Bugfixes and compatibility fixes with CPython.
Windows fixes.
NumPy effort progress; for the exact list of things that have been done, consult the numpy status page. A tentative list of things that has been done:
- multi dimensional arrays
- various sizes of dtypes
- a lot of ufuncs
- a lot of other minor changes
Right now the numpy module is available under both numpy and numpypy names. However, because it's incomplete, you have to import numpypy first before doing any imports from numpy.
New JIT hooks that allow you to hook into the JIT process from your python program. There is a brief overview of what they offer.
Standard library upgrade from 2.7.1 to 2.7.2.

Ongoing work

As usual, there is quite a bit of ongoing work that either didn't make it to the release or is not ready yet. Highlights include:

Non-x86 backends for the JIT: ARMv7 (almost ready) and PPC64 (in progress)
Specialized type instances - allocate instances as efficient as C structs, including type specialization
More numpy work
Since the last release there was a significant breakthrough in PyPy's fundraising. We now have enough funds to work on first stages of numpypy and py3k. We would like to thank again to everyone who donated.
It's also probably worth noting, we're considering donations for the Software Transactional Memory project. You can read more about our plans

Cheers,
The PyPy Team

PyCon: Sign up for the PyCon 5K today!

2012-02-10T10:23:50+00:00

If you sign up by Monday with at least a $20 ticket, you'll get an awesome shirt designed by Idan Gazit. On top of being one of PyCon's great designers, he's giving two talks at the conference!

The inaugural race kicks off at 7 AM on Saturday March 10, right before the day's festivities begin. From marathon runners to casual runners to walkers - everyone is invited. Registration for the race is being handled by conference sponsor Eventbrite, but note that this is separate from your conference registration.

For payment we're using the "pay what you want" model. If you want to run, simply sign up - it can cost you as low as $0. If you choose the $20 registration (by Monday!) you’ll get a really nice shirt.

The best part of the race is that it's for charity. Feel free to use the "pay what you want" option to help us help a number of great charities.

Autism Speaks - Autism affects 1 in 110 children, 1 in 70 boys. Autism Speaks has grown into the nation's largest autism science and advocacy organization, dedicated to funding research into the causes, prevention, treatments and a cure for autism; increasing awareness of autism spectrum disorders; and advocating for the needs of individuals with autism and their families.

American Cancer Society - The American Cancer Society is a nationwide, community-based voluntary health organization dedicated to eliminating cancer as a major health problem. Headquartered in Atlanta, Georgia, the ACS has 12 chartered Divisions, more than 900 local offices nationwide, and a presence in more than 5,100 communities.

Epilepsy Foundation - The Epilepsy Foundation of America® is the national voluntary agency dedicated solely to the welfare of the almost 3 million people with epilepsy in the U.S. and their families. The organization works to ensure that people with seizures are able to participate in all life experiences; to improve how people with epilepsy are perceived, accepted and valued in society; and to promote research for a cure.

For more information, see https://us.pycon.org/2012/5k/.

Be sure to sign up and help us help some great organizations. Lace up your shoes - we'll see you at the starting line.

Davy Wybiral: Astro-Phys.com

2012-02-10T09:31:29+00:00

Today, Astro-Phys got a new JSON API for querying ephemerides. Enjoy!
http://www.astro-phys.com/api

Can now return planetary states in various units of measure and coefficients as either chebyshev or polynomial (also in various units).

Doug Latornell: CSV Downloads from Web Apps

2012-02-10T05:34:51+00:00

One of the users of an intranet app I maintain was using copy/paste to put data from the app into Excel. She asked if there was a better way. The pages she was copying from have tables of datastore objects, so adding a CSV download feature was an obvious solution. Providing that feature turned out to be pretty easy with the help of the StringIO and csv modules in the standard library.

While the code below is from a Pylons controller class, I can't see it being difficult to implement this in Django, Pyramid, or other web framework stacks.

The request handler for the CSV download looks like:

import cString
import csv

def csv_download(self, ...):
    csv_buffer = cStringIO.StringIO()
    csv_writer = csv.writer(csv_buffer)
    header = self._build csv_header(...)
    csv_writer.writerow(header)
    query_result = self.get_data_for_csv(...)
    for result in query_result:
        row = self._build_csv_row(result)
        csv_writer.writerow(row)
    content = csv_buffer.getvalue()
    csv_buffer.close()
    response.content_type = 'text/csv; charset=utf-8'
    response.content_disposition = (
        'attachment; filename="your_file.csv"')
    return content

This method uses StringIO to set up a file-like memory buffer, and instantiates a default CSV writer to write to the buffer. Next we build the header row and write it to the buffer. Then we get an iterator for the content that we want to write the the CSV file from the datastore, and format and write it to the buffer, one line at a time. Finally we get the CSV data from the buffer, set the response headers appropriately, and return the CSV data for download.

One thing I'm uncertain about: Is it necessary to explicitly call close method on a StringIO instance, or could I just do:

return csv_buffer.getvalue()

and let garbage collection take care of releasing the memory allocated for csv_buffer?

To get Excel to play nice with UTF-8 encoded data it's necessary to include 3 specific bytes as a Byte Order Mark (BOM) at the beginning of the file. I did that by prepending them to the heading string for the first column:

def _build_csv_header(self, ...)
    UTF_8_BOM = '\xef\xbb\xbf'
    header = [
        UTF_8_BOM + 'Column 1 Heading',
        ...
    ]
    return header

Building the content for each row of the CSV file is just a matter of formatting each query result into an array of strings. Fields containing non-ASCII characters stored as Unicode need to be encoded to UTF-8:

def _build_csv_row(self, result)
    row = [
        result.column_1_value,
        '{:%Y-%m-%d}'.format(result.some_date)
        ...
        result.unicode_value.encode('utf-8'),
        ...
    ]
    return row

I had an additional complication to deal with. The data for one of the CSV columns is stored in the database as HTML fragments that may contain non-ASCII characters. That data had to be rendered to Unicode before it could be added to the CSV row (encoded as UTF-8). It turns out that the Python standard library provides a fairly painless way of handling that complication too, but I'll leave that for another post.

EmptysquarePython: Slides from my talk on asynchronous web frameworks, Python, and MongoDB

2012-02-10T03:49:22+00:00

Here’s the slides for the talk I gave at the NYC Python Meetup tonight, on asynchronous web frameworks, Python, and MongoDB.

Direct link.

Embedded:

Python, async web frameworks, and MongoDB

View more presentations from emptysquare.