Planet Python

Tarek Ziade: virtualenv and zc.buildout now with Distribute included

2009-11-07T01:43:13+00:00

We are still actively working in fixing all the remaining bugs in Distribute (our Setuptools fork).

But we have reached an important milestone this week: both virtualenv and zc.buildout now comes with an option to switch to Distribute.

In virtualenv:

$ virtualenv --distribute ENV

In zc.buildout, using its bootstrap.py file:

$ python bootstrap.py --distribute

Enjoy !

For those who may wonder why they should switch to Distribute over Setuptools, it’s quite simple:

Distribute 0.6.x is a drop-in replacement for Setuptools
Distribute is actively maintained, and has over 10 commiters
Distribute 0.6.x offers Python 3 support !

And if you still struggle with packaging issues, the place to hang around to get some help is the #distutils IRC channel on Freenode.

Alex Gaynor: Towards a Better Template Tag Definition Syntax

2009-11-06T23:17:38+00:00

Eric Holscher has blogged a few times this month about various template tag definition syntax ideas. In particular he's looked at a system based on Surlex (which is essentially an alternate syntax for certain parts of regular expressions), and a system based on keywords. I highly recommend giving his posts a read as they explain the ideas he's looked at in far better detail than I could. However, I wasn't particularly satisfied with either of these solution, I love Django's use of regular expressions for URL resolving, however, for whatever reason I don't really like the look of using regular expressions (or an alternate syntax like Surlex) for template tag parsing. Instead I've been thinking about an object based parsing syntax, similar to PyParsing.

This is an idea I've been thinking about for several months now, but Eric's posts finally gave me the kick in the pants I needed to do the work. Therefore, I'm pleased to announce that I've released django-kickass-templatetags. Yes, I'm looking for a better name, it's already been pointed out to me that a name like that won't fly in the US government, or most corporate environments. This library is essentially me putting to code everything I've been thinking about, but enough talking let's take a look at what the template tag definition syntax:


    @tag(register, [Constant("for"), Variable(), Optional([Constant("as"), Name()])]):
    def example_tag(context, val, asvar=None):

As you can see it's a purely object based syntax, with different classes for different components of a template tag. For example this would parse something like:


    {% example_tag for variable %}
    {% example_tag for variable as new_var %}

It's probably clear that this is significantly less code than the manual parsing, manual node construction, and manual resolving of variable that you would have needed to do with a raw templatetag definition. Then the function you have gets the resolved values for each of its parameters, and at that point it's basically the same as Node.render, it is expected to either return a string to be inserted into the template or alter the context. I'm looking forward to never writing manual template parsing again. However, there are still a few scenarios it doens't handle, it won't handle something like the logic in the {% if %} tag, and it won't handle tags with {% end %} type tags. I feel like these should both be solvable problems, but since it's a bolt-on addition to the existing tools it ultimately doesn't have to cover every use case, just the common ones (when's the last time you wrote your own implementation of the {% if %} or {% for %} tags).

It's my hope that something like this becomes popular, as a) developers will be happier, b) moving towards a community standard is the first step towards including a solution out of the box. The pain and boilerplate of defining templatetags has long been a complain about Django's template language (especially because the language itself limits your ability to perform any sort of advanced logic), therefore making it as painless as possible ultimately helps make the case for the philosophy of the language itself (which I very much agree with it).

In keeping with my promise I'm giving an overview of what my next post will be, and this time I'm giving a full 3-day forecast :). Tommorow I'm going to blog about pip, virtualenv, and my development workflow. Sunday's post will cover a new optimization that just landed in Unladen Swallow. And finally Monday's post will contain a strange metaphor, and I'm not saying any more :P. Go checkout the code and enjoy.

Shannon -jj Behrens: Suicide

2009-11-06T22:15:40+00:00

My friend committed suicide. He was a brilliant programmer, and he had everything going for him. He was very successful.

I'm crushed because I know I could have helped if only he had given me a chance. He never did.

We in the programming world aren't always the most emotionally balanced. I know of three others who took their lives in the programming world. I've hinted at this before on my Bipolar Lisp Programmer post. To compound matters, our society has been moving away from personal interaction and responsibility for decades, leading to a culture that is toxic.

Mother Theresa said that the greatest poverty that she ever saw was to see people who felt unloved. If your friends are feeling unloved, please reach out to them. We are each far more loved than we think. In the programming world, it's so easy to get caught up in petty struggles, like Pylons vs. Django, Ruby vs. Python, free software vs. open source, Linux vs. pretty much everything else ;) What we forget is that we're all people with hopes and dreams, fears and insecurities. We're all trying to change the world, but as Mother Theresa said, "In this life we cannot do great things. We can only do small things with great love."

Look out for each other, but if you have no where to turn, contact me! Call me directly at (925) 209-6439. Don't consider the middle of the night an inconvenience. When else am I going to work with five kids around ;) I will do everything I can to help. I don't want to wake up on another day to find another brilliant mind missing from this great community of thinkers. We are a community and we are only as great as the individuals that thrive and share with one another in that community. Everyone contributes, whether in small ways or large, and every loss is felt by more people than we consider in our moments of darkness.

Logilab: First contact with pupynere

2009-11-06T20:38:59+00:00

I spent some time this week evaluating Pupynere, the PUre PYthon NEtcdf REader written by Roberto De Almeida. I see several advantages in pupynere.

First it's a pure Python module with no external dependency. It doesn't even depend on the NetCDF lib and it is therefore very easy to deploy.

Second, it offers the same interface as Scientific Python's NetCDF bindings which makes transitioning from one module to another very easy.

Third pupynere is being integrated into Scipy as the scypi.io.netcdf module. Once integrated, this could ensure a wide adoption by the python community.

Finally it's easy to dig in this clear and small code base of about 600 lines. I have just sent several fixes and bug reports to the author.

However pupynere isn't mature yet. First it seems pupynere has been only used for simple cases so far. Many common cases are broken. Moreover there is no support for new NetCDF formats such as long-NetCDF and NetCDF4, and important features such as file update are still missing. In addition, The lack of a test suite is a serious issue. In my opinion, various bugs could already have been detected and fixed with simple unit tests. Contributions would be much more comfortable with the safety net offered by a test suite. I am not certain that the fixes and improvements I made this week did not introduce regressions.

To conclude, pupynere seems too young for production use. But I invite people to try it and provide feedback and fixes to the author. I'm looking forward to using this project in production in the future.

Python Secret Weblog: The Success of the Zope Component Architecture

2009-11-06T20:33:52+00:00

Prompted by recent brief negative pronunciations by Malthe on the Zope Component Architecture (ZCA), I thought I'd talk a bit about what I think about it. I'm not going to go into hermeneutics here of what Malthe might mean -- others attempts at exegesis exist in the comments to that blog entry already. Instead, I'll just talk about what I think makes the ZCA useful, and why it is successful. Finally I'll go into some reasons why people are frustrated by the ZCA.

What is the ZCA used for?

What is the ZCA used for? It's used to glue things to each other: glue event handlers to events, glue views to models, glue plugins into applications and libraries, and more abstractly, glue adapters to adaptees. In ZCA terms, providing such glue is termed providing configuration.

Why was the ZCA created? The Zope community had been building pluggable web applications for a long time and we noticed our components became overly complex and were hard to glue together and override. The ZCA is one answer to this problem.

The ZCA is implemented by zope.component. It's a library for gluing. It's built on top of zope.interface, a library that helps one define the bits that are being glued.

What advantage does such gluing bring?

many, perhaps all, larger applications contain glue. The ZCA makes the glue explicit and uniform.
ZCA glue can be overridden explicitly.
you can extend existing systems by gluing in new things.

One place where the ZCA is helpful is when you want to write a library that offers a few plugin points to configure it for a particular environment. For instance when I wrote hurry.resource, a library for handling javascript and other resources, I included a few plugin points in it that allow it to be plugged into a particular web framework. Then to allow it to be used with Zope Toolkit-based frameworks such as Grok, I wrote hurry.zoperesource to provide the knowledge about that.

This way, hurry.resource doesn't need to know anything in particular about URL generation or requests; its plugins can take care of this. This allowed me to write and test hurry.resource without worrying too much about the larger Zope Toolkit framework, knowing I could plug it in later, and now the library becomes more useful for a broader group of people.

The ZCA doesn't just allow one to glue one thing to another, but also to override the glue in specific cases. A common example of overriding glue occurs with views. Zope Toolkit applications follow a view/model approach, where the view is looked up on the model dependent on its class (or interface). It happens frequently enough in an application that I want many models to share a particular view, but override one model with a more specialized view. This is much like the way inheritance works in plain Python: I implement a method on a base class shared completely by some subclasses, but for one subclass I'd like to override it.

What I described just now is overriding for particular subclasses. The ZCA also allows other ways of overriding based on the zope.configuration library, overriding one glue registration with another one. I myself find myself using this kind of override less frequently, but it's still very useful in the 1 percent of cases where other options would be very ugly.

The question is sometimes asked why not just modify code dynamically for overrides? Why not monkey patch it? (or "open the class", or whatever other terminology one would like to use).

Brandon Rhodes at PyCon 2008 gave a great presentation explaining why the ZCA approach can be superior to monkey-patching and some other approaches. The point he makes is that the ZCA is the most composable and maintainable approach of the alternatives (subclassing, mixins, monkey-patching, adaptation). I recommend everyone interested in this topic to read his slides.

Of course as with everything in programming, everything depends on the trade-offs. In many circumstances the ZCA is overkill. The ZCA can be misused. But I also maintain that in many circumstances the ZCA is very useful.

Component approaches have become quite popular with web frameworks. Many popular ones adopt elements of it, but often in a somewhat limited way. For instance, one very popular interface is the WSGI interface, and one popular form of adaptation of this WSGI interface is to use WSGI middleware and framework components. With WSGI we see that just one well-defined, consensus interface has become an amazing source of creativity and pluggability within web development. With the ZCA we are able to define more than one interface in our applications, and potentially create ecosystems of creativity around those. Not that this is easy, but at least we have a mechanism and method to do so.

The ZCA is successful

Why do I say the Zope Component Architecture (ZCA) is successful? It's successful as it's being used, by many people, for many years now. Of course you can say it's only used by that weird Zope community, and a bit by weird Twisted people as well perhaps. That's fine, but realize that the wider Zope community is big and is made of many parts: Plone, Silva, Zope 2, Zope Toolkit, Grok. I'll also count BFG as part of the wider Zope community.

The ZCA is successful for me. Without it, I'd have to invent something very much like it. It comes back in much code that I've written. I'm able to do all kinds of small, cool things in my applications and libraries on a daily basis, and I'm creating more reusable code as a result.

Frustration with the ZCA

The ZCA isn't perfect. Humans aren't perfect, either. It's not easy to create good reusable interfaces, or to build pluggable frameworks and applications. It's easy to overdo the pluggability. It's easy to create the wrong pluggability points. It can get overly complex and overwhelming.

Writing XML configuration files to glue things together can be cumbersome and lead to repeating yourself, though his last issue has been overcome for some time now by the Grok project and its reusable solution (grokcore.component) is available to everybody.

It's easy to let the pain of the mistakes in the use of the ZCA, and the pain of complex applications in which it is used in general, overshadow the many successes of the ZCA: in a large part thanks to to the ZCA's ability to glue things together people are able to use Zope Toolkit libraries within a Zope 2 or Plone context. Grok was able to remix the Zope Toolkit in part because we could easily modify and extend the framework. I think the web frameworks that use the ZCA offer pluggability and flexibility unrivaled by the competition.

We generally don't say that the problems and complexities in our large libraries and applications are due to the Python programming language. While Python isn't perfect, we tend to think overall it helps. I also tend to think that the ZCA helps.

Are there alternatives to the ZCA? Certainly: any library or application specific pluggability system is an alternative. Sometimes a specific approach is better than a generic one. It can be easier to understand it's smaller.

Usually I'd say it's the other way around. Each custom pluggability system is another one to learn, and has its own limitations. I've found multiple times that I can make a far more powerful pluggability system for a library in a shorter time if I build it on top of the ZCA.

There are also more general approaches where there's at least some overlap with the ZCA: setuptools entry points, WSGI middleware, and PEAK-Rules. To use any of them for pluggability and overriding one needs to have a notion of an interface one can implement and plug into, a notion that the ZCA makes explicit.

Conclusions

The ZCA is useful. The ZCA is powerful. The ZCA is successful. The ZCA is imperfect. When I've run into its imperfections I've helped build solutions on top of it, such as Grok's in-python gluing approach. Sometimes I've considered its use overkill and gone for some other approach. I think it's useful to step back and consider alternative approaches. But let's consider them in the light of the success of the ZCA as much as in the light of its flaws.

Greg Wilson: Misa Sakamoto on DB2 Technology Explorer

2009-11-06T19:31:00+00:00

One of our PEY students, Misa Sakamoto, has an article up on DeveloperWorks about the stuff she’s doing at IBM — yay!

Jean-Paul Calderone: Twisted Web in 60 seconds: HTTP authentication

2009-11-06T15:08:26+00:00

Welcome to the 14th installment of "Twisted Web in 60 seconds". In many of the previous installments, I've demonstrated how to serve content by using existing resource classes or implementing new ones. In this installment, I'll demonstrate how you can use Twisted Web's basic or digest HTTP authentication to control access to these resources.

Guard, the Twisted Web module which provides most of the APIs which will be used in this example, helps you to add authentication and authorization to a resource hierarchy. It does this by providing a resource which implements getChild to return a dynamically selected resource. The selection is based on the authentication headers in the request. If those headers indicate the request is made on behalf of Alice, then Alice's resource will be returned. If they indicate it was made on behalf of Bob, his will be returned. If the headers contain invalid credentials, an error resource is returned. Whatever happens, once this resource is returned, URL traversal continues as normal from that resource.

The resource which implements this is HTTPAuthSessionWrapper, though it is directly is directly responsible for very little of the process. It will extract headers from the request and hand them off to a credentials factory to parse them according to the appropriate standards (eg HTTP Authentication: Basic and Digest Access Authentication) and then it hands the resulting credentials object off to a portal, the core of Twisted Cred, a system for uniform handling of authentication and authorization. I am not going to discuss Twisted Cred in much depth here. To make use of it with Twisted Web, the only thing you really need to know is how to implement a realm.

You need to implement a realm because the realm is the object which actually decides which resources are used for which users. This can be as complex or as simple as it suitable for your application. For this example, I'll keep it very simple: each user will have a resource which is a static file listing of the public_html directory in their UNIX home directory. First, I need to import implements from zope.interface and IRealm from twisted.cred.portal. Together these will let me mark this class as a realm (this is mostly - but notentirely - a documentation thing). I'll also need File for the actual implementation later.

  from zope.interface import implements

  from twisted.cred.portal import IRealm
  from twisted.web.static import File

  class PublicHTMLRealm(object):
      implements(IRealm)

A realm only needs to implement one method, requestAvatar. This is called after any successful authentication attempt (ie, Alice supplied the right password). Its job is to return the avatar for the user who succeeded in authenticating. An avatar is just an object that represents a user. In this case, it will be a File. In general, with Guard, the avatar must be a resource of some sort.

      def requestAvatar(self, avatarId, mind, *interfaces):
          if IResource in interfaces:
              return (IResource, File("/home/%s/public_html" % (avatarId,)), lambda: None)
          raise NotImplementedError()

A few notes on this method:

The avatarId parameter is essentially the username. It's the job of some other code to extract the username from the request headers and make sure it gets passed here.
The mind is always None when writing a realm to be used with Guard. You can ignore it until you want to write a realm for something else.
Guard always passed IResource for the interfaces parameter. If interfaces only contains interfaces your code doesn't understand, raising NotImplementedError is the thing to do, as above. You'll only need to worry about getting a different interface when you write a realm for something other than Guard.
If you want to track when a user logs out, that's what the last element of the returned tuple is for. It will be called when this avatar logs out. lambda: None is the idiomatic no-op logout function.
Notice that I have written the path handling code in this example very poorly. This example may be vulnerable to certain unintentional information disclosure attacks. This sort of problem is exactly the reason FilePath exists. However, that's an example for another day...

We're almost ready to set up the resource for this example. To create an HTTPAuthSessionWrapper, though, we need two things. First, a portal, which requires the realm above, plus at least one credentials checker:

  from twisted.cred.portal import Portal
  from twisted.cred.checkers import FilePasswordDB

  portal = Portal(PublicHTMLRealm(), [FilePasswordDB('httpd.password')])

FilePasswordDB is that credentials checker I mentioned. It knows how to read passwd(5)-style (loosely) files to check credentials against. It is responsible for the authentication work after HTTPAuthSessionWrapper extracts the credentials from the request.

Next we need either BasicCredentialFactory or DigestCredentialFactory. The former knows how to challenge HTTP clients to do basic authentication; the latter, digest authentication. I'll use digest here:

  from twisted.web.guard import DigestCredentialFactory

  credentialFactory = DigestCredentialFactory("md5", "example.org")

The two parameters to this constructor are the hash algorithm and the http authentication realm which will be used. The only other valid hash algorithm is "sha" (but be careful, MD5 is more widely supported than SHA). The http authentication realm is mostly just a string that is presented to the user to let them know why they're authenticating (you can read more about this in the RFC).

With those things created, we can finally instantiate HTTPAuthSessionWrapper:

  from twisted.web.guard import HTTPAuthSessionWrapper

  resource = HTTPAuthSessionWrapper(portal, [credentialFactory])

There's just one last thing that needs to be done here. When I introduced rpy scripts, I mentioned that they're evaluated in an unusual context. This is the first example which actually needs to take this into account. It so happens that DigestCredentialFactory instances are actually stateful. Authentication will only succeed if the same instance is used to generate challenges and examine the responses to those challenges. However, the normal mode of operation for an rpy script is for it to be re-executed for every request. This leads to a new DigestCredentialFactory being created for every request, preventing any authentication attempt from ever succeeding.

There are two ways to deal with this. First, the better of the two ways, I could move almost all of the code into a real Python module, including the code which instantiates the DigestCredentialFactory. This would make ensure the same instance was used for every request. Second, the easier of the two ways, I could add a call to cache to the beginning of the rpy script:

  cache()

cache is part of the globals of any rpy script, so you don't need to import it (it's okay to be cringing at this point). Calling cache makes Twisted re-use the result of the first evaluation of the rpy script for subsequent requests too. Just what I want in this case.

Here's the complete example (with imports re-arranged to the more conventional style):

cache()

from zope.interface import implements

from twisted.cred.portal import IRealm, Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.web.static import File
from twisted.web.resource import IResource
from twisted.web.guard import HTTPAuthSessionWrapper, DigestCredentialFactory

class PublicHTMLRealm(object):
    implements(IRealm)

    def requestAvatar(self, avatarId, mind, *interfaces):
        if IResource in interfaces:
            return (IResource, File("/home/%s/public_html" % (avatarId,)), lambda: None)
        raise NotImplementedError()

portal = Portal(PublicHTMLRealm(), [FilePasswordDB('httpd.password')])

credentialFactory = DigestCredentialFactory("md5", "localhost:8080")
resource = HTTPAuthSessionWrapper(portal, [credentialFactory])

And voila, a password-protected per-user Twisted Web server.

I've gotten several requests to write something about sessions, so there's a good chance that's what you'll find in the next installment.

Patrick Stinson: An interesting thread problem

2009-11-06T12:55:14+00:00

Interesting problem

I am trying to flesh out the thread locking for our embedded python interpreter. A user can write scripts for our application, and if a script defines a callback it can respond to events from the application (like a midi note or musical clock tick).

As it stands, we are not using process migration to allow for true multiprocessing, but instead are just hamming a single lock from multiple threads to protect the CPython library. This is good enough for now - it works quite well at relatively low latencies.

I compiled python without thread management to avoid an incredible number of 'NULL tstate' errors caused by unpredictable thread management by sequencing host applications. The app is solid, but now I want to integrate a simple stack frame mechanism to allow scripts to behave a little better when calling other scripts. Short intro, but the details don't matter anyway...

Here's the deal

I have one audio engine per thread, and several scripts per engine. The currently executing script is stored as a pointer on the engine object using my function setActiveScript(), and when a new script is called it is swapped with the current one, much like PyThreadState_Swap(). This in effect creates a stack of scripts in the engine, where when execution returns to the original script NULL is passed to setActiveScript().

This works great, but when there is more than one engine (and so more than one thread), I need to acquire a critical section lock and swap the engine for another one, while maintaining the current script on that engine. The script pointer is stored in the engine, but where to enter and exit the critical section is tough. Hairy.

The function looks like this:

Script *ScriptingApplication::instance()->setActiveScript(Script *)

I haven't written much complex code in the last while, so lining out the algorithm and thread locks is kind of clunky at best. it doesn't help that Ableton Live (a host that loads our audio plugin) suddenly doesn't want to be debugged.

*crash*. uggh.

Simon Willison: Python in the Scientific World

2009-11-06T11:04:04+00:00

Python in the Scientific World. Python continues to make strides in the scientific world—and the Hubble Space Telescope team have been using it for 10 years!

S. Lott: BBEdit Configuration

2009-11-06T10:07:37+00:00

After installing Python 2.6 in Mac OS X, I had problems with BBEdit not finding the right version of Python. It kept running an old 2.5 version.

I finally tracked down the BBEdit documentation, http://pine.barebones.com/manual/BBEdit_9_User_Manual.pdf.

Found this: "BBEdit expects to find Python in /usr/bin, /usr/local/bin, or /sw/bin. If you have installed Python elsewhere, you must create a symbolic link in /usr/local/bin pointing to your copy of Python in order to use pydoc and the Python debugger."

Checked in /usr/bin and found an old Python there. I think Fink did that. Removed it and BBEdit is much happier. As is Komodo Edit.

Thomas Vander Stichele: Dedicated separate Firefox windows

2009-11-06T09:59:59+00:00

Dear intarweb,

here’s what I’d like to be able to do. I would like to start a completely separate Firefox window, in a separate process, with a given webpage. This process should be completely separate from my regular browsing, not take new links in its window when I click links somewhere else (usually they go to the most recently opened window), not crash when the regular firefox process crashes, and not bog down because my regular firefox goes to 100% CPU and beyond.

It seems to be hard to google for this idea; is it possible ?

Ludvig Ericson: Multitouch on Unibody MacBooks

2009-11-06T09:56:18+00:00

A little known fact about the unibody touchpad is that it can distinguish between up to eleven touch points - i.e., fingers.

Another little known fact is that the sensor knows about the contact surface area (in terms of ellipsoids).

Another yet less known fact is that Johan Nordberg didn't sleep much last night, and wrote a neat OS X application which simply plots what the trackpad (or Magic Mouse!) is saying about touches.

Download blä.se/FingerMgmt.zip to try it out.

delicious.com

S. Lott: Parsing HTML from Microsoft Products (Like Front Page, etc.)

2009-11-06T09:53:15+00:00

Ugh. When you try to parse MS-generated HTML, you find some extension syntax that is completely befuddling.

I've tried a few things in the past, none were particularly good.

In reading a file recently, I found that even Beautiful Soup was unable to prettify or parse it.

The document was filled with  constructs that looked vaguely directive or comment-like, but still managed to stump the parser.

The BeautifulSoup parser has a markupMassage parameter that applies a sequence of regexps to the source document to cleanup things that are baffling. Some things, however, are too complex for simple regexp's. Specifically, these nested comment-like things were totally confusing.

Here's what I did. I wrote a simple generator which emitted the text that was unguarded by these things. The resulting sequence of text blocks could be assembled into a document that BeautifulSoup could parse.


def clean_directives( page ):
"""
Stupid Microsoft "Directive"-like comments!
Must remove all <!--[if...]>...<![endif]--> sequences.  Which can be nested.
Must remove all <![if...]>...<![endif]> sequences.  Which appear to be the nested version.
"""
if_endif_pat= re.compile(  r"(\<!-*\[if .*?\]\>)|(<!\[endif\]-*\>)" )
context= []
start= 0
for m in if_endif_pat.finditer( page ):
   if "[if" in m.group(0):
       if start is not None:
           yield page[start:m.start()]
       context.append(m)
       start= None
   elif "[endif" in m.group(0):
       context.pop(-1)
       if len(context) == 0:
           start= m.end()+1
if start is not None:
   yield page[start:]

Thomas Vander Stichele: Best post-sale screen ever

2009-11-06T09:24:09+00:00

Just gave in to the geeky side of the force this morning and bought a DVD documentary about BBS’s. (the version I bought comes with a DVD full with BBS text files, ANSI art, and random crap.)

After buying, this is the screen I got:

I don’t think I ever got a chuckle out of what happened right *after* I forked over cash. Here’s to you, Bob!

Ned Batchelder's blog: Continuous integration for Python with Hudson

2009-11-06T02:26:32+00:00

Continuous integration is a great idea: you configure a server to periodically pull your source code, build it, run the tests, run lint, measure coverage, and so on. Then it graphs everything, stores the results for examination, and so on.

I'd been trying to figure out how to use the Hudson CI server with Python, and the few times I tried to get my mind around it, it just wasn't clicking. I happened to mention my mental block to Joe Heck, and a few days later, he produced Setting up a python CI server with Hudson. It's a great step-by-step how-to covering everything you need to get Hudson going for a Python project.

Running through his guide finally cleared the last misconception for me: continuous integration isn't a build tool or a test runner. You don't run Hudson on your development machine. Sounds silly, but something needed to clear it up, and this was it.

Thanks, Joe!

PyPy Development: Düsseldorf Sprint Started

2009-11-06T01:25:48+00:00

The Düsseldorf sprint starts today. Only Samuele and me are there so far, but that should change over the course of the day. We will mostly work on the JIT during this sprint, trying to make it a lot more practical. For that we need to decrease its memory requirements some more and to make it use less aggressive inlining. We will post more as the sprint progresses.

Graham Dumpleton: A roadmap for the Python WSGI specification.

2009-11-06T00:43:34+00:00

EDIT: Note that the discussion around this on the Python WEB-SIG went round in many circles and died. Thus nothing much here came into being and thus should be viewed merely for amusement value and not taken as being any indicator of any sort of reality.The discussion on the Python WEB-SIG about what to do about WSGI and Python 3.X has been going round and round in circles for some time now

Eric Holscher: Adding testing to pip

2009-11-06T00:38:04+00:00

Python packaging has been in a bit of a state of disarray for as long as I've been using it. Pip has come along to make installing python packages easier. It has a lot of features that are useful, but they have been talked about in many other blog posts.

Today I want to talk about adding testing to pip. If you are familiar with the Perl community, then you probably know about CPAN. It is basically Pypi for Perl. They have a command, called cpan, which allows you to install packages in a similar way to pip.

One of the steps that a package goes through before being installed on your system is that the tests are run. This allows you to know if the package that you have installed is actually going to work on your system. It may be broken on your platform, or you may be missing a library that it thought you had. Currently, pip has no way to test packages when they are being installed. I went looking for a way to make that happen.

It should be noted that pip is based on setuptools. Setuptools is what parses and understands most of the logic inside of your setup() function in the setup.py for your project (which you have, right?). Setuptools has an option called test_suite, which allows you to run setup.py test on your package, and have it run the unit tests. This is done by calling whatever python function is defined in test_suite.

I added the ability for pip to run setup.py test on a package that it is installing. It is executed by running pip install --test <package>. The implementation is on a ticket on bitbucket, and in a repository on github.

If you want to check it out, go ahead and clone my repo and check out the test_command branch. Then you can simply run

python pip.py install --test wsgiref

for an example package. If a package doesn't have a test_suite, then it simply doesn't run anything.

Note that if the tests fail, it doesn't impact the installation of the package. The python community's tests aren't quite good enough,and almost any Django package you try this on will not have any tests. I wrote about how to add testing to your django package, but the process is long and involved. I'm working to improve the situation for Django and hopefully having the ability to run tests in the package management tool will spur people to add testing ability to their setup scripts!

Nose makes this really easy, by simply adding test_suite = 'nose.collector' to your setup.py, nose will run your tests correctly. This is the level of support that I am hoping to implement for Django.

On a side node, I talked to Ian Bicking about this, and he suggested writing the test command as a separate command, so you would be able to do pip test wsgiref, if it was installed. This has some other problems, which I will talk about after I have implemented this functionality.

I would love to hear feedback, or if anyone has ideas for improving testing in the python and django communities. I have lots of ideas, and I will be writing more of them up over the following weeks.

Python News: SciPy India Call for Papers

2009-11-05T23:46:00+00:00

SciPy India 2009 -- Call for Presentations The first Scientific Computing with Python conference in India has announced its Call for Presentations. The conference will be held at the Technopark in Trivandrum, Kerala, India from December 12th to 17th, 2009.

Malthe Borch: Emperor's new clothes

2009-11-05T23:29:00+00:00

The Zope Component Architecture is often touted for letting you override so-called components which have already been registered elsewhere (by a library, out of your immediate control).

It's wrong (edit: to clarify, using the overriding facility of the ZCA is wrong, and by extension, to use the ZCA because you can override components, is wrong too; I do not mean to suggest that the use of ZCA is misguided in general).

Shrink-wrap software is a circumstance. If you must change it from outside (instead of branching, which is a fine approach in many cases), just go ahead and update the functions or classes in question using the reload approach (object identity changes should be avoided at all costs).

There's nothing wrong with ZCA; it's in my view a misunderstanding to employ it for web application development at large. Rather, it's applicable for deployments that require live software component introspection. I don't know that web applications need this in general. What happens is that frameworks like Zope Toolkit and Plone use these introspection features to map requests to code. That's a bit like using RDIF tags for consumption tracking (hint: require credit card payment or identity-based rebate instead).

Most software is misunderstood. How to use it, when to use it. Sometimes it's worth the while to retrace your steps to learn why you ended up using a particular software.

Alex Gaynor: The Pycon Program Committee and my PyCon Talk

2009-11-05T23:09:02+00:00

Last year I presented at a conference for the first time in my life at PyCon, I moderated a panel on ORMs and I enjoyed it a ton (and based on the feedback I've gotten at least a few people enjoyed attending it). Above and beyond that the two PyCons I've attended have both been amazing conferences, tons of awesome talks, great people to hang out with, and an awesome environment for maximizing both. For both of the last two years I've hung around the PyCon organizers mailing list since the conference was in Chicago and I lived there, however this year I really wanted to increase my contributions to such a great conference. Therefore, I joined the Pycon program committee. This committee is responsible for reviewing all talk submissions and selecting the talks that will ultimately appear at PyCon.

This year the PyCon programming committee had a really gargantuan task. There were more talks submitted then ever before, more than 170 of them, for only 90 or so slots. Unfortunately this meant that we had to reject some really good talks, which always sucks. There's been a fair bit of discussion about the process this year and what can be done to improve it. As a reviewer the one thing I wish I'd known going in was that the votes left on talks were just a first round, and ultimately didn't count for a huge amount. Had I known this I would have been less tepid in giving positive reviews to talks which merely looked interesting.

Another hot topic in the aftermath is whether or not the speaker's identity should factor into a reviewer's decision. My position is that it should wherein the speaker has a reputation, be it good or bad. If I know a speaker is awesome I'm way more likely to give them the +1, likewise if I see a speaker has a history of poor talks I'm more likely to give them a -1. That being said I don't think new speakers, or slightly inexperienced ones should be penalized for that, I was a brand new speaker last time and I'm grateful I was given the chance to present.

To give an example of this one of the talks I'm really looking forward to is Mark Ramm's, "To relate or not to relate, that is the question". Mark and I spoke about this topic for quite a while at PyOhio, and every talk from Mark I've ever seen has been excellent. Therefore I was more than willing to +1 it. However, had I not known the speaker it would still have been a good proposal, and an interesting topic, I just would not have gotten immediately excited about going to see the talk.

As an attendee one of the things I've always found is that speakers who are very passionate about their topics almost always give talks I really enjoy. Thinking back to my first PyCon Maciej Fijalkowski managed to get me excited and interested in PyPy in just 30 minutes, because he was so passionate in speaking about the topic.

All that being said I wanted to share a short list of the talks I'm excited about this year, before I dive into what my own talk will be about:

Optimizations And Micro-Optimizations In CPython
Unladen Swallow: fewer coconuts, faster Python
Managing the world's oldest Django project
Understanding the Python GIL
The speed of PyPy
Teaching compilers with python
To relate or not to relate, that is the question
Modern version control: Mercurial internals
Eventlet: Asynchronous I/O with a synchronous interface
Hg and Git : Can't we all just get along?
Mastering Team Play: Four powerful examples of composing Python tools

It's a bit of a long list, but compared to the size of the list of accepted talks I'm sure there are quite a few gems I've missed.

The talk I'm going to be giving this year is about the real time web, also known as HTTP push, Comet, or reverse Ajax. All of those are basically synonyms for the server being able to push data to the browser, rather than having the browser constantly poll the server for data. Specifically I'm going to be looking at my experience building three different things, LeafyChat, DjangoDose's DjangoCon stream, and Hurricane.

Leafychat is an IRC client built for the DjangoDash by myself, Leah Culver, and Chris Wanstrath. The DjangoDose DjangoCon stream was a live stream of all the Twitter items about DjangoCon that Eric Florenzano and I built in the week leading up to DjangoCon. Finally, Hurricane is the library Eric Florenzano and I have been working on in order to abstract the lessons learned from our experience building "real time" applications in Python.

In the talk I'm going to try to zero in on what we did for each of these projects, what worked, what didn't, and what I'm taking away from the experience. Finally, Eric Florenzano and I are working to put together a new updated, better version of the DjangoCon stream for PyCon. I'm going to discuss what we do with that project, and why we do it that way in light of the lessons of previous projects.

I'm hoping both my talk, and all of them will be awesome. One thing's for sure, I'm already looking forward to PyCon 2010. Tomorrow I'm going to be writing about my thoughts on a more ideal template tag definition syntax for Django, and hopefully sharing some code if I have time to start working on it. See you then (and in Atlanta ;))!

Rene Dudfield: rakarrack is a decent effects rack for linux

2009-11-05T22:01:30+00:00

rakarrack is a decent effects rack for linux.

It's not packaged for ubuntu(of course), but is fairly easy to compile. I've been using the cvs version (yes, people are still quite happily using cvs it seems).

Once you get it set up you have access to 80 presets of realtime guitar effects(2-40ms depending on your computer sound card setup).

It works with the jack audio system, so you can route audio into it, and route its audio out to other programs easily enough. You can also control it with midi(alsa, or jack). Meaning you can hook it up to a midi controller of some sort. I've been using it with an maudio axiom 25 controller, and with python scripts via pygame.midi.

It even has a very nice help manual integrated into the program... I wish more programs had a good help section. It details the 17 different effects that come with the program.

One thing which is a bit funny is that it doesn't use LADSPA or lv2 plugins. You can of course still use these other plugins through other jack programs if you want. However the ones it comes with are all fairly good quality from my tinkering. There is a lot you can do with the various plugins in different orders with different settings in each one. The 80 presets show a good variations of sounds you can get out of it.

I've been using it in conjunction with the mixxx mixing program and the hydrogen drum machine. So you don't need to use it just with guitars... any program or audio input will do.

If you are using guitars it also has a tuner which might come in handy... if you've never quite got the hang of playing that song called Tuning that you hear at the start of every gig.

Patrick Stinson: Pure Art

2009-11-05T21:38:31+00:00

There comes a point when a piece of art is so refined that the products, byproducts, and even messiest and most defiled anty-products of that piece of art look good. The old rule that the back of the cross-stich should look as nice as the front applies throughout all kinds of art, whether it's computer programming, music, or athletic prowess.

You've got your face deep down in the last algorithm that will finish your crowning achievement of the last nine months of work. The goal is met, the task is complete, but the project is a mess.

Now its done, and you start polishing and packaging it up, writing API docs, making CD art, distributing re-usable instrument tracks for the rest of the world to see - the piece is packaged and ready to ship.

Ableton live is one of those pieces of fine art, where the purity of the design lends itself to producing beautiful musical sets, even among some of the crappiest work you've done. The layout of the clips, the coloring and grouping of the built-in tools, everything can end up being a work of art in it's own, not just the final audio dump that you'll stick on your new album.

Go download the free Múm set on Ableton's site and check out the work that they've done. If you don't have Live just download the free demo.

First off, everything sounds amazing. Any clip sounds good with any clip. Even a monkey could play a nice sounding song here. This is the meaning of the back of the cross-stitch work. The loops are clean, refined, polished, done.

They also spelled their name with inactive tracks down at the bottom of the set, and spelled their name again in midi notes in each of those tracks. I know this is poking at nerdy details, but this set just emanates the very cleanliness and artistic presentation that such a flexible tool makes possible. It's just a feeling I get, man.

This software makes me want to be creative, whether I like it or not. Not only that, but it plainly shows the way to easy and readily available creativity, without having to wade through the much of documentation and technical cribby-crap.

Ableton have nailed the solution to the music problem. They have nailed it, and then shown us what we didn't know we wanted to do in the first place. They've redefined the industry. While people preach this crap all the time, you almost never really see it.

Because of this, Ableton Live is the best piece of software out there.

S. Lott: Stored Procedures and Ad Hominem Arguments

2009-11-05T21:07:14+00:00

The question of "Stored Procedures and Triggers" comes up fairly frequently.

Over the years (since the 90's, when stored procedures were introduced to Oracle) I've learned precisely how awful a mistake this technology is.

I've seen numerous problems that have stored procedures as their root cause. I'll identify just a few. These are not "biases" or "opinions". These are experience.

The "DBA as Bottleneck" problem. In short, the DBA's take projects hostage while the development team waits for stored procedures to be written, corrected, performance tuned or maintained.
The "Data Cartel" problem. The DBA's own parts of the business process. They refuse (or complicate) changes to fundamental business rules for obscure database reasons.
The "Unmaintainability" problem. The stored procedures (and triggers) have reached a level of confusion and complexity that means that it's easier to drop the application and install a new one.
The "Doesn't Break the License" problem. For some reason, the interpreted and source-code nature of stored procedures makes them the first candidate for customization of purchased applications. Worse, the feeling is that doing so doesn't (or won't) impair the support agreements.

When I bring these up, I wind up subject to weird ad hominem attacks.

I've been told (more than once) that I'm not being "balanced" and that stored procedures have "There are pros and cons on both sides". This is bunk. I have plenty of facts. Stored procedures create a mess. I've never seen any good come from stored procedures.

I don't use GOTO's haphazardly. I don't write procedural spaghetti code. No one says that I should be more "balanced."

I don't create random database structures with 1NF, 2NF and 3NF violations in random places. No one says I should be more "balanced".

Indeed, asking me to examine my bias is an ad hominem argument. My fact-based experience with stored procedures is entirely negative.

But when it comes to stored procedures, there's a level of defensiveness that defies my understanding. I assume Oracle, IBM and Microsoft are paying kickbacks to DBA's to support stored procedures and PL/SQL over the more sensible alternatives.

Menno's Musings: Why I chose Disqus Comments

2009-11-05T19:49:00+00:00

I recently moved my blog comments away the PyBlosxom comments plugin to a hosted system. The main driver was the ability for people to subscribe to comments for an article using email or RSS. It's a pain for people to have to check back to the site to see if someone has replied to their comments. I was also keen on user-experience-enhancing features such as integration with external systems like OpenID, Twitter and Yahoo.

My criteria were:

email subscription, RSS a bonus
support for pre-formatted text sections in comments (essential for code samples)
an import mechanism for existing comments
threading of comments to allow commenters to respond to each other sensibly
clean look with some ability to customise
support for a variety of authentication/profile systems

There are a number of hosted comment systems out there. The most popular options seem to be Disqus, JS-Kit Echo and IntenseDebate.

IntenseDebate was eliminated first because it doesn't seem to provide an import mechanism for custom websites. Import only seems to be supported for well known blog platforms such as Wordpress. There's no comment API either. The approach seems to be to leave your old comment system in place and just have new comments go into IntenseDebate. Not good enough, I wanted to completely replace the existing comments system.

After some deliberation I decided on JS-Kit Echo for one tiny reason: it supports the <pre> tag. The closest Disqus supported was the <code> tag which doesn't preserve white-space (useless for Python code samples).

So I paid my US$12 (it's the only service that doesn't have a free option) and started looking at how to import my existing comments using their API ... and quickly found that it sucks. Comments can be submitted but you can't specify a timestamp so they are dated with the import date. Far from ideal. Then there's the API for retrieving comments: it returns the data as JavaScript code (no not JSON)! It's pretty clear that the API is what they use with the JavaScript for Echo itself and geared for that use only. They've just thrown it out there and documented it, warts and all.

Back to the drawing board.

The only showstopper for Disqus was the lack of <pre>. Everything else about it was great: it met all my requirements and the API was clean and comprehensive. If only there was a way to have properly formatted source code in the comments.

Light bulb moment: use a CSS hack to make <code> in comments behave like <pre>. The trick is to turn code into a block element and change how white-space is handled. The CSS snippet looks like:

.dsq-comment-message code {
  display:block;
  white-space:pre;
}

Works great.

With the only blocker gone, I wrote a Python script with the help of Ian Lewis' excellent disqus-python-client package to pull in the existing comments from the old system. Within an hour or so it was ready to go.

Hopefully this article saves someone else some time if they decide to use one of these systems. Getting things running chewed up a lot more time then I had expected.