Planet Python

Last update: February 14, 2012 06:46 PM

February 14, 2012

Python Diary

Too many micro webframeworks

There is a large amount of Python micro web frameworks out in the wild, and some of them feel similar to eachother, at least when I first approach them. I plan on going through a few of these micro web frameworks to see what the exact differences are. I will not be comparing them to full frameworks such as Django or Web2py.

Bottle

First up, Bottle. It is very micro in that it comes in a single Py file and is very portable across platforms. If you need a simple Python web app running on say an embedded system with very low resources and disk space, this will be a good option for you. Here is a snippet of code from their home page of an example application:

from bottle import route, run

@route('/hello/:name')
def index(name='World'):
    return '<b>Hello %s!</b>' % name

run(host='localhost', port=8080)

Simple, is it not. One could deploy a Python console application which merely displays output for diagnose of a server or heartbeat in a matter of minutes. I wouldn't recommend it for full web applications, but no one is stopping you.

web.py

Next we have web.py. I have heard many good things about this micro framework, that doesn't make it sounds so micro. A good advantage of this web framework is the ability to separate the requests from both GET and POST. This is great when you need to develop an application which depends on the type of request coming through, such as a REST API. Here is a snippet from their home page:

import web
        
urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:        
    def GET(self, name):
        if not name: 
            name = 'World'
        return 'Hello, ' + name + '!'

if __name__ == "__main__":
    app.run()

A little more complex than Bottle, but it's still simpler than say Django or Web2py.

Flask

Looking for something even simpler than Bottle? Try Flask, there are less imports to remember, but it is ever so similar to Bottle. But, it comes with more than bottle, so you will have more than a single file to work with. It comes with the Jinja2 template engine, RESTful dispatching, and other goodies that other micro frameworks lack. Here's a snippet from their homepage:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Notice how similar this code is to Bottle's example code? If your looking for a little extra in a micro web framework, without all the extras of Django or Web2py, flask may be a good option for you.

CherryPy

Finally we get to CherryPy. I have heard many great things about this framework and many Python developers seem to favor it. I have not personally tried it myself, but I do plan on trying it out to see how it differs from other frameworks out there, micro and otherwise. They say the framework is a minimalistic one, but I still believe that Bottle is the smallest one around. It supports similar components to Django, such as caching, sessions, static files, and even authorization. It also appears to run on many platforms including Android. Here is the example code snippet from their home page:

import cherrypy
class HelloWorld(object):
    def index(self):
        return "Hello World!"
    index.exposed = True

cherrypy.quickstart(HelloWorld())

It does look fairly simple, but the last statement quickstart sort of scares me. This would imply that there are other methods of starting the application, in effect for larger applications, this simple code snippet above may not be the real deal. I will look into this one further and see how complex web applications are built and how it would differ from other micro frameworks listed here.

Final thoughts

Now that this entry is coming to a close, I would like to express my final thoughts on so-called micro frameworks. Are all of these micro frameworks truly needed? Does having this many cause fragmentation in how Python web development should be done? Does it cause developer confusion, on which one a developer should learn or avoid? Let me know what you think about this micro-framework situation us Python developers are faced with in the comments below.

February 14, 2012 06:41 PM

Kristján Valur

Clearing weakrefs

I just had this problem which would have been elegantly solved with the ability to manually clear weak references pointing to an object. I am (for technical reasons) recycling an object, so instead of killing it and re-creating it, I re-initialize it. But that leaves old weak references in place. How nice wouldn’t it be to be able to call “myobject.clear_weakrefs()”?

February 14, 2012 02:17 PM

Rob Galanakis

Large initializers/ctors?

With closures (and to some extent with runtime attribute assignments), I find the signatures of my UI types shrink and shrink. A lot of times we have code like this (python, but the same would apply to C#):

class FooControl(Control):
  def __init__(self, value):
    super(FooControl).__init__()
    self.value = value
    self._InitButtons()    

  def _InitButtons(self):
    self.button = Button('Press Me!', parent=self)
    btn.clicked.addListener(self._OnButtonClick)

  def _OnButtonClick(self):
    print id(self.button), self.value

However we can easily rewrite this like so:

class FooControl(Control):
  def __init__(self, value):
    super(FooControl).__init__()
    btn = Button('Press Me!', parent=self)
    def onClick():
      print value
    btn.clicked.addListener(onClick)

Now this is a trivial example. But I find that many types, UI types in particular, can have most or all of these callback methods (like self._OnButtonClick) removed by turning them into inner functions. And then as you turn them into inner functions in init, you can get rid of stored state (self.value and self.button).

But as we take this to the extreme, we end up with very simple classes (and in fact I could replace FooControl with a function, it doesn’t need to be a class at all), but very long init methods (imagine doing all your sub-control creation, layout, AND all callback functionality, inside of one method!).

I’ve decided I’d rather have a long init method, usually broken up into several inner functions, rather than a larger signature on the class with layout, callbacks, and stored state. In my mind, it is easier to pull something out into a type attribute, rather than remove it, as anything on the type is liable to be used externally. And breaking up your layout into instance methods that can really only be called once (_InitButtons), from the init, adds a cognitive burden for me.

So I can justify this decision to eliminate extra attributes rationally, but what seals the deal is, I’m not unit testing any of this code anyway. So whether it is in one long method, or broken up into several methods, it isn’t getting tested.

I started out as very much in the ‘break into small methods’ camp but have wholesale moved into the ‘one giant __init__ with inner functions’ camp. I’m curious what you all prefer and why?

February 14, 2012 12:03 PM

PyCon

Need dinner plans? Yelp can help!

Through all of the talks, tutorials, open spaces and more, some of the best parts of PyCon are the meals. Especially dinner! Each PyCon ticket gets you catered breakfast and lunch at the conference center, with lunch being a great way to refuel and sit down at a table and meet your next project contributor, business partner, or friend. Dinner, however, is a meal you’re on your own for. Not to worry, because with 1500+ people needing to head out for dinner, there’s never a shortage of groups on the way out for a burger, pasta, falafel, beer, wine, you name it. Some people even use PyCon sponsor Yelp.com to find their dinner destination.

This year Yelp has stepped up as a sponsor for food! Not only will they help you find a destination, they’ll be giving rides to a few of the area’s highly rated restaurants. On Friday and Saturday night, Yelp will be renting several buses and driving conference goers to these select eateries, giving you a chance to get out and have a good meal with good company. The details are still being ironed out, but it looks like there will be around five buses each holding approximately 30 people. Yelp plans to send along one of their developers on each bus as an ambassador, and they might bring some nice swag as well!

As for getting on the bus, sign ups will be announced on site on the first day of the conference. You like what you see? Sign up and enjoy! Be sure to sign up on site - first come, first served.

Schedule-wise we’re looking at departure around a half hour after the day’s lightning talks are over, so about 6:30-7:00. We’ll certainly announce a more accurate time during each day’s morning session. Once you’re there, you’ll have 2-2.5 hours to spend at the restaurant and in the area. After that, the bus heads back to the hotel where you can join back up with the festivities. You’ll be back just in time for the evening’s open spaces and BoF sessions!


Bon Appétit!

February 14, 2012 10:20 AM

Shannon -jj Behrens

Python Concurrency Spreadsheet

I'm a fan of Nicholas Piël's Asynchronous Servers in Python blog post. In a similar vein, my buddy Shailen Tuli and I put together the following spreadsheet. Feel free to view it on Google Docs and make corrections.



A few notes:

It is not my goal to use this spreadsheet to pick favorites. Rather, I'm trying to use this spreadsheet to point out differences among the different approaches.

I know very little about DieselWeb, MultiTask, FriendlyFlow, Weightless, Fibra, and Cogen. All I know is that Nicholas Piël's pointed them out as generator-based libraries.

Some of these libraries support multiple approaches at the same time. For instance, Twisted lets you use a mix of callbacks, generators, and threads. Similarly, Tornado lets you use a mix of callbacks and threads. However, it's important to point out the compromises involved. For instance, if you're using Tornado, you can't have 5000 concurrent clients to a WSGI server; instead you have to switch over to the asynchronous callback API.

When I say things like "Can handle 5000 clients?", I don't mean that each of those requests is actively being served. There's only so much CPU to go around! Rather, I mean you can have 5000 clients that are each in various stages of completion, and the server as a whole is mostly waiting on IO.

You can also read my other blog posts Python: Concurrency and Python: Asynchronous Networking APIs and MySQL.

February 14, 2012 10:05 AM

PyCon

PyCon 2012: Want some business cards for PyCon 2012?

The deals and partnerships for PyCon 2012 keep rolling in!

The folks at custom print shop MOO are quite good at the printing business, and they’re also good at spotting trends. One they recently noticed was an uptick in business cards intended for PyCon, so they contacted us. With the conference being such a great place to network, from the hallways to the dinner tables, we were able to partner up with them and get you 50 free business cards of your own choosing.

The London-based business card, postcard, and sticker printer will even package up the shipment and deliver straight to Santa Clara, where you can pick up your cards at the registration desk! Talk about easy. All they require is that you pay a small shipping and handling charge and the cards are all yours. If you prefer, they can be mailed to you instead.

If you want to take advantage of this great offer, http://moo.com/link/h86f is the PyCon special page. From there you can fill out one of their templates or upload your own design. They even have a product available called Printfinity where you can have many different designs in a pack - 50 different ones if you choose!

Your fancy new cards will come in handy at this year’s new Job Fair. Mingle around with some of PyCon’s 122 sponsors and pass out your details in style.

February 14, 2012 09:52 AM

Maciej Fijalkowsk

PyPy and it's future challenges

Obviously I'm biased, but I think PyPy is progressing fairly well. However,
I would like to mention some areas where I think pypy is lagging ---
not living up to its promises or the design decisions simply didn't
turn out as good as we hoped for them. In a fairly arbitrary order:

• Whole program type inference. This decision has been haunting
  separate compilation effort for a while. It's also one of the reasons
  why RPython errors are confusing and why the compilation time is so long.
  This is less of a concern for users, but more of a concern for developers
  and potential developers.


• Memory impact. We never scientifically measured
  memory impact of PyPy on examples. There are reports of outrageous pypy
  memory usage, but they're usually very cryptic "my app uses 300M" and not
  really reported in a way that's reproducible for us. We simply have to start
  measuring memory impact on benchmarks. You can definitely help by providing
  us with reproducible examples (they don't have to be small, but they have
  to be open source).

The next group all are connected. The fundamental question is: What to do
in the situation where the JIT does not help? There are many causes, but,
in general, PyPy often is inferior to CPython for all of the examples.
A representative, difficult exammple is running tests. Ideally, for
perfect unit tests, each piece of code should be executed only once. There
are other examples, like short running scripts. It all can
be addressed by one or more of the following:

• Slow runtime. Our runtime is slow. This is caused by a combination
  of using a higher
  level language than C and a relative immaturity compared to CPython. The
  former is at least partly a GCC problem. We emit code that does not look
  like hand-written C and GCC is doing worse job at optimizing it. A good
  example is operations on longs, which are about 2x slower than CPython's,
  partly because GCC is unable to effectively optimize code generated
  by PyPy's translator.


• Too large JIT warmup time. This is again a combination of issues.
  Partly this is one of the design decisions of tracing on the metalevel,
  which takes more time, but partly this is an issue with our current
  implementation that can be addressed. It's also true that in some edge
  cases, like running large and complex programs with lots and lots
  of megamorphic call sites, we don't do a very good job tracing. Because
  a good example of this case is running PyPy's own test suite, I expect
  we will invest some work into this eventually.


• Slow interpreter. This one is very similar to the slow runtime - it's
  a combination of using RPython and the fact that we did not spend much
  time optimizing it. Unlike the runtime, we might solve it by having an
  unoptimizing JIT or some other medium-level solution that would work good
  enough. There were some efforts invested, but, as usual, we lack enough
  manpower to proceed as rapidly as we would like.

Thanks for bearing with me this far. This blog post was partly influenced
by accusations that we're doing dishonest PR that PyPy is always fast. I don't
think this is the case and I hope I clarified some of the weak spots, both here
and on the performance page.

EDIT:For what is worth I don't mention interfacing with C here and that's not because I think it's not relevant, it's simply because it did not quite fit with other stuff in this blog post. Consider the list non-exhaustive

Cheers,

fijal

February 14, 2012 09:43 AM

Python Diary

First steps into Kivy

I just finished the Kivy tutorial on how to build a simple Pong Game. I must say, Kivy is a very interesting framework, and I plan on building some Android apps using it soon.

The framework makes heavy use of classes to describe each component of your application, and also has a simple language for rapidly building interfaces and binding them in Python. It has native support for multi-touch screens, which makes it an ideal platform to learn.

Tablets and smartphones are rather huge right now, and I don't see them going away anytime soon. Most users of a tablet or smartphone prefer an app over a website, I know I do. Kivy makes it relatively easy to create an app which is network connected, as it has native support for JSON calls, it manages the web request and decoding of the JSON object for you.

A small project I plan to implement using Kivy is a simple map viewer for the OHRRPGCE engine. I have previously created a full map editor in another language and understand the underlying file formats. I am curious how creating such an application in Kivy will fair. From what I have read in the documentations for Kivy, I will need to use a texture to map each map tileset onto. From these map tilesets, I would then blit the required tiles to a new texture for the actual map layers. I will use touch controls for panning the map around, similar to how Google maps works on a tablet. I hope to add features such as pinch to zoom and such, since Kivy does support scaling in OpenGL. I will explain the process throughout this blog for those who are interested in Kivy app development.

Another interesting project would be to create a Kivy app for this here blog, that way Android users can easily access this blog directly on their device using a nice user interface.

February 14, 2012 09:40 AM

Yaniv Aknin

nginx+gzip module might silently corrupt data upon backend failure

There are several elements that make absolutely certain the page you’re reading in your browser right now is actually an accurate representation of the resource the HTTP server holds in the URL you requested¹. Disregarding caching for a minute, we have two elements making sure the representation you get is protected from errors. The first protecting element is, of course, TCP, making sure that if the server wrote two-hundred bytes in a particular order, either they’ll all arrive to your end, in order and without errors, or your TCP stack will realize something bad happen and give your user-agent (your browser) a chance to cope with the error. The need for the second protecting element is a bit more sneaky: TCP will guarantee everything the server wrote will arrive, i.e., bytes for which the server called write(2) or equivalent will arrive, or you’ll know something went wrong. But what about bytes the server should have written but didn’t write all – for example, because some component on the server’s side failed?

The original HTTP (HTTP 0.9, 1996 time) didn’t cope with this situation at all. The signal to the client that the server finished talking was to disconnect the TCP session, which, from the client’s side, is a vague signal. Did the TCP server disconnect because it finished or because it ran into trouble (software fault, sysadmin action, kernel behaviour due to memory pressure or even a bug, etc)? Thankfully, current HTTP kicks in to complement TCP, allowing the server to do one of several things in order to make sure you’ll at least know you didn’t receive the whole picture. By far the two most common thing the server will do are to specify a Content-Length in the response’s header or to use a Transfer-Encoding, most probably chunked transfer encoding.

Content length is simple to grasp. The server wishes to say 200 bytes. It explicitly says: “I will say 200 bytes” in the response header. If the user-agent didn’t receive 200 bytes of response, it knows something went wrong. Chunked transfer encoding is only slightly more complex – the server will send the response in chunks, each chunks prefixed by the length of the chunk. The end of the document is marked by a zero-length chunk. So if the user-agent saw a chunk cut in the middle, or didn’t receive a zero-length chunk, it also knows something went wrong and has a chance to decide what to do about it. For example, when faced with incorrect content length, Chrome displays an ERR_CONNECTION_CLOSED error, whereas Firefox would display the portion of the page it did receive. Different behaviour, yes, but at least both user-agents in this example had a chance realize the response they received is partial. Which is really, really important, you know why? I’ll tell you why.

Enter caching. HTTP caching is a non-trivial matter with many unexpected gotchas and pitfalls, and I can’t cover it all here (why the complexity? I think it’s because all caching is an intentional form of data/state repetition, and repetition is something that in my experience humans often have difficulty reasoning about). By far the best document I know about HTTP caching is this splendid guide, but if you’re in a hurry or impatient, let me summarize the points interesting for this particular post. First, caches might exist in many places, some of them might be surprising, some of them might be slightly broken or at least very aggressive (ISP transparent caches, mutter mutter cough cough). Second, among many other things, HTTP caching lets a server give a client a token together resource, telling the client “next time you request this resource, tell me you have this token; maybe I’ll just tell you that the representation you got with this token is still fresh, without transferring it all over again”. This is called an ETag, and the response that says “just use what you have in your cache” is called HTTP 304 NOT MODIFIED.

How is this relevant to HTTP responses cut in the middle? Well, if servers didn’t have a way of telling the user-agent how long is the document, and if the response was cut in the middle due to a server fault as described above, the user-agent/sneaky-caching-proxy might cache incorrect responses. If the server also sends an ETag along with the response, the caching entity will store this ETag along with the invalid cached representation, and even when it’s time to check the representation’s freshness with the origin server, the server will just take a look at the ETag, say “yep, this is fine”, tell the cache to keep using the bad representation and <sinister>never ever let it recover</sinister>. If this happens on a large ISP’s transparent cache, easily tens of thousand of your users could be affected. If the affected resource is a common element in many of your pages with strict syntax checking, like a javascript resource, you’re kinda screwed. The only hope in such a condition is that the client, for some reason, will specify Cache-Control: no-cache in the request, and that caching entities along the path to the server will honour this request. Browsers like caching, so they won’t usually request no-cache, although AFAIK, recently Chrome started sending no-cache when the user explicitly requests a force-reload (Cmd-R on a Mac). Other browsers don’t fare as well, and I think that hoping one of your Chrome users will force reload the bad resource in time to save the day is hardly a sturdy solution.

Bottom line is, it’s really important to know when a representation of a resource is broken. Which is why I was quite amazed to learn that my HTTP server of choice, nginx, doesn’t validate the Content-Length it receives from its upstreams and is simply unaware when the response it received from an upstream server is chopped off. If your response specifies a content length but closes the connection without delivering enough bytes, nginx will simply stall the request for a long time without closing the connection downstream, even though it has no hope of receiving additional data to push downstream. I tried this both with proxy_pass and uwsgi_pass, but I’m quite confident it’s true for other backends (fastcgi_pass, scgi_pass, etc). This is bad, but not as bad as the case where you want an nginx module to manipulate your content, removing existing content length/transfer encoding and applying its own (the gzip module indeed does that). If a backend error occurs while content-length-oblivious-nginx is altering the data, the content altering module will apply what it applies to the bytes it received, add new content-length/transfer-encoding, assuring everyone the response is OK, and entice user-agents or even proxies to enter the almost-never-recover bad cache scenario I described in the previews paragraph. Ouch!

The proper way to fix this, IMHO, is that nginx simply must start looking at the upstream’s content length (or transfer encoding, once nginx starts using chunked responses with its upstreams). Part of the reason I’m writing this post is that Maxim Dounin, venerable nginx comitter and an OK chap overall, told me he doesn’t consider this a top priority at the moment, but I humbly disagree with his assessment of how serious the issue is. Until such a time as nginx is fixed about this, I think you must disable all content-manipulating nginx modules and instead handle all message length affecting work in your upstream (compression, addition, etc). This is what I opted to do with my django based web app, I replaced nginx’s gzip module with Django’s GZipMiddleware. It’s a terrible shame though. It’s doing the job of nginx for it, probably in a lesser fashion than how nginx could, it violates a must not clause in Python’s WSGI PEP333, and I have empiric proof that Tim Berners-Lee chokes a kitten every time you do it.

But what’s the alternative? Risk invisibly cached corrupt data for an undetermined length of time? Ditch nginx, which I think is the best HTTP server on this planet despite this debacle? Nah. Both are unacceptable.

¹ This post assumes convenient values of “absolutely certain”; also, everything related to security/content tampering is out of scope in this post. I’m talking about possibly misbehaving but certainly well-meaning components.

February 14, 2012 09:07 AM

Catherine Devlin

Growth in PyCon sponsorship

When I teach Python, I usually include a slide or two about Python's importance and growth. At this point, that's begun to feel unnecessary - but I think I will include this illustration, because it's a neat one.

In 2004, I'd just started looking at Python. When I found out that PyCon would be just a few miles from my sister-in-law's house, I thought, "Why not?" - and I fell in love with the Python community forever. PyCon 2004 had nine sponsors.


EaIcazSWNM0/TzqSGPSnAPI/AAAAAAAAAJM/DDursOJS8PU/s1600/pycon2004sponsors.png">
EaIcazSWNM0/TzqSGPSnAPI/AAAAAAAAAJM/DDursOJS8PU/s1600/pycon2004sponsors.png">
EaIcazSWNM0/TzqSGPSnAPI/AAAAAAAAAJM/DDursOJS8PU/s1600/pycon2004sponsors.png">
This year, I barely squeaked my registration in before the blast doors slammed shut. Here are PyCon US 2012's sponsors.

February 14, 2012 08:59 AM

S. Lott

Multiprocessing Goodness -- Part 2 -- Class Defintions

The multiprocessing module includes a generic Process class, which can be used to wrap a simple function.

The function must be designed to work with Queues or Pipelines or other synchronization techniques.

There's an advantage, however, to defining a class which gracefully handles generator functions.  If we have Generator-Aware multi-processing, we can (1) write our algorithms as generators and then (2) trivially connect Processes with Queues to improve scalability.

We're looking at creating processing "pipelines" using Queues.  That way we can easily handle multiple-producer and multiple-consumer (fan-in, fan-out) processing that enhances concurrency.

See Multiprocessing Goodness -- Part 1 -- Use Cases for more information.

We have three use cases:  Producer, Consumer and Consumer-Producer.

Producer

A Producer gets data from somewhere and populates a queue with it.  This is the source that feeds data into the pipeline.

class ProducerProcess( Process ):
    """Produces items into a Queue.
    
    The "target" must be a generator function which yields
    pickable items.
    """
    def __init__( self, group=None, target=None, name=None, args=None, kwargs=None, output_queue=None, consumers=0 ):
        super( ProducerProcess, self ).__init__( name=name )
        self.target= target
        self.args= args if args is not None else []
        self.kwargs= kwargs if kwargs is not None else {}
        self.output_queue= output_queue
        self.consumers= consumers
    def run( self ):
        target= self.target
        for item in target(*self.args, **self.kwargs):
            self.output_queue.put( item )
        for x in range(self.consumers):
            self.output_queue.put( None )
        self.output_queue.close()

class ConsumerProcess( Process ):
    """Consumes items from a Queue.
    
    The "target" must be a function which expects an iterable as it's
    only argument.  Therefore, the args value is not used here.
    """
    def __init__( self, group=None, target=None, name=None, kwargs=None, input_queue=None, producers=0 ):
        super( ConsumerProcess, self ).__init__( name=name )
        self.target= target
        self.kwargs= kwargs if kwargs is not None else {}
        self.input_queue= input_queue
        self.producers= producers
    def items( self ):
        while self.producers != 0:
            for item in iter( self.input_queue.get, None ):
                yield item
            self.producers -= 1
    def run( self ):
        target= self.target
        target( self.items(), **self.kwargs )

        
class ConsumerProducerProcess( Process ):
    """Consumes items from a Queue and produces items onto a Queue.
    
    The "target" must be a generator function which yields
    pickable items and which expects an iterable as it's
    only argument.  Therefore, the args value is not used here.
    """
    def __init__( self, group=None, target=None, name=None, kwargs=None, input_queue=None, producers=0, output_queue=None, consumers=0 ):
        super( ConsumerProducerProcess, self ).__init__( name=name )
        self.target= target
        self.kwargs= kwargs if kwargs is not None else {}
        self.input_queue= input_queue
        self.producers= producers
        self.output_queue= output_queue
        self.consumers= consumers
    def items( self ):
        while self.producers != 0:
            for item in iter( self.input_queue.get, None ):
                yield item
            self.producers -= 1
    def run( self ):
        target= self.target
        for item in target(self.items(), **self.kwargs):
            self.output_queue.put( item )
        for x in range(self.consumers):
            self.output_queue.put( None )
        self.output_queue.close()

def prod( *args ):
    for item in some_function(*args):
       yield item

def cons( source ):
    for item in source:
       final_disposition(item)

def cons_prod( source ):
    for item in source:
       next_value= transform(item)
       yield next_value

for final in consumer( cons_prod( producer( *args ) ) ):
    print( final )

February 14, 2012 08:32 AM

TDRE - Test Driven Reverse Engineering Case Study

Background

Read up on compass variation or declination.  For example, this NOAA site provides some useful information.

Mariners use the magnetic variation to compute the difference between True north (i.e., aligned with the grid on the chart) and Magnetic north (i.e., aligned with the compass.)

The essential use case here is "What's the compass variation at a given point?"  The information is printed on paper charts, but it's more useful to simply calculate it.

There are two magnetic models: the US Department of Defense World Magnetic Model (WMM) and the International Association of Geomagnetism and Aeronomy (IAGA) International Geomagnetic Reference Field (IGRF).

A packaged solution is geomag7.0.  This includes both the WMM and the IGRF models.  This is quite complex.  However, it does have "sample output", which amount to unit test cases.

The essential spherical harmonic model is available separately as a small Fortran program, igrf11.f.

Which leads us to reverse engineering this program into Python.

TDRE Approach

The TDRE approach requires having some test cases to drive the reverse engineering process toward some kind of useful results.

The geomag7.0 package includes two "Sample Output" files that have the relevant unit test cases.  The file has column headings and 16 test cases.  This leads us to the following outline for the unit test application.

    class Test_Geomag( unittest.TestCase ):
        def __init__( self, row ):
            super( Test_Geomag, self ).__init__()
            self.row= row
        def runTest( self ):
            row= self.row
            if details: 
                print( "Source: {0:10s} {1} {2:7s} {3:10s} {4:10s} {5:5s} {6:5s}".format( row['Date'], row['Coord-System'], row['Altitude'], row['Latitude'], row['Longitude'], row['D_deg'], row['D_min'] ),
                file=details )
            
            date= self.parse_date( row['Date'] )
            lat= self.parse_lat_lon( row['Latitude'] )
            lon= self.parse_lat_lon( row['Longitude'] )
            alt= self.parse_altitude(row['Altitude'] )
            
            x, y, z, f = igrf11syn( date, lat*math.pi/180, lon*math.pi/180, alt, coord=row['Coord-System'] )
            D = 180.0/math.pi*math.atan2(y, x) # Declination 

            deg, min = deg2dm( D )
            
            if details: 
                print( "Result: {0:10.5f} {1} K{2:<6.1f} {3:<10.3f} {4:<10.3f} {5:5s} {6:5s}".format( date, row['Coord-System'], alt, lat, lon, str(deg)+"d", str(min)+"m" ), 
                    file=details )
                print( file=details )
            
            self.assertEqual( row['D_deg'], "{0}d".format(deg) )
            self.assertEqual( row['D_min'], "{0}m".format(min) )

    def suite():
        s= unittest.TestSuite()
        with open(sample_output,"r") as expected:
            rdr= csv.DictReader( expected, delimiter=' ', skipinitialspace=True )
            for row in rdr:
                case= Test_Geomag( row )
                s.addTest( case )
        return s

    r = unittest.TextTestRunner(sys.stdout)
    result= r.run( suite() )
    sys.exit(not result.wasSuccessful())

def  igrf11syn( date, nlat, elong, alt=0.0, coord='D' ):
    return None, None, None, None

• Hand Optimizations.  All repeated expressions were manually hoisted out of their context.  This is clever but makes the code particularly obscure.  It doesn't help when local variables are named ONE, TWO and THREE.  Bad is it is, not much needs to be done about this.  Python code looks a bit like Fortran code, so very little needs to be done except add `math.` to the various function calls like sort, cos and sin.
• Sparse Array Chicanery.  There are actually two spherical harmonic series.  The older 10-order and the new 13-order.   Each model has two sets of coefficients: g and h.  These form two half-matrices plus a vector.  The old models have 55 g values in one matrix, 55 h values in second matrix, and a set of 10 more g values that form some kind of vector; 160 values.  The new models have 91 g, 91 h and 13 g in the extra vector; 195 values.  There are 23 sets of these coefficients (for 1900, 1905, ... 2015).  The worst case is 23×195=4,485 values.  This appears to be too much memory, so the two matrices and vectors are optimized into a single opaque collection of 3,256 numbers and delightfully complex set of index calculations.

February 14, 2012 08:00 AM

Wallix

Announcing PyLogsParser 0.4

Wallix LogBox team is happy to announce version 0.4 of PyLogsParser. New normalizers Wallix AdminBastion authentication logs, written by Nassim Babaci Cisco ASA logs. Dansguardian logs. Features Adds Common Callbacks facility : a library of functions that are ready to … Continue reading →

February 14, 2012 07:41 AM

Yuval Greenfield

Python isn’t English and iterator “labels”

Us python fanboys like to think of python as similar to English and thus more readable. Let’s examine a simple piece of code:

for item in big_list:
    if item.cost > 5:
        continue
    item.purchase()

For our discussion there are only 3 kinds of people:

1. People who have never seen a line of code in their life.
2. Have programmed in other languages but have never seen python.
3. Python programmers.

• “for item in big_list” – either we’re talking about doing something for a specific item in a big_list or we’re talking about every single item. Ambiguous but the first option doesn’t really make sense so that’s fine.
• “if item.cost > 5″ – non-programmers are going to talk about the period being in a strange place, but programmers will know exactly what’s up.
• “continue” – That’s fine, keep going. English speakers are going to get the completely wrong idea. As programmers we’ve grown used to this convention though its meaning in English is very specifically equivalent to what pythonistas call “pass” or “nop” in assembly. We really should have called this “skip” or something.
• “item.purchase()” – non-programmers are going to ask about the period and the parentheses but the rest grok that easily.

So I’m pretty sure this isn’t English. But it’s fairly readable for a programmer. I believe programmers of any of the top 8 languages on the TIOBE index can understand simple python. I definitely can’t say the same for Lisp and Haskell. Not that there’s anything wrong with Lisp/Haskell, these languages have specialized syntax for their honorable reasons.

Continue is a silly word, what about iterator labels?

Let’s say I want to break out of an outer loop from a nested loop, eg:

for item in big_list:
    for review in item.reviews:
        if review < 3.0:
            # next item or next review?
            continue
        if review > 9.0:
            # stop reading reviews or stop looking for items?
            break

Java supports specific breaks and continues by adding labels to the for loops but I think we can do better. How about this:

items_gen = (i for i in big_list)
for item in items_gen:
    for review in item.reviews:
        if review < 3.0:
            items_gen.continue()
        if review > 9.0:
            items_gen.break()

But how can that even be possible you may ask? Well, nowadays it isn’t but maybe one day if python-ideas like this idea we can have nice things. Here’s how I thought it could work: a for-loop on a generator can theoretically look like this:

while True:
    try:
        item = next(gen)
        # do stuff with item
    except StopIteration:
        break

But if it worked like I propose below we can support the specific breaks and continues:

while True:
    try:
        item = next(gen)
        # do stuff with item
    except gen.ContinueIteration:
        pass
    except gen.StopIteration:
        break
    except StopIteration:
        break

So every generator could have a method which throws its relevant exception and we could write specific breaks and continues. Or if you prefer a different spelling could be “break from mygen” or “continue from mygen” as continue and break aren’t allowed as method names normally.

I think this could be nice. Although many times I found myself using nested loops I actually preferred to break the monster into 2 functions with one loop each. That way I could use the return value to do whatever I need in the outer loop (break/continue/etc). So perhaps it’s a good thing the language doesn’t help me build monstrosity’s and forces me to flatten my code. I wonder.

February 14, 2012 07:41 AM

EmptysquarePython

Third Normal Form and Ultimate Truth

I have an opinion: most people learned about relational databases as if RDBMSes were designed to store the ultimate truth about some data. They figured that once the schema had been properly diagrammed and normalized, then they could load all their data into it, and finally, start doing some queries.

To pick on an easy target, look at Wikipedia’s article on schema design. It summarizes the two steps a designer must take:

1. Determine the relationships between the different data elements.
2. Superimpose a logical structure upon the data on the basis of these relationships.

Do you see a step that’s missing? If you’ve deployed and maintained a large-scale application you’ll probably see what the Wikipedia authors omitted. In fact, it’s the first step: Figure out what one question your database must answer. Then, design your schema to answer that question as fast as possible. And now you’re done. Come to think of it, you never had to do steps 1 and 2 at all.

There’s a total disconnect between the approaches of introductory SQL courses and real-world application development, and I think this disconnect is slowing down adoption of NoSQL.

Consider Facebook Messages. After a (now rather well-publicized) evaluation process, Facebook chose HBase, a NoSQL data store, as the main database for their message system. I haven’t talked to anyone there, but I figure they chose it based on this criterion:

How fast will our database answer the question, “What are this user’s most recent 10 messages?”

They chose the database system that could answer that question the fastest, and they designed the best schema they could think of to answer that question. Anything else they need to ask HBase may be slow, or difficult, but that doesn’t matter, because “What are this user’s most recent 10 messages?” probably accounts for 99% of the load on their system.

If you learned about databases in college, following some textbook, I expect you were guided through a long process of modeling real-world data using rows and columns, to express some profound truth about the data. Then, you were introduced to SQL, with which you could query the data. At the end of the course, maybe there was a brief discussion of database performance. Probably not.

Data at the scale that the largest websites handle doesn’t work that way. Large applications design their schemas to answer one question as quickly as possible, and no other considerations are significant.

The next time you read about a NoSQL database you might wonder, “What about foreign keys, or normalization? What about transactions? Why can’t I define secondary indexes? Why are range queries prohibited?” (I’m just picking some limitations at random—each system is different.) Consider who built these new database systems, and what their experience has been. The ideas behind NoSQL databases mostly originated at places like Google, Amazon, and Yahoo. They build huge systems, and huge systems’ loads are usually dominated by a handful of queries. Companies build their database systems from the ground up to optimize the performance of these queries. NoSQL databases encourage you to figure out ahead of time, “What one question do I need to answer?” Figure that out, and choose your database software and your schema based on that. Nothing else really matters.

February 14, 2012 06:31 AM

ShiningPanda

Selenium, Python and Jenkins on Debian - 3/3

In our first post on Selenium in Python, we saw how to prepare your continuous integration environment on Debian. In the second one, we saw how to enable Selenium tests in your existing web project. Now it's time to see how to integrate all this within Jenkins!

Global Configuration

First ensure that the latest version of the ShiningPanda Plugin and Xvnc Plugin are installed (check on Manage Jenkins > Manage Plugins > Installed page). If not, look for them in the Available tab and perform the installation (don't forget to restart Jenkins).

• The description.
• The source repository, here https://github.com/shiningpanda/djangotutorial-selenose.git.
• The build trigger policy: checking for modifications every five minutes in this example.

• The Web Driver one to specify the web browser to use,
• The Python one to specify the Python interpreter running the web server.

• In Name define the environment variable that will contains the Web Driver environment to test on, here WEBDRIVER,
• In Values write a space separated list of Web Driver environment you want to test on, here firefox and chrome.

• Install dependencies with pip,
• Step in tests folder,
• Run the tests using the script run.py (a nose wrapper) without forgetting to specify the Web Driver to use with the --selenium-driver= option (note that its value $WEBDRIVER comes from the axis defined previously),
• Convert code coverage report in XML.

• If this build is successful, builds on other versions will be triggered.
• If this build fails, other versions will not be tested.

February 14, 2012 03:00 AM

Montreal Python User Group

Cloud Robotics Hackathon

Hello Montreal hackers

The cloud robotics week-end is happening the 2nd, 3rd and 4th of March.

Come hack on robots and web services to create cool projects in robotics. See   http://roboticshackathon.com/ for more information.

We are waiting for you if you would like to create a team Montreal-Python and if you would like to join us for this week-end of hack and programming.

February 14, 2012 02:38 AM

Stephen Ferg

Backing up your email

Just in case someone might find this useful …

I recently had something bad happen to me. I use Thunderbird (on Windows Vista) as my email client. I asked Thunderbird to compact my email files, and it wiped out a bunch of my email messages. (I think that one of my email files must have been corrupt, and when I compacted it, the compaction process wiped out messages that should not have been wiped out.)

You can recover deleted email messages … but not after the email file has been compacted. So the messages were not recoverable. Bummer.

The upside is that this nasty incident led me to learn some things.

One thing that I learned was that the disk backup utility that I was using at the time did NOT backup my email files. The email files were stored in a directory called AppData, and the AppData directory is a “hidden” directory. So the backup utility didn’t see the AppData directory, and didn’t back it up. So I had no backup of the deleted messages.

Learning that led me to investigate ways to backup my email files, and I found this: Five ways to keep your emails backed up

For backing up Thunderbird files, it recommends MozBackup as being fast, free and easy to use. So I tried MozBackup, and those claims seem to be true.

Now I’m evaluating different disk backup options.

The take-away here is that you need to pay special attention to backing up your email files. So if you’re not backing up your email files, take a look at Five ways to keep your emails backed up (and read the comments, which are useful) or google something like “email backup”.

[Note that this applies only if you are using an email client such as Thunderbird, Outlook, Outlook Express, etc. If you don't use an email client, and do all of your email work through a Web interface to your Internet Service Provider, then this is not an issue.]

February 14, 2012 02:17 AM

February 13, 2012

EmptysquarePython

Philly MongoDB User Group: Python, MongoDB, and Asynchronous Web Frameworks

Photo (C) Parent5446

I’ll be recapping last week’s talk on Python, MongoDB, and Asynchronous Web Frameworks this Thursday at 7pm, in Philadelphia, at the Philly MongoDB User Group’s inaugural meetup. We’ll be at the Devnuts office, at 908 North 3rd Street. We’ll have pizza, naturally.

First Philly MongoDB User Group

February 13, 2012 08:26 PM

Cormoran Project

Continuous integration with Travis

We recently started using Travis CI as continuous integration server for Cormoran. Travis is a distributed build system for the open source community used by projects like Ruby, Rails, Rubinius, Rubygems and a large number of open source projects.

1
2
3
4
5
6
7
8
9

language: python
before_script:
    - virtualenv cormoran_env
    - source cormoran_env/bin/activate
    - pip install nose pyhamcrest pydoubles coverage

script:
    - python setup.py develop
    - nosetests tests/unit

February 13, 2012 06:42 PM

Jesse Noller

A letter to my love, my friend, my wife.

A letter to my love, my friend, my wife and my partner — Dusty:

I know it’s the day before Valentines — some things can’t wait just for a day.

Ten years — that’s how long we’ve been with one another. Ten years feels like a lifetime — so much has changed — our lives altered in subtle — and not so subtle ways by the gentle currents of each other. In the time I’ve known you, we have both changed for the better — we compliment and act as one another’s confidant, friend, partner and lovers.

“The most powerful symptom of love is a tenderness which becomes at times almost insupportable.” — Victor Hugo

We’ve been through our times of trial — little things like accidentally renting an apartment in a war zone (my bad!) — and much bigger things from health, to finances, to not know what we were doing or where we were going. We both know that this past year has been probably the one most filled with trials and tribulations.

We’ve sat across from one another not knowing what we were going to do, we’ve held each others hands watching our infant daughter laying in a hospital bed — I’ve held your hand at your bedside in watching your pain and not knowing what to do about it, except to sit there and watch your pain. We’ve been through a lot in ten years.

Despite the trials — we have made each other stronger. You have changed who I am in such fundamental and subtle ways, that I attribute much of who I am now, to you. You have made me happier, stronger, more empathetic — you have also given me the cherished gift of your love, your tears and support in my times of pain.

You have given me more than just your love; you gave me our first daughter Abby — who might as well be a tiny clone of myself in female form (god help us all), who despite her willfulness and strong personality makes my heart jump each time I hear her laugh, each time she runs to me and hugs me and tell me she loves me.

Abby is almost five! Five years old! All parents gush about how smart their children are — but we both know there’s something special and unique about her. There’s more to her than a pushy 4.5 year old, there’s something magical about her that we both see. I can not verbalize or put to words my thanks to you for her. She’s a gift you’ve given to me.

Then there is Addison, our bubbling eight month old. What can I say about someone who greats me with a smile and a laugh whether it’s five in the morning, or me just coming home from a hard day at work?

Addison is more than a gift; she’s a blessing — the past year shows that even in our darkest hours, sitting there in a hospital not knowing what will happen, something watches over us. Addison’s happiness and flourishing is not just due to doctors, or therapists — it’s directly tied to the amazing love and care you provide to her.

Every time I look at Addison, I see an extension of you — your smile, your happiness (and when she giggles when she rams me with her walker, your sense of humor). Addison is again, a gift and blessing you’ve given me.

You’ve given me so much; you’ve changed me so much. You’ve made me look outside of myself and think of others — you, our daughters, you’ve driven me to try to change the world and help as many people as I can. You’ve driven me to be better — a better man, a better husband, father and human.

Times change — people change. We have our hard times — we have those times when we both want to go lock ourselves in the bathroom just to get a moment of quiet. We have times when we just don’t know what will come, and times when we wish what had came had not. We have persevered over the hard times we’ve faced until now, and those hard times we face now, we face together, as one.

You are beautiful — you always have been, you are strong — you are honest and critical. I might say half-jokingly that you’re my better half some times — but you really and truly are (You are also better looking than me!).

You, and the gifts have given me — our daughters, have given me more than a reason to just keep working, just to keep moving from day to day. You’ve given me a reason to truly live, to truly push myself beyond anything I could have imagined eleven years ago. You’ve given me a place and arms to cry in, to laugh in, and to grow in. You’ve given me a view of life, of living, of loving I never dreamed of having.

I know that once again we face hard times. I thought that perhaps this year might be a little easier on us — but so far, we both know it isn’t, and there are probably harder times coming for us. I am sorry that I can not always give to you all the things you so richly deserve — I’d give you anything, I’d buy you anything if I could. I am sorry I don’t have anything I can give you today other than my words — darn those hard times!

My gift to you is this — my expression of how much I truly value you, cherish you and how grateful I am — in spite of all the hard times — the good times, the memories, our daughters and most importantly our love. I am but a broken man, but with you I am whole.

Thank you for being who you are.

Thank you for being with me.

Thank you for loving me.

Thank you for letting me love you in return.

Jesse

p.s. Churchill loves you too:

February 13, 2012 05:32 PM

PyPy Development

A Larger Example for the Flow Graph Language

Part 4 of Comparing Partial Evaluation to Tracing

This is the fourth and final blog post in a series about comparing partial evaluation and tracing. We've come a long way: In the first post of the series I showed an interpreter for a small flow-graph language together with a partial evaluator it. In the second post I showed how a tracer for the same language works and how it relates to both execution and to partial evaluation. The third post described an optimizer for traces.

In this final post we can compare and contrast the two different approaches of tracing and partial evaluation by means of an example. The programs in the flow chart language seen so far have been rather small, so I want to give an example of a larger program: an interpreter for an extremely simple bytecode instruction set. I will look at how the partial evaluator deals with that interpreter, and what the tracer does with it. The code for that, as well as all the code of the series can be found here: http://paste.pocoo.org/show/550282/ (some small additions have been made, such as a nicer way to print traces).

A Bytecode Interpreter

Writing programs in the flow graph language is painful, but I still want to give an example that is a bit more interesting than the tiny ones that we've seen so far. The example is an interpreter for the bytecode of a very trivial register-based language. The language has four registers, one of which is an accumulator on which all the actual operations are performed.

The opcodes of the language are:

• jump_if_a, jumps to a target address when the accumulator is non-zero
• mov_a_r0, mov_a_r1, mov_a_r2 move the value of the accumulator to the respective register
• mov_r0_a, mov_r1_a, mov_r2_a move the value of a register to the accumulator
• add_r0_to_a, add_r1_to_a, add_r2_to_a add the value of the register to the accumulator
• decr_a decrement the accumulator
• return_a stop the program and print the accumulator

The interpreter has a main loop that reads the opcode at the current program counter, does a (lengthy) dispatch to the right bytecode via a series of if statements and then executes the right opcode. Afterwards the next opcode is treated equivalently.

Here is a part of the source code in the flow graph language. As pseudocode:

bytecode_loop:
    opcode = bytecode[pc]
    pc = pc + 1
    c = opcode == 'jump_if_a'
    if c goto op_jump_if_a else goto not_jump_if_a

# select the right bytecode via a long series of if statements
not_jump_if_a:
    c = opcode == 'mov_a_r0'
    if y goto op_mov_a_r0 else goto not_mov_a_r0
not_mov_a_r0:
    c = opcode == 'mov_a_r0'
    if y goto op_mov_a_r1 else goto not_mov_a_r1
...

# bytecode implementations
op_mov_a_r0:
    r0 = a
    goto bytecode_loop

op_jump_if_a:
    c = a == 0
    target = bytecode[pc]
    pc += 1
    if c goto bytecode_loop else goto op_jump_if_a_jump

op_jump_if_a_jump:
    pc = target
    goto bytecode_loop
...

And actually working, as Prolog facts (the full implementation can be found at the link above):

% bytecode dispatch loop
block(bytecode_loop,
      op2(opcode, readlist, var(bytecode), var(pc),
      op2(pc, add, var(pc), const(1),
      op2(c, eq, var(opcode), const(jump_if_a),
      if(c, op_jump_if_a, not_jump_if_a))))).

% select the right bytecode via a long series of if statements
block(not_jump_if_a,
      op2(c, eq, var(opcode), const(mov_a_r0),
      if(c, op_mov_a_r0, not_mov_a_r0))).
block(not_mov_a_r0,
      op2(c, eq, var(opcode), const(mov_a_r1),
      if(c, op_mov_a_r1, not_mov_a_r1))).
...

% bytecode implementations
block(op_jump_if_a,
      op2(c, eq, var(a), const(0),
      op2(target, readlist, var(bytecode), var(pc),
      op2(pc, add, var(pc), const(1),
      if(c, bytecode_loop, op_jump_if_a_jump))))).
block(op_jump_if_a_jump,
      op1(pc, same, var(target),
      promote(bytecode, bytecode_loop))).
block(op_mov_a_r0,
      op1(r0, same, var(a), jump(bytecode_loop))).
...

The bytecode_loop block is the main dispatch loop. It reads an opcode out of the bytecode list at the program counter position, then has a long series of if statements that compares the current opcode to the various existing opcodes. The full code of the interpreter can be found under the link above.

The bytecodes of the interpreter don't really permit hugely complex programs, but it can be used to write a program that computes the square of a number with the following program:

mov_a_r0     # r0 = a
mov_a_r1     # r1 = a
# 2:
mov_r0_a     # r0--
decr_a
mov_a_r0
mov_r2_a     # r2 += a
add_r1_to_a
mov_a_r2
mov_r0_a     # if r0!=0: goto 2
jump_if_a 2
mov_r2_a     # return r2
return_a

Partially Evaluating the Bytecode Interpreter

The partial evaluator from the first blog post can be easily used to partially evaluate the bytecode interpreter. The static input is the bytecode for computing the square and the initial program counter value, as given above. The dynamic input are the content of the accumulator (the number to be squared). This can be done as follows:

?- bytecode_square(B),
Env = [bytecode/B, pc/0],
do_pe(bytecode_loop, Env, Label),
REnv = [a/16, r0/0, r1/0, r2/0],
interp(jump(Label), REnv), listing(block).
256
:- dynamic block/2.

<lots of generated code>

The code that is generated by the partial evaluation process is somewhat hard to read. It contains a lot of passages like this:

...
block(op_return_a1, print_and_stop(var(a))).
block(not_decr_a1, jump(op_return_a1)).
block(not_add_r2_to_a2, jump(not_decr_a1)).
block(not_add_r1_to_a2, jump(not_add_r2_to_a2)).
block(not_add_r0_to_a3, jump(not_add_r1_to_a2)).
block(not_mov_r2_a3, jump(not_add_r0_to_a3)).
block(not_mov_r1_a5, jump(not_mov_r2_a3)).
block(not_mov_r0_a5, jump(not_mov_r1_a5)).
block(not_mov_a_r27, jump(not_mov_r0_a5)).
block(not_mov_a_r18, jump(not_mov_a_r27)).
block(not_mov_a_r09, jump(not_mov_a_r18)).
block(not_jump_if_a11, jump(not_mov_a_r09)).
block(bytecode_loop12, jump(not_jump_if_a11)).
block(op_mov_r2_a2, op1(a, same, var(r2), jump(bytecode_loop12))).
...

I.e. lots of blocks that do nothing but jump to another block, interspersed with some blocks that contain an actual operation. I cleaned the output up manually and got something like the following (this sort of cleanup is something a good partial evaluation system would do itself, after partial evaluation has occurred):

block(bytecode_loop1,
    op1(r0, same, var(a),
    op1(r1, same, var(a),
    op1(a, same, var(r0),
    op2(a, sub, var(a), const(1),
    op1(r0, same, var(a),
    op1(a, same, var(r2),
    op2(a, add, var(a), var(r1),
    op1(r2, same, var(a),
    op1(a, same, var(r0),
    op2(c, eq, var(a), const(0),
    if(c, bytecode_loop11, op_jump_if_a_jump1)))))))))))).

block(bytecode_loop11,
    op1(a, same, var(r2),
    print_and_stop(var(a))).

block(op_jump_if_a_jump1,
    op1(a, same, var(r0),
    op2(a, sub, var(a), const(1),
    op1(r0, same, var(a),
    op1(a, same, var(r2),
    op2(a, add, var(a), var(r1),
    op1(r2, same, var(a),
    op1(a, same, var(r0),
    op2(c, eq, var(a), const(0),
    if(c, bytecode_loop11, op_jump_if_a_jump1)))))))))).

What do we see here? The partial evaluator has generated a block bytecode_loop1, which corresponds to the initialization opcodes mov_a_r0 and mov_a_r1 together with one iteration of the loop. Then it either jumps to a copy of the main loop (label op_jump_if_a_jump1) or to block bytecode_loop11, which prints the result and then stops. The residual code does exactly what the bytecode did: It squares the accumulator then prints that. All the uses of the bytecode and pc variable are gone.

Why did the partial evaluator produce two copies of the main loop that look the same? The reason for that is that in the second copy, the additional static information target = 2 is known, where target is a variable in the interpreter source that stores the jump target, for very brief periods of time. This additional static information does not have any effect on the residual code, so the same code is uselessly generated twice. This is an example of overspecialization.

Tracing the Interpreter

In this section we will look at what happens if we try to trace the interpreter. The naive way of doing that yields traces that are not very useful, because they abort after one iteration. We will look at a way of avoiding this problem. The problems described in this section are at the core of the paper Tracing the meta-level: PyPy's tracing JIT compiler (that paper uses a slightly more advanced version of the bytecode interpreter as an example).

To trace the interpreter, it is useful to change the bytecode_loop block from above to always promote the bytecode and the pc variables, because without knowing them the trace produced is not really interesting. This is similar to making these variables static in the partial evaluation example above:

block(bytecode_loop,
      promote(bytecode, bytecode_loop_promote_bytecode)).
block(bytecode_loop_promote_bytecode,
      promote(pc, bytecode_loop_promote_pc)).
block(bytecode_loop_promote_pc,
      op2(opcode, readlist, var(bytecode), var(pc),
      op2(pc, add, var(pc), const(1),
      op2(c, eq, var(opcode), const(0),
      if(c, op_jump_if_a, not_jump_if_a))))).
...

The rest of the interpreter stays unchanged.

To trace the interpreter we would start naively at the bytecode_loop label, because that's the label in the interpreter that is jumped to most often (which a profiler could establish easily). The following command can be used for that (this output prints traces in a slightly more readable way than in previous blog posts):

?- bytecode_square(B),
        A = 16, Env = [bytecode/B, pc/2, a/A, r0/A, r1/A, r2/0],
        do_trace(bytecode_loop, Env).
trace
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,2,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  op2(c,eq,var(opcode),const(jump_if_a))
  guard_false(c,[],op_jump_if_a)
  op2(c,eq,var(opcode),const(mov_a_r0))
  guard_false(c,[],op_mov_a_r0)
  op2(c,eq,var(opcode),const(mov_a_r1))
  guard_false(c,[],op_mov_a_r1)
  op2(c,eq,var(opcode),const(mov_a_r2))
  guard_false(c,[],op_mov_a_r2)
  op2(c,eq,var(opcode),const(mov_r0_a))
  guard_true(c,[],not_mov_r0_a)
  op1(a,same,var(r0))
  loop

opttrace
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,2,[bytecode/[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]],bytecode_loop_promote_pc)
  op1(a,same,var(r0))
  op1(bytecode,same,const([mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]))
  op1(pc,same,const(3))
  op1(opcode,same,const(mov_r0_a))
  op1(c,same,const(1))
  loop

256
B = [mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a, mov_a_r2, mov_r0_a|...],
A = 16,
Env = [bytecode/[mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a|...], pc/2, a/16, r0/16, r1/16, r2/0]

These traces are very short. They start with promoting the bytecode and the pc, followed by the execution of the opcode mov_r0_a, which is the one at position 2 in the given bytecode. Then they increment the pc and loop back to the beginning. Looking at the optimized trace, it is clear that the trace is essentially useless. It will run only for one iteration, because in the second iteration the pc is 3, thus the guard_value at the beginning will fail.

This problem can be solved by tracing more than just one iteration of the bytecode dispatch loop, which is called meta-tracing. To get this behaviour, in this simple example it is enough to start (and thus end) tracing at a different label, op_jump_if_a_jump. This label is hit when the interpreter executes a jump_if_a bytecode and the jump is taken. In a loop on the level of the executed bytecode program there is one such jump. Thus tracing from this label, a full loop in the bytecode program is traced, containing potentially many iterations of the bytecode dispatch loop in the control flow graph language.

Doing that yields the following:

?- bytecode_square(B),
        A = 16, Env = [bytecode/B, pc/11, a/A, r0/A, r1/A, r2/0, target/2],
        do_trace(op_jump_if_a_jump, Env).
trace
  op1(pc,same,var(target))
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop)
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,2,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  op2(c,eq,var(opcode),const(jump_if_a))
  guard_false(c,[],op_jump_if_a)
  op2(c,eq,var(opcode),const(mov_a_r0))
  guard_false(c,[],op_mov_a_r0)
  op2(c,eq,var(opcode),const(mov_a_r1))
  guard_false(c,[],op_mov_a_r1)
  op2(c,eq,var(opcode),const(mov_a_r2))
  guard_false(c,[],op_mov_a_r2)
  op2(c,eq,var(opcode),const(mov_r0_a))
  guard_true(c,[],not_mov_r0_a)
  op1(a,same,var(r0))
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,3,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  ...
  lots of operations ommitted
  ...
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,9,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  op2(c,eq,var(opcode),const(jump_if_a))
  guard_true(c,[],not_jump_if_a)
  op2(c,eq,var(a),const(0))
  op2(target,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  guard_false(c,[],bytecode_loop)
  loop

opttrace
  op1(pc,same,var(target))
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop)
  guard_value(pc,2,[bytecode/[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]],bytecode_loop_promote_pc)
  op1(a,same,var(r0))
  op2(a,sub,var(a),const(1))
  op1(r0,same,var(a))
  op1(a,same,var(r2))
  op2(a,add,var(a),var(r1))
  op1(r2,same,var(a))
  op1(a,same,var(r0))
  op2(c,eq,var(a),const(0))
  guard_false(c,[bytecode/[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],pc/11,opcode/jump_if_a,target/2],bytecode_loop)
  op1(bytecode,same,const([mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]))
  op1(pc,same,const(11))
  op1(opcode,same,const(jump_if_a))
  op1(target,same,const(2))
  op1(c,same,const(0))
  loop

256
B = [mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a, mov_a_r2, mov_r0_a|...],
A = 16,
Env = [bytecode/[mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a|...], pc/11, a/16, r0/16, r1/16, r2/0, target/2] .

That looks better. The trace corresponds to the interpreter running all the bytecodes in the loop of the squaring function in the example bytecode above. The optimized code starts with two guards (checking that the bytecode is still the one for the squaring function, checking that the pc is 2) and then only does the operations that actually do the computation. No bytecode dispatching is performed, thus the interpretation overhead is fully removed, apart from the two guard_value operations at the beginning.

Many of the assignments in the trace are superfluous, e.g. all the copying back and forth between registers r1, r1, r2 and accumulator a. This could be easily solved by an even more intelligent optimization utilizing SSA form.

Conclusion About the Interpreter

Both partial evaluation and meta-tracing can be used to transform the example bytecode computing a square into a form that shows the essential computation that is going on, without the interpretation overhead. The naive partial evaluator produces lots of extra blocks that just jump around, which could be solved with a post-processing step. The tracer by itself produces uselessly short traces, but with a simple trick of starting the trace at a different point the results become a lot better.

In a real meta-tracing system, the meta-tracer would need a way for the author of the interpreter to mark which bytecode corresponds to a backward jump. It would also need better integration with the interpreter to start tracing automatically, as well as cache the traces. Additionally, it would have to deal better with guards that fail a lot, attaching new traces to the failing guards. However, all that is "just" engineering on top of the ideas presented in this series of blog posts.

High-Level Conclusion

Some concluding high-level thoughts about the similarities of tracing and partial evaluation: Tracing and partial evaluation try to tackle a similar problem, that of automatically reducing the interpreter overhead, their approaches are slightly different though.

Tracing is very close to normal evaluation, only keeping some extra information in the process. But then, the optimizer that is used in a tracer is again very similar in structure to a partial evaluator. The task of the optimizer is much simpler though, because it does not need to deal with control flow at all, just a linear list of operations.

So in a sense tracing is taking those parts of partial evaluation that work (the "just evaluate those things that you can, and leave the others") and replacing the parts that don't (controlling unfolding) by a much more pragmatic mechanism. That mechanism observes actual execution runs of the program to choose control flow paths that are typical. At the same time, the tracer's focus is on loops, because they are where most programs spend significant amounts of time.

Another point of view of tracing is that it is a form of partial evaluation that replaces the control components of a partial evaluator with an oracle (the actual execution runs) that provide the information which paths to look at.

Already in the quite trivial interpreter here the effects of this are visible. The simple partial evaluator over-specializes the loop and produces two identical versions of it, that aren't different. The tracer doesn't, and it also generates only code for the loop itself, not for the initialization opcodes.

That's it for this series. To those that made it, thanks for following along. Also thanks to Samuele and Sven, who consistently gave me good feedback on the posts before I put them here.

February 13, 2012 03:50 PM

Matt Harrison

Amazon KDP Non-Fiction Week 2 Update

I am two weeks into my Amazon KDP non-fiction experiment. Guide to: Learning Python Decorators is currently #15 on the Python list as I write this, but was up at 5 during the week. It is aso #1 on the Python “Hot New Releases”, which surprises me because #2 is a really strong title. Sales of both of my books have been slow and steady. I heard that the average “indie” author only sells 4 books/month. I’m happy to report that I’m slightly better than that and I will be receiving a check from Amazon this month ($100+ in sales).

What else have I done this week? I have wondered about Amazon’s lack of existence in BRIC (Brazil, Russia, India and China). When I had my own store front, I was able to sell to a few individuals from some of those countries. Now I am not because Amazon has no presence there. Again this is probably not something Indie fiction writers care about (with perhaps the exception of India) — I assume fiction is somewhat language dependent. But many programmers know English, so presumably I am losing some sales. How big would they be? I am not sure, but I have received at least one request from someone who claimed they did not have Kindle reader access. Currently my non-US sales are less than 10% of US sales. I am not sure if that is a function of my marketing, or how strong the Amazon presence is in those countries (UK, DE, ES, IT, FR), or some combination of the two.

I also signed up for Amazon Author Central. Perhaps this will help with SEO, I am not sure, but I now have an author page, which also aggregates this blog and my tweetstream.

While at the Utah Python group this week, the first presenter actually mentioned my decorator book in his talk. When the meeting ended, an attendee informed me that he thought my book was pretty good (which he had purchased and was reading during the second presentation). Also he gave me some more feedback, which is always nice.

I also spent some time looking into pbook generation. My calculation is around 4% of people would prefer dead tree books (very unscientific, consists of requests divided by current sales at some point in time). KDP only requires exclusive digital rights to your sales, not physical, so this is a very real possibility. But, my current pdf generation has issues and I starting learning LaTex to solve those. I hope to have some sort of dead tree versions released soon. I’m thinking about using CreateSpace, but would be open to using other POD services. If you have experience with POD feel free to drop me some notes.

My book was submitted to the January 2012 Ebook Cover Design Awards, which was released this week. I was hoping to get some feedback on my cover, but did not. I interpret that response as an indication of the “blah”-ness of my cover. Not sufficiently interesting to warrant any praise, nor blatantly bad enough to expose the horrible design elements.

Finally, a huge advantage to KDP, is I can easily push updates and Amazon handles everything else. A disadvantage is anonymity of purchases. With sales from my site, I had emails for all purchases and could interact with buyers, but pushing updates was a bit more of a pain. Some food for thought if you are considering one or the other. I really like to think that ebooks are somehow “alive” and that I can incorporate reader feedback very quickly and actually push it out, though this is not something that “real” publishers seem to be doing yet. I’m thinking that I might stay on KDP with my more expensive/longer Treading on Python which receives a higher percentage of “rentals”, and make distribution of smaller ebooks more widely available — Kobo, my own site, and perhaps BN again.

February 13, 2012 01:03 PM

Frank Wierzbicki

Jython dev notes part II - Adding a New Builtin Type

Releases of Python (and so releases of Jython) sometimes add new built-in types. In 2.6, a new such buitin is the "bytes" type. In the 2.x series, "bytes" is just a synonym for "str". In 3.x "bytes" is the name used for 8 bit strings while "str" is a unicode string (and so the "unicode" type disappears in 3.x). This will make it a great example for adding a builtin since there is no added functionality to obscure the basics.

First we'll verify that no bytes builtin exists at this time:

[frank jython]$ ./dist/bin/jython
Jython 2.6a0+ (default:9c9c311c201b, Feb 10 2012, 10:29:32) 
[OpenJDK 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_23
Type "help", "copyright", "credits" or "license" for more information.
>>> bytes('foo')
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'bytes' is not defined

[frank jython]$ hg diff
diff --git a/src/org/python/core/__builtin__.java b/src/org/python/core/__builtin__.java
--- a/src/org/python/core/__builtin__.java
+++ b/src/org/python/core/__builtin__.java
@@ -305,6 +305,7 @@
         dict.__setitem__("Ellipsis", Py.Ellipsis);
         dict.__setitem__("True", Py.True);
         dict.__setitem__("False", Py.False);
+        dict.__setitem__("bytes", PyString.TYPE);
 

        dict.__setitem__("str", PyString.TYPE);

[frank jython]$ ./dist/bin/jython
*sys-package-mgr*: processing modified jar, '/home/frank/hg/jython/jython/dist/jython-dev.jar'
Jython 2.6a0+ (default:9c9c311c201b+, Feb 13 2012, 09:52:10) 
[OpenJDK 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_23
Type "help", "copyright", "credits" or "license" for more information.
>>> x = bytes('foo')
>>> x
'foo'
>>> type(x)
<type 'str'>

February 13, 2012 01:01 PM

Katie Cunningham

Pygame: Getting started

As I've mentioned before, I'll be helping Richard Jones with a tutorial on PyGame at PyCon this year. When I agreed to help him, I admit, I knew nothing about PyGame. I was coming on as someone who would talk about tropes and designs and pitfalls (and the crusher of ...

February 13, 2012 12:44 PM

• • RSS feed
  • Powered by Planet!
• Other Python Planets
  • Planet Jython
  • Planet IronPython
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
  • Zope
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • 2degrees
  • A Little Bit of Python
  • Aahz
  • Adam Pletcher
  • Adrián Deccico
  • Alec Munro
  • Alex Clark
  • Alex Gaynor
  • Alexander Artemenko
  • Alexander Limi
  • Alexandre Bourget
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Ali Afshar
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrew Dalke
  • Andriy Drozdyuk
  • Andy Dustman
  • Andy Todd
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Armin Ronacher
  • Arthur Koziel
  • Ashish Vidyarthi
  • Ashley Camba
  • Atamert Olcgen
  • Atul Varma -- Toolness
  • Audrey Roy
  • Baiju Muthukadan
  • Balthazar Rouberol
  • Barbara Shaurette
  • Barry Warsaw
  • Ben Bangert
  • Ben Bass
  • Benjamin Peterson
  • Benjamin W. Smith
  • Benji York
  • Bertrand Mathieu
  • Bill Mill
  • BioPython News
  • Bit of Cheese
  • Björn Tillenius
  • Blended Technologies
  • BlueBream
  • BlueDynamics Alliance
  • Bojan Mihelac
  • Brandon Rhodes
  • Brett Cannon
  • Brian Curtin
  • Brian Harring
  • Brian Jones
  • Brian Jones
  • Bryce Verdier
  • Caktus Consulting Group
  • Calvin Spealman
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catherine Devlin
  • Chad Cooper
  • Chad Whitacre
  • Checking and Sharing
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Perkins
  • Christian Heimes
  • Christian Joergensen
  • Christian Scholz
  • Christopher Denter
  • Christopher Lenz
  • Christopher Perkins
  • Chuck Thier
  • Chui Tey
  • Cloudnumbers
  • CodeSnipers
  • Collin Winter
  • Corey Goldberg
  • Cormoran Project
  • Cosmic Seriosity Balance
  • Craig Kerstiens
  • Creative Commons
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Daniel Brown
  • Daniel Greenfeld
  • Daniel Nouri
  • Dariusz Suchojad
  • Dave Beazley
  • David Ascher
  • David Goodger
  • David Grant
  • David J C Beach
  • David Janes
  • David Malcolm
  • David Stanek
  • Davy Mitchell
  • Davy Wybiral
  • Dazza
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diane Trout
  • Django Weblog
  • Djangocon
  • Djangofeed
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • Duncan McGreggor
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eli Bendersky
  • EmptysquarePython
  • Enthought
  • Eric Florenzano
  • Eric Holscher
  • Eric Wu's Pythonfilter
  • EuroPython
  • Evan Fosmark
  • Fabio Zadrozny
  • Fazal Majid
  • Flavio Coelho
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Marier
  • Fredrik Håård's Blaag
  • Fredrik Lundh
  • From Python Import Podcast
  • Gael Varoquaux
  • Gary Bernhardt
  • Gary Wilson
  • Geek Scrap
  • Geert Vanderkelen
  • Gene Campbell
  • Georg Brandl
  • Gilad Raphaelli
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Godson Gera
  • Gonçalo Margalho
  • Graham Dumpleton
  • Gramps
  • Grant Baillie
  • Greg Taylor
  • Greg Turnquist
  • Greg Wilson
  • Gregory Pinero
  • Grig Gheorghiu
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hanno Schlichting
  • Hans Nowak
  • Hany Fahim
  • Harrison Erd
  • Hector Garcia
  • Heikki Toivonen
  • Hilary Mason
  • Holger Krekel
  • Huy Nguyen
  • IPython0 blog
  • Ian Bicking
  • Ian Ozsvald
  • Ilian Iliev
  • Imaginary Landscape
  • Invent with Python
  • Ionel Cristian
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • Isotoma
  • Ivan Krstic
  • Ivan Krstic
  • Jack Diederich
  • Jacob Perkins
  • Jaime Buelta
  • James Polera
  • James Tauber
  • Jared Forsyth
  • Jarrod Millman
  • Jason Baker
  • Jason Hildebrand
  • Jason Webb
  • Jean-Paul Calderone
  • Jeet Sukumaran
  • Jeethu Rao
  • Jeff Hinrichs
  • Jeff McNeil
  • Jeff Rush
  • Jeff Shell
  • Jeff Winkler
  • Jeremy Hylton
  • Jesse Noller
  • Jim Baker
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Paulett
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan LaCour
  • Jonathan Street
  • Juan Rivas
  • Juho Vepsäläinen
  • Julien Anguenot
  • Julien Palard
  • Juri Pakaste
  • Just a little Python
  • Kai Diefenbach
  • Kai Lautaportti
  • Kamon Ayeva
  • Katie Cunningham
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Konrad Delong
  • Kristján Valur
  • Krys Wilken
  • Kulbir Saini
  • Kumar McMillan
  • Kun Xi
  • Kun Xi
  • Laurent Luce
  • Laurent Szyster
  • Lawrence Oluyede
  • Lee Braiden
  • Leigh Honeywell
  • Lennart Regebro
  • Lesscode.org
  • Lightning Fast CMS
  • Lightning Fast Shop
  • Lion Kimbro
  • Lionel Tan
  • Logilab
  • Low Kian Seong
  • Ludovico Fischer
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Malthe Borch
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marcin Kuźmiński
  • Marcus Whybrow
  • Mario Boikov
  • Marius Gedminas
  • Mark Dennehy
  • Mark Dufour
  • Mark McMahon
  • Mark Mruss
  • Mark Nottingham
  • Mark Paschal
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martin Blais
  • Mathieu Fenniak
  • Mathieu Virbel
  • Matt Goodall
  • Matt Harrison
  • Matt Wilkes
  • Matthew Rollings
  • Matthew Scott
  • Matthew Scott
  • Matthew Wilson
  • Mattias Brändström
  • Max Ischenko
  • Max Khesin
  • Menno's Musings
  • Michael Bayer
  • Michael Crute
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael J.T. O'Kelly
  • Michael Nelson
  • Michael Schurter
  • Michael Sparks
  • Michael Watkins
  • Michal Kwiatkowski
  • Michał Domański
  • Michele Simionato
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Naberezny
  • Mike Pirnat
  • Mikeal Rogers
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Montreal Python User Group
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Nadav Samet
  • Nadia Alramli
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Piël
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nicolas Dumazet
  • Nicolas Evrard
  • Nicolas Paris
  • Nikhil Gopal
  • Noah Gift
  • Nuxeo
  • Omaha Python Users Group
  • Omar Abo-Namous
  • Ondřej Čertík
  • OpenHatch Python posts
  • Orbited
  • Orestis Markou
  • Partecs
  • Patrice Neff
  • Patricio Paez
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paulo Nuin
  • Pedro Kroger
  • Pedro Lima
  • Pete Hunt
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Halliday
  • Peter Harkins
  • Peter Hoffmann
  • Peter Parente
  • Petro Verkhogliad
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Phillip Pearson - Second p0st
  • Pietro Abate
  • Pradeep Gowda
  • PyAMF Blog
  • PyCharm
  • PyCon
  • PyCon 2008 on YouTube
  • PyCon Australia
  • PyCon Australia 2012 and 2013
  • PyCon Podcast
  • PyLadies
  • PyPy Development
  • PyTexas
  • Pylons News Feed
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python African Tour
  • Python Diary
  • Python Insider
  • Python News
  • Python Open Mike
  • Python Software Foundation
  • Python Sprints
  • Python Testing Cookbook
  • Python User Groups
  • PythonThreads
  • Pythonology
  • Péter Szabó
  • R David Murray
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Ramana
  • Randell Benavidez
  • Raymond Hettinger
  • Reinout van Rees
  • Rene Dudfield
  • Richard Jones
  • Richard Tew
  • Rob Cakebread
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Lujo
  • Roberto Alsina
  • Robin Dunn
  • Robin Parmar
  • Roche Compaan
  • Rocky Burt
  • Rodrigo Araúj
  • Rok Garbas
  • Ruslan Spivak
  • Ryan Cox
  • Ryan Tomayko
  • S. Lott
  • SDJournal
  • SPE Weblog
  • Salman Haq
  • Samuel Sutch
  • Sandro Tosi
  • Scripting the web with Python
  • Sean McGrath
  • Sean Reifschneider
  • Senthil Kumaran
  • Shannon -jj Behrens
  • ShiningPanda
  • Shriphani Palakodety
  • Simeon Franklin
  • Simon Brunning
  • Simon Willison
  • Simon Wittber
  • Speno's Pythonic Avocado
  • SpringSource
  • Spyder IDE
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Steven Kryskalla
  • Steven Wilcox
  • Stijn Debrouwere
  • Swaroop C H
  • Sylvain Hellegouarch
  • Tarek Ziade
  • Ted Leung
  • Ted Nyman
  • Teemu Harju
  • Tennessee Leeuwenburg
  • Terry Jones
  • Terry Peppers
  • The Artificial Intelligence Cookbook
  • The Blagin' Wraith
  • The Monkey Project
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • Thomas Guest
  • Thomas Vander Stichele
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim Parkin
  • Titus Brown
  • Tobias Ivarsson
  • Tom Christie
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Vern Ceder
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Virgil Dupras
  • Viva La Chipperfish
  • Wallix
  • Washington Times OpenSource
  • Wayne Witzel
  • Wes Mason
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Reade
  • William's Journal
  • Wingware
  • Wolfram Kriesing
  • Wyatt Baldwin
  • Xavier Spriet
  • Yaniv Aknin
  • Yannick Gingras
  • Yuval Greenfield
  • Zachary Voase
  • Zeth
  • codeboje
  • eGenix.com
  • nl-project
  • planet.python.org updates
  • wiredobjects
  • Éric Araujo
  • To request addition or removal:
    e-mail planet at python.org (note, responses can take up to a few days)