Planet Python

Mike Driscoll: wxPython: All about Menus

Tue, 14 Feb 2012 20:16:01 +0000

Menus are ubiquitous. They’re in almost all desktop programs. You use them for editing preferences or configuring your program. In wxPython, there are several menu options to choose from. The most familiar is probably wx.Menu. But there are are popup menus and a pure Python implementation known as FlatMenu. We will just be covering wx.Menu and popup menus here because they’re related to each other. FlatMenu also includes a toolbar API, so you’ll have to wait for another article that will just cover that widget all by itself. Let’s get this menu party started!

A Simple Menu Example

We’ll start with something really simple: a menu with just an Exit MenuItem. Here’s the code:

import wx
 
########################################################################
class MyForm(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="wx.Menu Tutorial")
 
        self.panel = wx.Panel(self, wx.ID_ANY)
 
        menuBar = wx.MenuBar()
        fileMenu = wx.Menu()
        exitMenuItem = fileMenu.Append(wx.NewId(), "Exit",
                                       "Exit the application")
        menuBar.Append(fileMenu, "&File")
        self.Bind(wx.EVT_MENU, self.onExit, exitMenuItem)
        self.SetMenuBar(menuBar)
 
    #----------------------------------------------------------------------
    def onExit(self, event):
        """"""
        self.Close()
 
#----------------------------------------------------------------------
# Run the program
if __name__ == "__main__":
    app = wx.App(False)
    frame = MyForm().Show()
    app.MainLoop()

Let’s break this down a little. To create the menubar, we instantiate an instance of wx.MenuBar. Then we create a an instance of wx.Menu that we call filemenu. Finally, to add the “Exit” item, we append some data to it. In essence, we are appending a wx.MenuItem, but this is kind of a shortcut since we didn’t actually create a wx.MenuItem instance first. We’ll show how to do that in our next example though. Note that when we append the item, we have to pass an id, a label string and a status string. That last one will show up in the Status bar when you hover over the menu item, assuming you have a status bar. Note that to attach an event handler to a menu item, you need use the EVT_MENU event and bind it to the frame. Next we append the Menu itself to the MenuBar object and pass it a string too, which in this case is “File”. Finally we call the frame’s SetMenuBar method to attach the menu bar to the frame.

Adding a Picture to a Menu

That’s all there is to creating a menu! Now let’s look at a more complex example! Note: to follow the example below, you’ll need to use your own image file.

import wx
 
########################################################################
class MyForm(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="wx.Menu Tutorial")
 
        self.panel = wx.Panel(self, wx.ID_ANY)
 
        # create the menubar
        menuBar = wx.MenuBar()
 
        # create the first menu (starting on left)
        carMenu = wx.Menu()
        carMenu.Append(101, "&Ford", "An American Automaker")
        carMenu.Append(102, "&Nissan", "")
        carMenu.Append(103, "&Toyota", "Buy Japanese!")
        carMenu.Append(104, "&Close", "Close the application")
 
        # add a picture to a menu
        picMenu = wx.Menu()
        item = wx.MenuItem(picMenu, wx.ID_ANY, "Snake", "This menu has a picture!")
        img = wx.Image('snake32.bmp', wx.BITMAP_TYPE_ANY)
        item.SetBitmap(wx.BitmapFromImage(img))
        picMenu.AppendItem(item)
 
        # add menus to menubar
        menuBar.Append(carMenu, "&Vehicles")
        menuBar.Append(picMenu, "&Picture")
        self.SetMenuBar(menuBar)
 
    #----------------------------------------------------------------------
    def onExit(self, event):
        """"""
        self.Close()
 
#----------------------------------------------------------------------
# Run the program
if __name__ == "__main__":
    app = wx.App(False)
    frame = MyForm().Show()
    app.MainLoop()

This example is similar to the first one. The main difference is that we add multiple items to out File menu and we have two menus. Note that this time, we actually specify our ID numbers explicitly. This usually isn’t recommended as you might accidentally shadow one of wx’s required IDs. However, you do see examples of this from time to time on the web, so you should be aware of this practice. The next big difference doesn’t come until we get to the picMenu. Here we actually create a wx.MenuItem and add a picture to it via wx.Image and the MenuItem’s SetBitmap method. The rest is pretty much the same.

Now we will spend some time looking at how to add radio and check buttons to our menu.

Adding Radio or Check Buttons

Adding a radio or check button to your menu is actually pretty easy. Let’s take a moment and see how it’s done!

# radiocheck.py
import wx
 
########################################################################
class MyForm(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="wx.Menu Tutorial")
 
        self.panel = wx.Panel(self, wx.ID_ANY)
 
        # Create menu bar
        menuBar = wx.MenuBar()
 
        # Create radio menu
        radioMenu = wx.Menu()
        idleItem = radioMenu.Append(wx.NewId(), "IDLE",
                                   "a Python shell using tcl/tk as GUI",
                                   wx.ITEM_RADIO)
        pyCrustItem = radioMenu.Append(wx.NewId(),"PyCrust",
                                      "a Python shell using wxPython as GUI",
                                      wx.ITEM_RADIO)
        psiItem = radioMenu.Append(wx.NewId(), "psi",
                                  "a simple Python shell using wxPython as GUI",
                                  wx.ITEM_RADIO)
        menuBar.Append(radioMenu, "&Radio")
 
        # create check menu
        checkMenu = wx.Menu()
        wgItem = checkMenu.Append(wx.NewId(), "Wells Fargo", "", wx.ITEM_CHECK)
        citiItem = checkMenu.Append(wx.NewId(), "Citibank", "", wx.ITEM_CHECK)
        geItem = checkMenu.Append(wx.NewId(), "GE Money Bank", "", wx.ITEM_CHECK)
        menuBar.Append(checkMenu, "&Check")
 
        # Attach menu bar to frame
        self.SetMenuBar(menuBar)
 
#----------------------------------------------------------------------
# Run the program
if __name__ == "__main__":
    app = wx.App(False)
    frame = MyForm().Show()
    app.MainLoop()

Yes, as you can see, all you have to do is add a wx.ITEM_RADIO or wx.ITEM_CHECK flag as the kind parameter, which is the 4th parameter. Why is it called “kind” rather than “style” like other widgets? Well, while discussing this on the wxPython IRC channel, Robin Dunn (creator of wxPython) pointed out that it may be because these are different kinds of menu items.

Sub-Menus

The wxPython library also supports sub-menus. Here’s a really simple example to show you how it’s done.

# submenu.py
import wx
 
########################################################################
class MyForm(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="wx.Menu Tutorial")
 
        self.panel = wx.Panel(self, wx.ID_ANY)
 
        menuBar = wx.MenuBar()
        fileMenu = wx.Menu()
        openMenuItem = fileMenu.Append(wx.NewId(), "Open")
 
        # create a submenu
        subMenu = wx.Menu()
        historyMenuItem = subMenu.Append(wx.NewId(), "Show History")
        fileMenu.AppendMenu(wx.NewId(), "History", subMenu)
 
        exitMenuItem = fileMenu.Append(wx.NewId(), "Exit",
                                       "Exit the application")
        menuBar.Append(fileMenu, "&File")
        self.Bind(wx.EVT_MENU, self.onExit, exitMenuItem)
        self.SetMenuBar(menuBar)
 
    #----------------------------------------------------------------------
    def onExit(self, event):
        """"""
        self.Close()
 
#----------------------------------------------------------------------
# Run the program
if __name__ == "__main__":
    app = wx.App(False)
    frame = MyForm().Show()
    app.MainLoop()

The key here is that instead of using the filemenu’s Append method, we instead use its AppendMenu method. As the name implies, it allows the programmer to actually append a menu instead of a menu item. Yup, that’s it!

Pop-up Menus (AKA: ContextMenus)

Pop-up menus are the kind you see when you right-click on a link in a browser or on a file. They are also known as Context Menus. Here is a fairly trivial example for you to study:

# submenu.py
import wx
 
########################################################################
class MyForm(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="Popup Menu Tutorial")
 
        panel = wx.Panel(self, wx.ID_ANY)
 
        lbl = wx.StaticText(panel, label="Right click anywhere!")
        self.Bind(wx.EVT_CONTEXT_MENU, self.onContext)
 
    #----------------------------------------------------------------------
    def onContext(self, event):
        """
        Create and show a Context Menu
        """
 
        # only do this part the first time so the events are only bound once 
        if not hasattr(self, "popupID1"):
            self.popupID1 = wx.NewId()
            self.itemTwoId = wx.NewId()
            self.itemThreeId = wx.NewId()
            self.Bind(wx.EVT_MENU, self.onPopup, id=self.popupID1)
            self.Bind(wx.EVT_MENU, self.onPopup, id=self.itemTwoId)
            self.Bind(wx.EVT_MENU, self.onExit, id=self.itemThreeId)
 
        # build the menu
        menu = wx.Menu()
        itemOne = menu.Append(self.popupID1, "ItemOne")
        itemTwo = menu.Append(self.itemTwoId, "ItemTwo")
        itemThree = menu.Append(self.itemThreeId, "Exit")
 
        # show the popup menu
        self.PopupMenu(menu)
        menu.Destroy()
 
    #----------------------------------------------------------------------
    def onExit(self, event):
        """
        Exit program
        """
        self.Close()
 
    #----------------------------------------------------------------------
    def onPopup(self, event):
        """
        Print the label of the menu item selected
        """
        itemId = event.GetId()
        menu = event.GetEventObject()
        menuItem = menu.FindItemById(itemId)
        print menuItem.GetLabel()
 
#----------------------------------------------------------------------
# Run the program
if __name__ == "__main__":
    app = wx.App(False)
    frame = MyForm().Show()
    app.MainLoop()

To begin, we bind wx.EVT_CONTEXT_MENU to the frame. This allows us to right-click anywhere and fire the context menu event, which will create and show our popup menu. The code in the onContext method is based loosely on the wxPython demo for the popup menu. As you can see, we use a conditional statement to check if we’ve already bound the menu events. If we have, we don’t bind them again. Next we create our menu in much the same way as before. Finally, we call the frame’s PopupMenu method and pass it our new menu. This will show the menu to the user. When the user clicks on a menu item, it will fire an event and the menu will be destroyed.

The first two menu items are both bound to our onPopup method. This allows us to see how we can get access to the Menu and the MenuItem attributes. You can get the menu’s id with the event and the Menu itself with the event’s GetEventObject method. Then you can use the menu’s FindItemById method to get a handle to the menu item itself. Finally we print out the menu item’s label.

Wrapping Up

Now you should know most of the menu methods and how to create them, bind events and make different kinds of menu items. You even know how to create popup menus! Now you can make your applications have fancy menus too.

Source

Python Diary: Too many micro webframeworks

Tue, 14 Feb 2012 18:41:00 +0000

There is a large amount of Python micro web frameworks out in the wild, and some of them feel similar to eachother, at least when I first approach them. I plan on going through a few of these micro web frameworks to see what the exact differences are. I will not be comparing them to full frameworks such as Django or Web2py.

Bottle

First up, Bottle. It is very micro in that it comes in a single Py file and is very portable across platforms. If you need a simple Python web app running on say an embedded system with very low resources and disk space, this will be a good option for you. Here is a snippet of code from their home page of an example application:

from bottle import route, run

@route('/hello/:name')
def index(name='World'):
    return '<b>Hello %s!</b>' % name

run(host='localhost', port=8080)

Simple, is it not. One could deploy a Python console application which merely displays output for diagnose of a server or heartbeat in a matter of minutes. I wouldn't recommend it for full web applications, but no one is stopping you.

web.py

Next we have web.py. I have heard many good things about this micro framework, that doesn't make it sounds so micro. A good advantage of this web framework is the ability to separate the requests from both GET and POST. This is great when you need to develop an application which depends on the type of request coming through, such as a REST API. Here is a snippet from their home page:

import web
        
urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:        
    def GET(self, name):
        if not name: 
            name = 'World'
        return 'Hello, ' + name + '!'

if __name__ == "__main__":
    app.run()

A little more complex than Bottle, but it's still simpler than say Django or Web2py.

Flask

Looking for something even simpler than Bottle? Try Flask, there are less imports to remember, but it is ever so similar to Bottle. But, it comes with more than bottle, so you will have more than a single file to work with. It comes with the Jinja2 template engine, RESTful dispatching, and other goodies that other micro frameworks lack. Here's a snippet from their homepage:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Notice how similar this code is to Bottle's example code? If your looking for a little extra in a micro web framework, without all the extras of Django or Web2py, flask may be a good option for you.

CherryPy

Finally we get to CherryPy. I have heard many great things about this framework and many Python developers seem to favor it. I have not personally tried it myself, but I do plan on trying it out to see how it differs from other frameworks out there, micro and otherwise. They say the framework is a minimalistic one, but I still believe that Bottle is the smallest one around. It supports similar components to Django, such as caching, sessions, static files, and even authorization. It also appears to run on many platforms including Android. Here is the example code snippet from their home page:

import cherrypy
class HelloWorld(object):
    def index(self):
        return "Hello World!"
    index.exposed = True

cherrypy.quickstart(HelloWorld())

It does look fairly simple, but the last statement quickstart sort of scares me. This would imply that there are other methods of starting the application, in effect for larger applications, this simple code snippet above may not be the real deal. I will look into this one further and see how complex web applications are built and how it would differ from other micro frameworks listed here.

Final thoughts

Now that this entry is coming to a close, I would like to express my final thoughts on so-called micro frameworks. Are all of these micro frameworks truly needed? Does having this many cause fragmentation in how Python web development should be done? Does it cause developer confusion, on which one a developer should learn or avoid? Let me know what you think about this micro-framework situation us Python developers are faced with in the comments below.

PyCon: PyCon 2012: New Relic, Loggly and Skullcandy party!

Tue, 14 Feb 2012 14:47:59 +0000

One of the things we love about our sponsors is their willingness to not just sponsor PyCon - but to get involved, help volunteer, and well - help throw parties!

The first party is courtesy of New Relic, Loggly and Skullcandy - from their event page:

PyCon 2012 Attendees! Join New Relic, Loggly and Skullcandy for a special PyCon party event on March 9, from 7 pm - 9 pm at the Hyatt Regency Santa Clara (Across from the Convention Center).
Don't miss this opportunity to network with your Python Community peers over drinks and hors d'oeuvres.
Even better, bring your biz card and enter to win some sweet shwag from Skullcandy, maker of premier headphones, earbuds, and headsets. Best of all, it's free! Space is limited to 250 so be sure to register and reserve your spot today!

You can sign up for the party right over here on their events page - but you'd better hurry, spots are going fast.

Be warned… It's the night before the PyCon 5k, which starts at 7am sharp the next day :)

Kristján Valur: Clearing weakrefs

Tue, 14 Feb 2012 14:17:09 +0000

I just had this problem which would have been elegantly solved with the ability to manually clear weak references pointing to an object. I am (for technical reasons) recycling an object, so instead of killing it and re-creating it, I re-initialize it. But that leaves old weak references in place. How nice wouldn’t it be to be able to call “myobject.clear_weakrefs()”?

Rob Galanakis: Large initializers/ctors?

Tue, 14 Feb 2012 12:03:53 +0000

With closures (and to some extent with runtime attribute assignments), I find the signatures of my UI types shrink and shrink. A lot of times we have code like this (python, but the same would apply to C#):

class FooControl(Control):
  def __init__(self, value):
    super(FooControl).__init__()
    self.value = value
    self._InitButtons()    

  def _InitButtons(self):
    self.button = Button('Press Me!', parent=self)
    btn.clicked.addListener(self._OnButtonClick)

  def _OnButtonClick(self):
    print id(self.button), self.value

However we can easily rewrite this like so:

class FooControl(Control):
  def __init__(self, value):
    super(FooControl).__init__()
    btn = Button('Press Me!', parent=self)
    def onClick():
      print value
    btn.clicked.addListener(onClick)

Now this is a trivial example. But I find that many types, UI types in particular, can have most or all of these callback methods (like self._OnButtonClick) removed by turning them into inner functions. And then as you turn them into inner functions in init, you can get rid of stored state (self.value and self.button).

But as we take this to the extreme, we end up with very simple classes (and in fact I could replace FooControl with a function, it doesn’t need to be a class at all), but very long init methods (imagine doing all your sub-control creation, layout, AND all callback functionality, inside of one method!).

I’ve decided I’d rather have a long init method, usually broken up into several inner functions, rather than a larger signature on the class with layout, callbacks, and stored state. In my mind, it is easier to pull something out into a type attribute, rather than remove it, as anything on the type is liable to be used externally. And breaking up your layout into instance methods that can really only be called once (_InitButtons), from the init, adds a cognitive burden for me.

So I can justify this decision to eliminate extra attributes rationally, but what seals the deal is, I’m not unit testing any of this code anyway. So whether it is in one long method, or broken up into several methods, it isn’t getting tested.

I started out as very much in the ‘break into small methods’ camp but have wholesale moved into the ‘one giant __init__ with inner functions’ camp. I’m curious what you all prefer and why?

PyCon: Need dinner plans? Yelp can help!

Tue, 14 Feb 2012 10:20:44 +0000

Through all of the talks, tutorials, open spaces and more, some of the best parts of PyCon are the meals. Especially dinner! Each PyCon ticket gets you catered breakfast and lunch at the conference center, with lunch being a great way to refuel and sit down at a table and meet your next project contributor, business partner, or friend. Dinner, however, is a meal you’re on your own for. Not to worry, because with 1500+ people needing to head out for dinner, there’s never a shortage of groups on the way out for a burger, pasta, falafel, beer, wine, you name it. Some people even use PyCon sponsor Yelp.com to find their dinner destination.

This year Yelp has stepped up as a sponsor for food! Not only will they help you find a destination, they’ll be giving rides to a few of the area’s highly rated restaurants. On Friday and Saturday night, Yelp will be renting several buses and driving conference goers to these select eateries, giving you a chance to get out and have a good meal with good company. The details are still being ironed out, but it looks like there will be around five buses each holding approximately 30 people. Yelp plans to send along one of their developers on each bus as an ambassador, and they might bring some nice swag as well!

As for getting on the bus, sign ups will be announced on site on the first day of the conference. You like what you see? Sign up and enjoy! Be sure to sign up on site - first come, first served.

Schedule-wise we’re looking at departure around a half hour after the day’s lightning talks are over, so about 6:30-7:00. We’ll certainly announce a more accurate time during each day’s morning session. Once you’re there, you’ll have 2-2.5 hours to spend at the restaurant and in the area. After that, the bus heads back to the hotel where you can join back up with the festivities. You’ll be back just in time for the evening’s open spaces and BoF sessions!

Bon Appétit!

Shannon -jj Behrens: Python Concurrency Spreadsheet

Tue, 14 Feb 2012 10:05:54 +0000

I'm a fan of Nicholas Piël's Asynchronous Servers in Python blog post. In a similar vein, my buddy Shailen Tuli and I put together the following spreadsheet. Feel free to view it on Google Docs and make corrections.

A few notes:

It is not my goal to use this spreadsheet to pick favorites. Rather, I'm trying to use this spreadsheet to point out differences among the different approaches.

I know very little about DieselWeb, MultiTask, FriendlyFlow, Weightless, Fibra, and Cogen. All I know is that Nicholas Piël's pointed them out as generator-based libraries.

Some of these libraries support multiple approaches at the same time. For instance, Twisted lets you use a mix of callbacks, generators, and threads. Similarly, Tornado lets you use a mix of callbacks and threads. However, it's important to point out the compromises involved. For instance, if you're using Tornado, you can't have 5000 concurrent clients to a WSGI server; instead you have to switch over to the asynchronous callback API.

When I say things like "Can handle 5000 clients?", I don't mean that each of those requests is actively being served. There's only so much CPU to go around! Rather, I mean you can have 5000 clients that are each in various stages of completion, and the server as a whole is mostly waiting on IO.

You can also read my other blog posts Python: Concurrency and Python: Asynchronous Networking APIs and MySQL.

PyCon: PyCon 2012: Want some business cards for PyCon 2012?

Tue, 14 Feb 2012 09:52:33 +0000

The deals and partnerships for PyCon 2012 keep rolling in!

The folks at custom print shop MOO are quite good at the printing business, and they’re also good at spotting trends. One they recently noticed was an uptick in business cards intended for PyCon, so they contacted us. With the conference being such a great place to network, from the hallways to the dinner tables, we were able to partner up with them and get you 50 free business cards of your own choosing.

The London-based business card, postcard, and sticker printer will even package up the shipment and deliver straight to Santa Clara, where you can pick up your cards at the registration desk! Talk about easy. All they require is that you pay a small shipping and handling charge and the cards are all yours. If you prefer, they can be mailed to you instead.

If you want to take advantage of this great offer, http://moo.com/link/h86f is the PyCon special page. From there you can fill out one of their templates or upload your own design. They even have a product available called Printfinity where you can have many different designs in a pack - 50 different ones if you choose!

Your fancy new cards will come in handy at this year’s new Job Fair. Mingle around with some of PyCon’s 122 sponsors and pass out your details in style.

Maciej Fijalkowsk: PyPy and it's future challenges

Tue, 14 Feb 2012 09:43:51 +0000

Obviously I'm biased, but I think PyPy is progressing fairly well. However,
I would like to mention some areas where I think pypy is lagging ---
not living up to its promises or the design decisions simply didn't
turn out as good as we hoped for them. In a fairly arbitrary order:

Whole program type inference. This decision has been haunting
separate compilation effort for a while. It's also one of the reasons
why RPython errors are confusing and why the compilation time is so long.
This is less of a concern for users, but more of a concern for developers
and potential developers.

Memory impact. We never scientifically measured
memory impact of PyPy on examples. There are reports of outrageous pypy
memory usage, but they're usually very cryptic "my app uses 300M" and not
really reported in a way that's reproducible for us. We simply have to start
measuring memory impact on benchmarks. You can definitely help by providing
us with reproducible examples (they don't have to be small, but they have
to be open source).

The next group all are connected. The fundamental question is: What to do
in the situation where the JIT does not help? There are many causes, but,
in general, PyPy often is inferior to CPython for all of the examples.
A representative, difficult exammple is running tests. Ideally, for
perfect unit tests, each piece of code should be executed only once. There
are other examples, like short running scripts. It all can
be addressed by one or more of the following:

Slow runtime. Our runtime is slow. This is caused by a combination
of using a higher
level language than C and a relative immaturity compared to CPython. The
former is at least partly a GCC problem. We emit code that does not look
like hand-written C and GCC is doing worse job at optimizing it. A good
example is operations on longs, which are about 2x slower than CPython's,
partly because GCC is unable to effectively optimize code generated
by PyPy's translator.

Too large JIT warmup time. This is again a combination of issues.
Partly this is one of the design decisions of tracing on the metalevel,
which takes more time, but partly this is an issue with our current
implementation that can be addressed. It's also true that in some edge
cases, like running large and complex programs with lots and lots
of megamorphic call sites, we don't do a very good job tracing. Because
a good example of this case is running PyPy's own test suite, I expect
we will invest some work into this eventually.

Slow interpreter. This one is very similar to the slow runtime - it's
a combination of using RPython and the fact that we did not spend much
time optimizing it. Unlike the runtime, we might solve it by having an
unoptimizing JIT or some other medium-level solution that would work good
enough. There were some efforts invested, but, as usual, we lack enough
manpower to proceed as rapidly as we would like.

Thanks for bearing with me this far. This blog post was partly influenced
by accusations that we're doing dishonest PR that PyPy is always fast. I don't
think this is the case and I hope I clarified some of the weak spots, both here
and on the performance page.

EDIT:For what is worth I don't mention interfacing with C here and that's not because I think it's not relevant, it's simply because it did not quite fit with other stuff in this blog post. Consider the list non-exhaustive

Cheers,

fijal

Python Diary: First steps into Kivy

Tue, 14 Feb 2012 09:40:01 +0000

I just finished the Kivy tutorial on how to build a simple Pong Game. I must say, Kivy is a very interesting framework, and I plan on building some Android apps using it soon.

The framework makes heavy use of classes to describe each component of your application, and also has a simple language for rapidly building interfaces and binding them in Python. It has native support for multi-touch screens, which makes it an ideal platform to learn.

Tablets and smartphones are rather huge right now, and I don't see them going away anytime soon. Most users of a tablet or smartphone prefer an app over a website, I know I do. Kivy makes it relatively easy to create an app which is network connected, as it has native support for JSON calls, it manages the web request and decoding of the JSON object for you.

A small project I plan to implement using Kivy is a simple map viewer for the OHRRPGCE engine. I have previously created a full map editor in another language and understand the underlying file formats. I am curious how creating such an application in Kivy will fair. From what I have read in the documentations for Kivy, I will need to use a texture to map each map tileset onto. From these map tilesets, I would then blit the required tiles to a new texture for the actual map layers. I will use touch controls for panning the map around, similar to how Google maps works on a tablet. I hope to add features such as pinch to zoom and such, since Kivy does support scaling in OpenGL. I will explain the process throughout this blog for those who are interested in Kivy app development.

Another interesting project would be to create a Kivy app for this here blog, that way Android users can easily access this blog directly on their device using a nice user interface.

Yaniv Aknin: nginx+gzip module might silently corrupt data upon backend failure

Tue, 14 Feb 2012 09:07:01 +0000

There are several elements that make absolutely certain the page you’re reading in your browser right now is actually an accurate representation of the resource the HTTP server holds in the URL you requested¹. Disregarding caching for a minute, we have two elements making sure the representation you get is protected from errors. The first protecting element is, of course, TCP, making sure that if the server wrote two-hundred bytes in a particular order, either they’ll all arrive to your end, in order and without errors, or your TCP stack will realize something bad happen and give your user-agent (your browser) a chance to cope with the error. The need for the second protecting element is a bit more sneaky: TCP will guarantee everything the server wrote will arrive, i.e., bytes for which the server called write(2) or equivalent will arrive, or you’ll know something went wrong. But what about bytes the server should have written but didn’t write all – for example, because some component on the server’s side failed?

The original HTTP (HTTP 0.9, 1996 time) didn’t cope with this situation at all. The signal to the client that the server finished talking was to disconnect the TCP session, which, from the client’s side, is a vague signal. Did the TCP server disconnect because it finished or because it ran into trouble (software fault, sysadmin action, kernel behaviour due to memory pressure or even a bug, etc)? Thankfully, current HTTP kicks in to complement TCP, allowing the server to do one of several things in order to make sure you’ll at least know you didn’t receive the whole picture. By far the two most common thing the server will do are to specify a Content-Length in the response’s header or to use a Transfer-Encoding, most probably chunked transfer encoding.

Content length is simple to grasp. The server wishes to say 200 bytes. It explicitly says: “I will say 200 bytes” in the response header. If the user-agent didn’t receive 200 bytes of response, it knows something went wrong. Chunked transfer encoding is only slightly more complex – the server will send the response in chunks, each chunks prefixed by the length of the chunk. The end of the document is marked by a zero-length chunk. So if the user-agent saw a chunk cut in the middle, or didn’t receive a zero-length chunk, it also knows something went wrong and has a chance to decide what to do about it. For example, when faced with incorrect content length, Chrome displays an ERR_CONNECTION_CLOSED error, whereas Firefox would display the portion of the page it did receive. Different behaviour, yes, but at least both user-agents in this example had a chance realize the response they received is partial. Which is really, really important, you know why? I’ll tell you why.

Enter caching. HTTP caching is a non-trivial matter with many unexpected gotchas and pitfalls, and I can’t cover it all here (why the complexity? I think it’s because all caching is an intentional form of data/state repetition, and repetition is something that in my experience humans often have difficulty reasoning about). By far the best document I know about HTTP caching is this splendid guide, but if you’re in a hurry or impatient, let me summarize the points interesting for this particular post. First, caches might exist in many places, some of them might be surprising, some of them might be slightly broken or at least very aggressive (ISP transparent caches, mutter mutter cough cough). Second, among many other things, HTTP caching lets a server give a client a token together resource, telling the client “next time you request this resource, tell me you have this token; maybe I’ll just tell you that the representation you got with this token is still fresh, without transferring it all over again”. This is called an ETag, and the response that says “just use what you have in your cache” is called HTTP 304 NOT MODIFIED.

How is this relevant to HTTP responses cut in the middle? Well, if servers didn’t have a way of telling the user-agent how long is the document, and if the response was cut in the middle due to a server fault as described above, the user-agent/sneaky-caching-proxy might cache incorrect responses. If the server also sends an ETag along with the response, the caching entity will store this ETag along with the invalid cached representation, and even when it’s time to check the representation’s freshness with the origin server, the server will just take a look at the ETag, say “yep, this is fine”, tell the cache to keep using the bad representation and <sinister>never ever let it recover</sinister>. If this happens on a large ISP’s transparent cache, easily tens of thousand of your users could be affected. If the affected resource is a common element in many of your pages with strict syntax checking, like a javascript resource, you’re kinda screwed. The only hope in such a condition is that the client, for some reason, will specify Cache-Control: no-cache in the request, and that caching entities along the path to the server will honour this request. Browsers like caching, so they won’t usually request no-cache, although AFAIK, recently Chrome started sending no-cache when the user explicitly requests a force-reload (Cmd-R on a Mac). Other browsers don’t fare as well, and I think that hoping one of your Chrome users will force reload the bad resource in time to save the day is hardly a sturdy solution.

Bottom line is, it’s really important to know when a representation of a resource is broken. Which is why I was quite amazed to learn that my HTTP server of choice, nginx, doesn’t validate the Content-Length it receives from its upstreams and is simply unaware when the response it received from an upstream server is chopped off. If your response specifies a content length but closes the connection without delivering enough bytes, nginx will simply stall the request for a long time without closing the connection downstream, even though it has no hope of receiving additional data to push downstream. I tried this both with proxy_pass and uwsgi_pass, but I’m quite confident it’s true for other backends (fastcgi_pass, scgi_pass, etc). This is bad, but not as bad as the case where you want an nginx module to manipulate your content, removing existing content length/transfer encoding and applying its own (the gzip module indeed does that). If a backend error occurs while content-length-oblivious-nginx is altering the data, the content altering module will apply what it applies to the bytes it received, add new content-length/transfer-encoding, assuring everyone the response is OK, and entice user-agents or even proxies to enter the almost-never-recover bad cache scenario I described in the previews paragraph. Ouch!

The proper way to fix this, IMHO, is that nginx simply must start looking at the upstream’s content length (or transfer encoding, once nginx starts using chunked responses with its upstreams). Part of the reason I’m writing this post is that Maxim Dounin, venerable nginx comitter and an OK chap overall, told me he doesn’t consider this a top priority at the moment, but I humbly disagree with his assessment of how serious the issue is. Until such a time as nginx is fixed about this, I think you must disable all content-manipulating nginx modules and instead handle all message length affecting work in your upstream (compression, addition, etc). This is what I opted to do with my django based web app, I replaced nginx’s gzip module with Django’s GZipMiddleware. It’s a terrible shame though. It’s doing the job of nginx for it, probably in a lesser fashion than how nginx could, it violates a must not clause in Python’s WSGI PEP333, and I have empiric proof that Tim Berners-Lee chokes a kitten every time you do it.

But what’s the alternative? Risk invisibly cached corrupt data for an undetermined length of time? Ditch nginx, which I think is the best HTTP server on this planet despite this debacle? Nah. Both are unacceptable.

¹ This post assumes convenient values of “absolutely certain”; also, everything related to security/content tampering is out of scope in this post. I’m talking about possibly misbehaving but certainly well-meaning components.

Tagged: django, http, nginx, python

Catherine Devlin: Growth in PyCon sponsorship

Tue, 14 Feb 2012 08:59:12 +0000

When I teach Python, I usually include a slide or two about Python's importance and growth. At this point, that's begun to feel unnecessary - but I think I will include this illustration, because it's a neat one.

In 2004, I'd just started looking at Python. When I found out that PyCon would be just a few miles from my sister-in-law's house, I thought, "Why not?" - and I fell in love with the Python community forever. PyCon 2004 had nine sponsors.

EaIcazSWNM0/TzqSGPSnAPI/AAAAAAAAAJM/DDursOJS8PU/s1600/pycon2004sponsors.png">
EaIcazSWNM0/TzqSGPSnAPI/AAAAAAAAAJM/DDursOJS8PU/s1600/pycon2004sponsors.png">
EaIcazSWNM0/TzqSGPSnAPI/AAAAAAAAAJM/DDursOJS8PU/s1600/pycon2004sponsors.png">
This year, I barely squeaked my registration in before the blast doors slammed shut. Here are PyCon US 2012's sponsors.

S. Lott: Multiprocessing Goodness -- Part 2 -- Class Defintions

Tue, 14 Feb 2012 08:32:13 +0000

The multiprocessing module includes a generic Process class, which can be used to wrap a simple function.

The function must be designed to work with Queues or Pipelines or other synchronization techniques.

There's an advantage, however, to defining a class which gracefully handles generator functions. If we have Generator-Aware multi-processing, we can (1) write our algorithms as generators and then (2) trivially connect Processes with Queues to improve scalability.

We're looking at creating processing "pipelines" using Queues. That way we can easily handle multiple-producer and multiple-consumer (fan-in, fan-out) processing that enhances concurrency.

See Multiprocessing Goodness -- Part 1 -- Use Cases for more information.

We have three use cases: Producer, Consumer and Consumer-Producer.

Producer

A Producer gets data from somewhere and populates a queue with it. This is the source that feeds data into the pipeline.

class ProducerProcess( Process ):
    """Produces items into a Queue.
    
    The "target" must be a generator function which yields
    pickable items.
    """
    def __init__( self, group=None, target=None, name=None, args=None, kwargs=None, output_queue=None, consumers=0 ):
        super( ProducerProcess, self ).__init__( name=name )
        self.target= target
        self.args= args if args is not None else []
        self.kwargs= kwargs if kwargs is not None else {}
        self.output_queue= output_queue
        self.consumers= consumers
    def run( self ):
        target= self.target
        for item in target(*self.args, **self.kwargs):
            self.output_queue.put( item )
        for x in range(self.consumers):
            self.output_queue.put( None )
        self.output_queue.close()

This class will wrap a "target" function which must be a generator. Every value yielded is put into the "output_queue". When the source data runs out, enough sentinel tokens are put into the queue to satisfy all consumers.

Consumer

A Consumer gets data from a queue and does some final processing. Perhaps it loads a database, or writes a file. It is the sink that consumes data on the pipeline.

class ConsumerProcess( Process ):
    """Consumes items from a Queue.
    
    The "target" must be a function which expects an iterable as it's
    only argument.  Therefore, the args value is not used here.
    """
    def __init__( self, group=None, target=None, name=None, kwargs=None, input_queue=None, producers=0 ):
        super( ConsumerProcess, self ).__init__( name=name )
        self.target= target
        self.kwargs= kwargs if kwargs is not None else {}
        self.input_queue= input_queue
        self.producers= producers
    def items( self ):
        while self.producers != 0:
            for item in iter( self.input_queue.get, None ):
                yield item
            self.producers -= 1
    def run( self ):
        target= self.target
        target( self.items(), **self.kwargs )

This class will wrap a "target" function which must be ready to work with any iterable. Every value from the queue will be provided to the target function for processing. When enough sentinel tokens have been consumed from producers, it terminates processing.

Consumer-Producer

The middle of a processing pipeline is consumer-producer processes which consume from one queue and the produce to another queue.

        
class ConsumerProducerProcess( Process ):
    """Consumes items from a Queue and produces items onto a Queue.
    
    The "target" must be a generator function which yields
    pickable items and which expects an iterable as it's
    only argument.  Therefore, the args value is not used here.
    """
    def __init__( self, group=None, target=None, name=None, kwargs=None, input_queue=None, producers=0, output_queue=None, consumers=0 ):
        super( ConsumerProducerProcess, self ).__init__( name=name )
        self.target= target
        self.kwargs= kwargs if kwargs is not None else {}
        self.input_queue= input_queue
        self.producers= producers
        self.output_queue= output_queue
        self.consumers= consumers
    def items( self ):
        while self.producers != 0:
            for item in iter( self.input_queue.get, None ):
                yield item
            self.producers -= 1
    def run( self ):
        target= self.target
        for item in target(self.items(), **self.kwargs):
            self.output_queue.put( item )
        for x in range(self.consumers):
            self.output_queue.put( None )
        self.output_queue.close()

This class will wrap a "target" function which must be a generator function that consumes an iterable.
Every value from the queue is provided to the target generator. Every value yielded by the generator is sent to the output queue. The input side counts sentinels to know when to stop. The output side produces enough sentinels to alert downstream processes.

Target Functions

A producer function must be a generator function of this form

def prod( *args ):
    for item in some_function(*args):
       yield item

A consumer function looks like this:

def cons( source ):
    for item in source:
       final_disposition(item)

Finally, a consumer-producer function looks like this.

def cons_prod( source ):
    for item in source:
       next_value= transform(item)
       yield next_value

These functions can be tested and debugged like this.


for final in consumer( cons_prod( producer( *args ) ) ):
    print( final )

That way we're confident that our algorithm is correct before attempting to scale it with multiprocessing.

S. Lott: TDRE - Test Driven Reverse Engineering Case Study

Tue, 14 Feb 2012 08:00:04 +0000

Background

Read up on compass variation or declination. For example, this NOAA site provides some useful information.

Mariners use the magnetic variation to compute the difference between True north (i.e., aligned with the grid on the chart) and Magnetic north (i.e., aligned with the compass.)

The essential use case here is "What's the compass variation at a given point?" The information is printed on paper charts, but it's more useful to simply calculate it.

There are two magnetic models: the US Department of Defense World Magnetic Model (WMM) and the International Association of Geomagnetism and Aeronomy (IAGA) International Geomagnetic Reference Field (IGRF).

A packaged solution is geomag7.0. This includes both the WMM and the IGRF models. This is quite complex. However, it does have "sample output", which amount to unit test cases.

The essential spherical harmonic model is available separately as a small Fortran program, igrf11.f.

Which leads us to reverse engineering this program into Python.

TDRE Approach

The TDRE approach requires having some test cases to drive the reverse engineering process toward some kind of useful results.

The geomag7.0 package includes two "Sample Output" files that have the relevant unit test cases. The file has column headings and 16 test cases. This leads us to the following outline for the unit test application.


    class Test_Geomag( unittest.TestCase ):
        def __init__( self, row ):
            super( Test_Geomag, self ).__init__()
            self.row= row
        def runTest( self ):
            row= self.row
            if details: 
                print( "Source: {0:10s} {1} {2:7s} {3:10s} {4:10s} {5:5s} {6:5s}".format( row['Date'], row['Coord-System'], row['Altitude'], row['Latitude'], row['Longitude'], row['D_deg'], row['D_min'] ),
                file=details )
            
            date= self.parse_date( row['Date'] )
            lat= self.parse_lat_lon( row['Latitude'] )
            lon= self.parse_lat_lon( row['Longitude'] )
            alt= self.parse_altitude(row['Altitude'] )
            
            x, y, z, f = igrf11syn( date, lat*math.pi/180, lon*math.pi/180, alt, coord=row['Coord-System'] )
            D = 180.0/math.pi*math.atan2(y, x) # Declination 

            deg, min = deg2dm( D )
            
            if details: 
                print( "Result: {0:10.5f} {1} K{2:<6.1f} {3:<10.3f} {4:<10.3f} {5:5s} {6:5s}".format( date, row['Coord-System'], alt, lat, lon, str(deg)+"d", str(min)+"m" ), 
                    file=details )
                print( file=details )
            
            self.assertEqual( row['D_deg'], "{0}d".format(deg) )
            self.assertEqual( row['D_min'], "{0}m".format(min) )

    def suite():
        s= unittest.TestSuite()
        with open(sample_output,"r") as expected:
            rdr= csv.DictReader( expected, delimiter=' ', skipinitialspace=True )
            for row in rdr:
                case= Test_Geomag( row )
                s.addTest( case )
        return s

    r = unittest.TextTestRunner(sys.stdout)
    result= r.run( suite() )
    sys.exit(not result.wasSuccessful())

The Test_Geomag class does two things. First, it parses the source values to create a usable test case. We've omitted the parsers to reduce clutter. Second, it produces details to help with debugging. This is reverse engineering, and there's lots of debugging. It depends on a global variable, details, which is either set to sys.stderr or None.

This suite() function builds a suite of test cases from the input file.

The unit under test isn't obvious, but there's a call to the igrf11syn() function where the important work gets done. We can start with this.

def  igrf11syn( date, nlat, elong, alt=0.0, coord='D' ):
    return None, None, None, None

This lets us run the tests and find that we have work to do.

Reverse Engineering

The IGRF11.F fortran code contains this IGRF11SYN "subroutine" that does the work we want. The geomag 7.0 package has a function called shval3 which is essentially the same thing.

Both are implementations of the same underlying "13th order spherical harmonic series" or a "truncated series expansion".

The Fortran code contains numerous Fortran "optimizations". These are irritating hackarounds because of actual (and perceived) limitations of Fortran. They fall into two broad classes.

Hand Optimizations. All repeated expressions were manually hoisted out of their context. This is clever but makes the code particularly obscure. It doesn't help when local variables are named ONE, TWO and THREE. Bad is it is, not much needs to be done about this. Python code looks a bit like Fortran code, so very little needs to be done except add `math.` to the various function calls like sort, cos and sin.
Sparse Array Chicanery. There are actually two spherical harmonic series. The older 10-order and the new 13-order. Each model has two sets of coefficients: g and h. These form two half-matrices plus a vector. The old models have 55 g values in one matrix, 55 h values in second matrix, and a set of 10 more g values that form some kind of vector; 160 values. The new models have 91 g, 91 h and 13 g in the extra vector; 195 values. There are 23 sets of these coefficients (for 1900, 1905, ... 2015). The worst case is 23×195=4,485 values. This appears to be too much memory, so the two matrices and vectors are optimized into a single opaque collection of 3,256 numbers and delightfully complex set of index calculations.

Phase 1. Do the smallest "literal" transformation of Fortran to Python.

This means things like this:

Transforming the subroutine into a Python function with multiple return values.
Reasoning out the overall "steps". There's a bunch of setup followed by the essential series calculation followed by some final calculations.
Locating and populating the global variables.
Reformatting the if statements.
Removing the GOTO's. Either make them separate functions or properly nest the code.
Reformatting the do loop.
Handling the 1-based indexing. In almost all cases, Fortran "arrays" are best handled as Python dictionaries (not lists).

Once this is done, there are some remaining special-case discrepancies. Most of these are tacit assumptions about the problem domain that turn out to be untrue. For example, the Geodetic, Geocentric features seemed needless. However, they're not handled trivially, and need to be left in place. Also, conversion of signed values in radians to degrees and minutes isn't trivial.

This leads to passing all 16 unit tests with the single opaque collection of 3,256 numbers and delightfully complex set of index calculations.

Phase 2. Optimize so that it makes some sense in Python.

This involves unwinding the index calculations to simplify the array. The raw coefficients are available (igrf11coeffs.txt) and they have a sensible structure that separates the two matrices very cleanly. The code uses the combined matrix (called gh) in a very few places. The index calculations aren't obvious at all, but a few calls to print reveal how the matrix is accessed.

Given (1) unit tests that already work and (2) the pattern of access, it's relatively easy to hypothesize a dictionary by year that contains a pair of simple dictionaries, g[n,m] and h[n,m], for the coefficients.

Cleanup and Packaging

Once the tests pass, the package -- as a whole -- needs to be made reasonably Pythonic. In this case, it means a number of additional changes. For example, converting the API from degrees to radians, supplying appropriate default values for parameters, providing convenience functions.

Additionally, there are Python ways to populate the coefficients neatly and eliminate global variables. In this case, it seemed sensible to create a Callable class which could load the coefficients during construction.

Note that there's little point in profiling to apply further optimizations. The legacy Fortran code was already meticulously hand optimized.

Wallix: Announcing PyLogsParser 0.4

Tue, 14 Feb 2012 07:41:35 +0000

Wallix LogBox team is happy to announce version 0.4 of PyLogsParser. New normalizers Wallix AdminBastion authentication logs, written by Nassim Babaci Cisco ASA logs. Dansguardian logs. Features Adds Common Callbacks facility : a library of functions that are ready to … Continue reading →

Yuval Greenfield: Python isn’t English and iterator “labels”

Tue, 14 Feb 2012 07:41:26 +0000

Us python fanboys like to think of python as similar to English and thus more readable. Let’s examine a simple piece of code:

for item in big_list:
    if item.cost > 5:
        continue
    item.purchase()

For our discussion there are only 3 kinds of people:

People who have never seen a line of code in their life.
Have programmed in other languages but have never seen python.
Python programmers.

We’ll dabble between the first 2 groups and how they parse the above. Let’s try to forget what we know about python or programming and read that in English:

“for item in big_list” – either we’re talking about doing something for a specific item in a big_list or we’re talking about every single item. Ambiguous but the first option doesn’t really make sense so that’s fine.
“if item.cost > 5″ – non-programmers are going to talk about the period being in a strange place, but programmers will know exactly what’s up.
“continue” – That’s fine, keep going. English speakers are going to get the completely wrong idea. As programmers we’ve grown used to this convention though its meaning in English is very specifically equivalent to what pythonistas call “pass” or “nop” in assembly. We really should have called this “skip” or something.
“item.purchase()” – non-programmers are going to ask about the period and the parentheses but the rest grok that easily.

So I’m pretty sure this isn’t English. But it’s fairly readable for a programmer. I believe programmers of any of the top 8 languages on the TIOBE index can understand simple python. I definitely can’t say the same for Lisp and Haskell. Not that there’s anything wrong with Lisp/Haskell, these languages have specialized syntax for their honorable reasons.

Continue is a silly word, what about iterator labels?

Let’s say I want to break out of an outer loop from a nested loop, eg:

for item in big_list:
    for review in item.reviews:
        if review < 3.0:
            # next item or next review?
            continue
        if review > 9.0:
            # stop reading reviews or stop looking for items?
            break

Java supports specific breaks and continues by adding labels to the for loops but I think we can do better. How about this:

items_gen = (i for i in big_list)
for item in items_gen:
    for review in item.reviews:
        if review < 3.0:
            items_gen.continue()
        if review > 9.0:
            items_gen.break()

But how can that even be possible you may ask? Well, nowadays it isn’t but maybe one day if python-ideas like this idea we can have nice things. Here’s how I thought it could work: a for-loop on a generator can theoretically look like this:

while True:
    try:
        item = next(gen)
        # do stuff with item
    except StopIteration:
        break

But if it worked like I propose below we can support the specific breaks and continues:

while True:
    try:
        item = next(gen)
        # do stuff with item
    except gen.ContinueIteration:
        pass
    except gen.StopIteration:
        break
    except StopIteration:
        break

So every generator could have a method which throws its relevant exception and we could write specific breaks and continues. Or if you prefer a different spelling could be “break from mygen” or “continue from mygen” as continue and break aren’t allowed as method names normally.

I think this could be nice. Although many times I found myself using nested loops I actually preferred to break the monster into 2 functions with one loop each. That way I could use the return value to do whatever I need in the outer loop (break/continue/etc). So perhaps it’s a good thing the language doesn’t help me build monstrosity’s and forces me to flatten my code. I wonder.

EmptysquarePython: Third Normal Form and Ultimate Truth

Tue, 14 Feb 2012 06:31:57 +0000

I have an opinion: most people learned about relational databases as if RDBMSes were designed to store the ultimate truth about some data. They figured that once the schema had been properly diagrammed and normalized, then they could load all their data into it, and finally, start doing some queries.

To pick on an easy target, look at Wikipedia’s article on schema design. It summarizes the two steps a designer must take:

Determine the relationships between the different data elements.
Superimpose a logical structure upon the data on the basis of these relationships.

Do you see a step that’s missing? If you’ve deployed and maintained a large-scale application you’ll probably see what the Wikipedia authors omitted. In fact, it’s the first step: Figure out what one question your database must answer. Then, design your schema to answer that question as fast as possible. And now you’re done. Come to think of it, you never had to do steps 1 and 2 at all.

There’s a total disconnect between the approaches of introductory SQL courses and real-world application development, and I think this disconnect is slowing down adoption of NoSQL.

Consider Facebook Messages. After a (now rather well-publicized) evaluation process, Facebook chose HBase, a NoSQL data store, as the main database for their message system. I haven’t talked to anyone there, but I figure they chose it based on this criterion:

How fast will our database answer the question, “What are this user’s most recent 10 messages?”

They chose the database system that could answer that question the fastest, and they designed the best schema they could think of to answer that question. Anything else they need to ask HBase may be slow, or difficult, but that doesn’t matter, because “What are this user’s most recent 10 messages?” probably accounts for 99% of the load on their system.

If you learned about databases in college, following some textbook, I expect you were guided through a long process of modeling real-world data using rows and columns, to express some profound truth about the data. Then, you were introduced to SQL, with which you could query the data. At the end of the course, maybe there was a brief discussion of database performance. Probably not.

Data at the scale that the largest websites handle doesn’t work that way. Large applications design their schemas to answer one question as quickly as possible, and no other considerations are significant.

The next time you read about a NoSQL database you might wonder, “What about foreign keys, or normalization? What about transactions? Why can’t I define secondary indexes? Why are range queries prohibited?” (I’m just picking some limitations at random—each system is different.) Consider who built these new database systems, and what their experience has been. The ideas behind NoSQL databases mostly originated at places like Google, Amazon, and Yahoo. They build huge systems, and huge systems’ loads are usually dominated by a handful of queries. Companies build their database systems from the ground up to optimize the performance of these queries. NoSQL databases encourage you to figure out ahead of time, “What one question do I need to answer?” Figure that out, and choose your database software and your schema based on that. Nothing else really matters.

ShiningPanda: Selenium, Python and Jenkins on Debian - 3/3

Tue, 14 Feb 2012 03:00:01 +0000

In our first post on Selenium in Python, we saw how to prepare your continuous integration environment on Debian. In the second one, we saw how to enable Selenium tests in your existing web project. Now it's time to see how to integrate all this within Jenkins!

Global Configuration

First ensure that the latest version of the ShiningPanda Plugin and Xvnc Plugin are installed (check on Manage Jenkins > Manage Plugins > Installed page). If not, look for them in the Available tab and perform the installation (don't forget to restart Jenkins).

Then declare all the Python installations you want to test on with the Manage Jenkins > Configure System page (one shot configuration):

To add an installation, search the Python section and click on Add Python. Then give the installation a name and enter its home folder (ie. PYTHONHOME).

Sample project

This tutorial is based on the sample described in our last post. Our goal is to test the project on several different web browsers, or in this context: Web Driver environments.

Create a new job

First of all, create a New Job, enter its name (here djangotutorial) and select Build multi-configuration project before validating.

Basic setup

As usual, setup:

The description.
The source repository, here https://github.com/shiningpanda/djangotutorial-selenose.git.
The build trigger policy: checking for modifications every five minutes in this example.

Axis

Axis provide parameters to the build. At least two axis are required for this project:

The Web Driver one to specify the web browser to use,
The Python one to specify the Python interpreter running the web server.

Web Driver axis

To create a Web Driver axis, click on Add axis in Configuration Matrix section and select User-defined axis.

Then:

In Name define the environment variable that will contains the Web Driver environment to test on, here WEBDRIVER,
In Values write a space separated list of Web Driver environment you want to test on, here firefox and chrome.

For more informations on Web Driver environments, see selenose documentation.

Python axis

Click once more on Add axis and select Python to be able to select the Python interpreter running the web server.

In this example, only Python 2.7 is tested but feel free to add additional interpreters.

Start a display

A display is required to run Selenium tests.

To start one, enable Run Xvnc during build in the Build Environment section.

Builder

To be able to install all required package, a Virtualenv Builder is recommended.

Click on Add build step in Build section and select Virtualenv Builder.

In the Command field enter all required steps:

Install dependencies with pip,
Step in tests folder,
Run the tests using the script run.py (a nose wrapper) without forgetting to specify the Web Driver to use with the --selenium-driver= option (note that its value $WEBDRIVER comes from the axis defined previously),
Convert code coverage report in XML.

Post-build actions

Add all the desired post-build actions. Here Jenkins is asked to parse the XML test and coverage reports and to send mails on failure.

Results

Finally start a new build by clicking on Build Now: execution results by Web Driver environment are directly available on the main page of the project.

Tips

Testing on multiple Web Driver environments can be time and resource consuming.
To ensure that a revision is worthwhile to test on all environments, you can execute a touchstone build first:

If this build is successful, builds on other versions will be triggered.
If this build fails, other versions will not be tested.

To enable this option, look for a Execute a touchstone build first checkbox in the Configuration Matrix and check it.

The syntax for filter is WEBDRIVER=="<environment>", where <environment> is the name of the Web Driver environment you want to test on first.
Here the touchstone build is executed with Firefox (WEBDRIVER=="firefox") but it could also have been Chrome (WEBDRIVER=="chrome").

Montreal Python User Group: Cloud Robotics Hackathon

Tue, 14 Feb 2012 02:38:37 +0000

Hello Montreal hackers

The cloud robotics week-end is happening the 2nd, 3rd and 4th of March.

Come hack on robots and web services to create cool projects in robotics. See http://roboticshackathon.com/ for more information.

We are waiting for you if you would like to create a team Montreal-Python and if you would like to join us for this week-end of hack and programming.

Stephen Ferg: Backing up your email

Tue, 14 Feb 2012 02:17:03 +0000

Just in case someone might find this useful …

I recently had something bad happen to me. I use Thunderbird (on Windows Vista) as my email client. I asked Thunderbird to compact my email files, and it wiped out a bunch of my email messages. (I think that one of my email files must have been corrupt, and when I compacted it, the compaction process wiped out messages that should not have been wiped out.)

You can recover deleted email messages … but not after the email file has been compacted. So the messages were not recoverable. Bummer.

The upside is that this nasty incident led me to learn some things.

One thing that I learned was that the disk backup utility that I was using at the time did NOT backup my email files. The email files were stored in a directory called AppData, and the AppData directory is a “hidden” directory. So the backup utility didn’t see the AppData directory, and didn’t back it up. So I had no backup of the deleted messages.

Learning that led me to investigate ways to backup my email files, and I found this: Five ways to keep your emails backed up

For backing up Thunderbird files, it recommends MozBackup as being fast, free and easy to use. So I tried MozBackup, and those claims seem to be true.

Now I’m evaluating different disk backup options.

The take-away here is that you need to pay special attention to backing up your email files. So if you’re not backing up your email files, take a look at Five ways to keep your emails backed up (and read the comments, which are useful) or google something like “email backup”.

[Note that this applies only if you are using an email client such as Thunderbird, Outlook, Outlook Express, etc. If you don't use an email client, and do all of your email work through a Web interface to your Internet Service Provider, then this is not an issue.]

EmptysquarePython: Philly MongoDB User Group: Python, MongoDB, and Asynchronous Web Frameworks

Mon, 13 Feb 2012 20:26:39 +0000

Photo (C) Parent5446

I’ll be recapping last week’s talk on Python, MongoDB, and Asynchronous Web Frameworks this Thursday at 7pm, in Philadelphia, at the Philly MongoDB User Group’s inaugural meetup. We’ll be at the Devnuts office, at 908 North 3rd Street. We’ll have pizza, naturally.

First Philly MongoDB User Group

Cormoran Project: Continuous integration with Travis

Mon, 13 Feb 2012 18:42:44 +0000

We recently started using Travis CI as continuous integration server for Cormoran. Travis is a distributed build system for the open source community used by projects like Ruby, Rails, Rubinius, Rubygems and a large number of open source projects.

Travis is integrated with GitHub and supports various programming languages like Ruby, Node.js or PHP. Although Travis doesn't support Python language it's possible to use it with Python projects. Here's how to do it ;)

First of all you need to sign in through GitHub and follow the getting started guide here. In the third step you need to create a .travis.yml configuration file to tell Travis how to build your project. This is an example of the structure that must have.

language: python
before_script:
    - virtualenv cormoran_env
    - source cormoran_env/bin/activate
    - pip install nose pyhamcrest pydoubles coverage

script:
    - python setup.py develop
    - nosetests tests/unit

There are only three sections: language, before_script and script. The first section tells Travis the programming language of the project. We use Python, although is not supported, to pass travis-lint check.

The next two sections are a set of shell commands to build the project. The before_script section is executed before main script and script is the main script to build your project. We basically create a virtual environment and install the dependencies and then build project and run tests.

That's all! We already have our continuous integration system.

Travis CI Documentation

Jesse Noller: A letter to my love, my friend, my wife.

Mon, 13 Feb 2012 17:32:53 +0000

A letter to my love, my friend, my wife and my partner — Dusty:

I know it’s the day before Valentines — some things can’t wait just for a day.

Ten years — that’s how long we’ve been with one another. Ten years feels like a lifetime — so much has changed — our lives altered in subtle — and not so subtle ways by the gentle currents of each other. In the time I’ve known you, we have both changed for the better — we compliment and act as one another’s confidant, friend, partner and lovers.

“The most powerful symptom of love is a tenderness which becomes at times almost insupportable.” — Victor Hugo

We’ve been through our times of trial — little things like accidentally renting an apartment in a war zone (my bad!) — and much bigger things from health, to finances, to not know what we were doing or where we were going. We both know that this past year has been probably the one most filled with trials and tribulations.

We’ve sat across from one another not knowing what we were going to do, we’ve held each others hands watching our infant daughter laying in a hospital bed — I’ve held your hand at your bedside in watching your pain and not knowing what to do about it, except to sit there and watch your pain. We’ve been through a lot in ten years.

Despite the trials — we have made each other stronger. You have changed who I am in such fundamental and subtle ways, that I attribute much of who I am now, to you. You have made me happier, stronger, more empathetic — you have also given me the cherished gift of your love, your tears and support in my times of pain.

You have given me more than just your love; you gave me our first daughter Abby — who might as well be a tiny clone of myself in female form (god help us all), who despite her willfulness and strong personality makes my heart jump each time I hear her laugh, each time she runs to me and hugs me and tell me she loves me.

Abby is almost five! Five years old! All parents gush about how smart their children are — but we both know there’s something special and unique about her. There’s more to her than a pushy 4.5 year old, there’s something magical about her that we both see. I can not verbalize or put to words my thanks to you for her. She’s a gift you’ve given to me.

Then there is Addison, our bubbling eight month old. What can I say about someone who greats me with a smile and a laugh whether it’s five in the morning, or me just coming home from a hard day at work?

Addison is more than a gift; she’s a blessing — the past year shows that even in our darkest hours, sitting there in a hospital not knowing what will happen, something watches over us. Addison’s happiness and flourishing is not just due to doctors, or therapists — it’s directly tied to the amazing love and care you provide to her.

Every time I look at Addison, I see an extension of you — your smile, your happiness (and when she giggles when she rams me with her walker, your sense of humor). Addison is again, a gift and blessing you’ve given me.

You’ve given me so much; you’ve changed me so much. You’ve made me look outside of myself and think of others — you, our daughters, you’ve driven me to try to change the world and help as many people as I can. You’ve driven me to be better — a better man, a better husband, father and human.

Times change — people change. We have our hard times — we have those times when we both want to go lock ourselves in the bathroom just to get a moment of quiet. We have times when we just don’t know what will come, and times when we wish what had came had not. We have persevered over the hard times we’ve faced until now, and those hard times we face now, we face together, as one.

You are beautiful — you always have been, you are strong — you are honest and critical. I might say half-jokingly that you’re my better half some times — but you really and truly are (You are also better looking than me!).

You, and the gifts have given me — our daughters, have given me more than a reason to just keep working, just to keep moving from day to day. You’ve given me a reason to truly live, to truly push myself beyond anything I could have imagined eleven years ago. You’ve given me a place and arms to cry in, to laugh in, and to grow in. You’ve given me a view of life, of living, of loving I never dreamed of having.

I know that once again we face hard times. I thought that perhaps this year might be a little easier on us — but so far, we both know it isn’t, and there are probably harder times coming for us. I am sorry that I can not always give to you all the things you so richly deserve — I’d give you anything, I’d buy you anything if I could. I am sorry I don’t have anything I can give you today other than my words — darn those hard times!

My gift to you is this — my expression of how much I truly value you, cherish you and how grateful I am — in spite of all the hard times — the good times, the memories, our daughters and most importantly our love. I am but a broken man, but with you I am whole.

Thank you for being who you are.

Thank you for being with me.

Thank you for loving me.

Thank you for letting me love you in return.

Jesse

p.s. Churchill loves you too:

PyPy Development: A Larger Example for the Flow Graph Language

Mon, 13 Feb 2012 15:50:31 +0000

Part 4 of Comparing Partial Evaluation to Tracing

This is the fourth and final blog post in a series about comparing partial evaluation and tracing. We've come a long way: In the first post of the series I showed an interpreter for a small flow-graph language together with a partial evaluator it. In the second post I showed how a tracer for the same language works and how it relates to both execution and to partial evaluation. The third post described an optimizer for traces.

In this final post we can compare and contrast the two different approaches of tracing and partial evaluation by means of an example. The programs in the flow chart language seen so far have been rather small, so I want to give an example of a larger program: an interpreter for an extremely simple bytecode instruction set. I will look at how the partial evaluator deals with that interpreter, and what the tracer does with it. The code for that, as well as all the code of the series can be found here: http://paste.pocoo.org/show/550282/ (some small additions have been made, such as a nicer way to print traces).

A Bytecode Interpreter

Writing programs in the flow graph language is painful, but I still want to give an example that is a bit more interesting than the tiny ones that we've seen so far. The example is an interpreter for the bytecode of a very trivial register-based language. The language has four registers, one of which is an accumulator on which all the actual operations are performed.

The opcodes of the language are:

jump_if_a, jumps to a target address when the accumulator is non-zero
mov_a_r0, mov_a_r1, mov_a_r2 move the value of the accumulator to the respective register
mov_r0_a, mov_r1_a, mov_r2_a move the value of a register to the accumulator
add_r0_to_a, add_r1_to_a, add_r2_to_a add the value of the register to the accumulator
decr_a decrement the accumulator
return_a stop the program and print the accumulator

The interpreter has a main loop that reads the opcode at the current program counter, does a (lengthy) dispatch to the right bytecode via a series of if statements and then executes the right opcode. Afterwards the next opcode is treated equivalently.

Here is a part of the source code in the flow graph language. As pseudocode:

bytecode_loop:
    opcode = bytecode[pc]
    pc = pc + 1
    c = opcode == 'jump_if_a'
    if c goto op_jump_if_a else goto not_jump_if_a

# select the right bytecode via a long series of if statements
not_jump_if_a:
    c = opcode == 'mov_a_r0'
    if y goto op_mov_a_r0 else goto not_mov_a_r0
not_mov_a_r0:
    c = opcode == 'mov_a_r0'
    if y goto op_mov_a_r1 else goto not_mov_a_r1
...

# bytecode implementations
op_mov_a_r0:
    r0 = a
    goto bytecode_loop

op_jump_if_a:
    c = a == 0
    target = bytecode[pc]
    pc += 1
    if c goto bytecode_loop else goto op_jump_if_a_jump

op_jump_if_a_jump:
    pc = target
    goto bytecode_loop
...

And actually working, as Prolog facts (the full implementation can be found at the link above):

% bytecode dispatch loop
block(bytecode_loop,
      op2(opcode, readlist, var(bytecode), var(pc),
      op2(pc, add, var(pc), const(1),
      op2(c, eq, var(opcode), const(jump_if_a),
      if(c, op_jump_if_a, not_jump_if_a))))).

% select the right bytecode via a long series of if statements
block(not_jump_if_a,
      op2(c, eq, var(opcode), const(mov_a_r0),
      if(c, op_mov_a_r0, not_mov_a_r0))).
block(not_mov_a_r0,
      op2(c, eq, var(opcode), const(mov_a_r1),
      if(c, op_mov_a_r1, not_mov_a_r1))).
...

% bytecode implementations
block(op_jump_if_a,
      op2(c, eq, var(a), const(0),
      op2(target, readlist, var(bytecode), var(pc),
      op2(pc, add, var(pc), const(1),
      if(c, bytecode_loop, op_jump_if_a_jump))))).
block(op_jump_if_a_jump,
      op1(pc, same, var(target),
      promote(bytecode, bytecode_loop))).
block(op_mov_a_r0,
      op1(r0, same, var(a), jump(bytecode_loop))).
...

The bytecode_loop block is the main dispatch loop. It reads an opcode out of the bytecode list at the program counter position, then has a long series of if statements that compares the current opcode to the various existing opcodes. The full code of the interpreter can be found under the link above.

The bytecodes of the interpreter don't really permit hugely complex programs, but it can be used to write a program that computes the square of a number with the following program:

mov_a_r0     # r0 = a
mov_a_r1     # r1 = a
# 2:
mov_r0_a     # r0--
decr_a
mov_a_r0
mov_r2_a     # r2 += a
add_r1_to_a
mov_a_r2
mov_r0_a     # if r0!=0: goto 2
jump_if_a 2
mov_r2_a     # return r2
return_a

Partially Evaluating the Bytecode Interpreter

The partial evaluator from the first blog post can be easily used to partially evaluate the bytecode interpreter. The static input is the bytecode for computing the square and the initial program counter value, as given above. The dynamic input are the content of the accumulator (the number to be squared). This can be done as follows:

?- bytecode_square(B),
Env = [bytecode/B, pc/0],
do_pe(bytecode_loop, Env, Label),
REnv = [a/16, r0/0, r1/0, r2/0],
interp(jump(Label), REnv), listing(block).
256
:- dynamic block/2.

<lots of generated code>

The code that is generated by the partial evaluation process is somewhat hard to read. It contains a lot of passages like this:

...
block(op_return_a1, print_and_stop(var(a))).
block(not_decr_a1, jump(op_return_a1)).
block(not_add_r2_to_a2, jump(not_decr_a1)).
block(not_add_r1_to_a2, jump(not_add_r2_to_a2)).
block(not_add_r0_to_a3, jump(not_add_r1_to_a2)).
block(not_mov_r2_a3, jump(not_add_r0_to_a3)).
block(not_mov_r1_a5, jump(not_mov_r2_a3)).
block(not_mov_r0_a5, jump(not_mov_r1_a5)).
block(not_mov_a_r27, jump(not_mov_r0_a5)).
block(not_mov_a_r18, jump(not_mov_a_r27)).
block(not_mov_a_r09, jump(not_mov_a_r18)).
block(not_jump_if_a11, jump(not_mov_a_r09)).
block(bytecode_loop12, jump(not_jump_if_a11)).
block(op_mov_r2_a2, op1(a, same, var(r2), jump(bytecode_loop12))).
...

I.e. lots of blocks that do nothing but jump to another block, interspersed with some blocks that contain an actual operation. I cleaned the output up manually and got something like the following (this sort of cleanup is something a good partial evaluation system would do itself, after partial evaluation has occurred):

block(bytecode_loop1,
    op1(r0, same, var(a),
    op1(r1, same, var(a),
    op1(a, same, var(r0),
    op2(a, sub, var(a), const(1),
    op1(r0, same, var(a),
    op1(a, same, var(r2),
    op2(a, add, var(a), var(r1),
    op1(r2, same, var(a),
    op1(a, same, var(r0),
    op2(c, eq, var(a), const(0),
    if(c, bytecode_loop11, op_jump_if_a_jump1)))))))))))).

block(bytecode_loop11,
    op1(a, same, var(r2),
    print_and_stop(var(a))).

block(op_jump_if_a_jump1,
    op1(a, same, var(r0),
    op2(a, sub, var(a), const(1),
    op1(r0, same, var(a),
    op1(a, same, var(r2),
    op2(a, add, var(a), var(r1),
    op1(r2, same, var(a),
    op1(a, same, var(r0),
    op2(c, eq, var(a), const(0),
    if(c, bytecode_loop11, op_jump_if_a_jump1)))))))))).

What do we see here? The partial evaluator has generated a block bytecode_loop1, which corresponds to the initialization opcodes mov_a_r0 and mov_a_r1 together with one iteration of the loop. Then it either jumps to a copy of the main loop (label op_jump_if_a_jump1) or to block bytecode_loop11, which prints the result and then stops. The residual code does exactly what the bytecode did: It squares the accumulator then prints that. All the uses of the bytecode and pc variable are gone.

Why did the partial evaluator produce two copies of the main loop that look the same? The reason for that is that in the second copy, the additional static information target = 2 is known, where target is a variable in the interpreter source that stores the jump target, for very brief periods of time. This additional static information does not have any effect on the residual code, so the same code is uselessly generated twice. This is an example of overspecialization.

Tracing the Interpreter

In this section we will look at what happens if we try to trace the interpreter. The naive way of doing that yields traces that are not very useful, because they abort after one iteration. We will look at a way of avoiding this problem. The problems described in this section are at the core of the paper Tracing the meta-level: PyPy's tracing JIT compiler (that paper uses a slightly more advanced version of the bytecode interpreter as an example).

To trace the interpreter, it is useful to change the bytecode_loop block from above to always promote the bytecode and the pc variables, because without knowing them the trace produced is not really interesting. This is similar to making these variables static in the partial evaluation example above:

block(bytecode_loop,
      promote(bytecode, bytecode_loop_promote_bytecode)).
block(bytecode_loop_promote_bytecode,
      promote(pc, bytecode_loop_promote_pc)).
block(bytecode_loop_promote_pc,
      op2(opcode, readlist, var(bytecode), var(pc),
      op2(pc, add, var(pc), const(1),
      op2(c, eq, var(opcode), const(0),
      if(c, op_jump_if_a, not_jump_if_a))))).
...

The rest of the interpreter stays unchanged.

To trace the interpreter we would start naively at the bytecode_loop label, because that's the label in the interpreter that is jumped to most often (which a profiler could establish easily). The following command can be used for that (this output prints traces in a slightly more readable way than in previous blog posts):

?- bytecode_square(B),
        A = 16, Env = [bytecode/B, pc/2, a/A, r0/A, r1/A, r2/0],
        do_trace(bytecode_loop, Env).
trace
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,2,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  op2(c,eq,var(opcode),const(jump_if_a))
  guard_false(c,[],op_jump_if_a)
  op2(c,eq,var(opcode),const(mov_a_r0))
  guard_false(c,[],op_mov_a_r0)
  op2(c,eq,var(opcode),const(mov_a_r1))
  guard_false(c,[],op_mov_a_r1)
  op2(c,eq,var(opcode),const(mov_a_r2))
  guard_false(c,[],op_mov_a_r2)
  op2(c,eq,var(opcode),const(mov_r0_a))
  guard_true(c,[],not_mov_r0_a)
  op1(a,same,var(r0))
  loop

opttrace
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,2,[bytecode/[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]],bytecode_loop_promote_pc)
  op1(a,same,var(r0))
  op1(bytecode,same,const([mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]))
  op1(pc,same,const(3))
  op1(opcode,same,const(mov_r0_a))
  op1(c,same,const(1))
  loop

256
B = [mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a, mov_a_r2, mov_r0_a|...],
A = 16,
Env = [bytecode/[mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a|...], pc/2, a/16, r0/16, r1/16, r2/0]

These traces are very short. They start with promoting the bytecode and the pc, followed by the execution of the opcode mov_r0_a, which is the one at position 2 in the given bytecode. Then they increment the pc and loop back to the beginning. Looking at the optimized trace, it is clear that the trace is essentially useless. It will run only for one iteration, because in the second iteration the pc is 3, thus the guard_value at the beginning will fail.

This problem can be solved by tracing more than just one iteration of the bytecode dispatch loop, which is called meta-tracing. To get this behaviour, in this simple example it is enough to start (and thus end) tracing at a different label, op_jump_if_a_jump. This label is hit when the interpreter executes a jump_if_a bytecode and the jump is taken. In a loop on the level of the executed bytecode program there is one such jump. Thus tracing from this label, a full loop in the bytecode program is traced, containing potentially many iterations of the bytecode dispatch loop in the control flow graph language.

Doing that yields the following:

?- bytecode_square(B),
        A = 16, Env = [bytecode/B, pc/11, a/A, r0/A, r1/A, r2/0, target/2],
        do_trace(op_jump_if_a_jump, Env).
trace
  op1(pc,same,var(target))
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop)
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,2,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  op2(c,eq,var(opcode),const(jump_if_a))
  guard_false(c,[],op_jump_if_a)
  op2(c,eq,var(opcode),const(mov_a_r0))
  guard_false(c,[],op_mov_a_r0)
  op2(c,eq,var(opcode),const(mov_a_r1))
  guard_false(c,[],op_mov_a_r1)
  op2(c,eq,var(opcode),const(mov_a_r2))
  guard_false(c,[],op_mov_a_r2)
  op2(c,eq,var(opcode),const(mov_r0_a))
  guard_true(c,[],not_mov_r0_a)
  op1(a,same,var(r0))
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,3,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  ...
  lots of operations ommitted
  ...
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop_promote_bytecode)
  guard_value(pc,9,[],bytecode_loop_promote_pc)
  op2(opcode,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  op2(c,eq,var(opcode),const(jump_if_a))
  guard_true(c,[],not_jump_if_a)
  op2(c,eq,var(a),const(0))
  op2(target,readlist,var(bytecode),var(pc))
  op2(pc,add,var(pc),const(1))
  guard_false(c,[],bytecode_loop)
  loop

opttrace
  op1(pc,same,var(target))
  guard_value(bytecode,[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],[],bytecode_loop)
  guard_value(pc,2,[bytecode/[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]],bytecode_loop_promote_pc)
  op1(a,same,var(r0))
  op2(a,sub,var(a),const(1))
  op1(r0,same,var(a))
  op1(a,same,var(r2))
  op2(a,add,var(a),var(r1))
  op1(r2,same,var(a))
  op1(a,same,var(r0))
  op2(c,eq,var(a),const(0))
  guard_false(c,[bytecode/[mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a],pc/11,opcode/jump_if_a,target/2],bytecode_loop)
  op1(bytecode,same,const([mov_a_r0,mov_a_r1,mov_r0_a,decr_a,mov_a_r0,mov_r2_a,add_r1_to_a,mov_a_r2,mov_r0_a,jump_if_a,2,mov_r2_a,return_a]))
  op1(pc,same,const(11))
  op1(opcode,same,const(jump_if_a))
  op1(target,same,const(2))
  op1(c,same,const(0))
  loop

256
B = [mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a, mov_a_r2, mov_r0_a|...],
A = 16,
Env = [bytecode/[mov_a_r0, mov_a_r1, mov_r0_a, decr_a, mov_a_r0, mov_r2_a, add_r1_to_a|...], pc/11, a/16, r0/16, r1/16, r2/0, target/2] .

That looks better. The trace corresponds to the interpreter running all the bytecodes in the loop of the squaring function in the example bytecode above. The optimized code starts with two guards (checking that the bytecode is still the one for the squaring function, checking that the pc is 2) and then only does the operations that actually do the computation. No bytecode dispatching is performed, thus the interpretation overhead is fully removed, apart from the two guard_value operations at the beginning.

Many of the assignments in the trace are superfluous, e.g. all the copying back and forth between registers r1, r1, r2 and accumulator a. This could be easily solved by an even more intelligent optimization utilizing SSA form.

Conclusion About the Interpreter

Both partial evaluation and meta-tracing can be used to transform the example bytecode computing a square into a form that shows the essential computation that is going on, without the interpretation overhead. The naive partial evaluator produces lots of extra blocks that just jump around, which could be solved with a post-processing step. The tracer by itself produces uselessly short traces, but with a simple trick of starting the trace at a different point the results become a lot better.

In a real meta-tracing system, the meta-tracer would need a way for the author of the interpreter to mark which bytecode corresponds to a backward jump. It would also need better integration with the interpreter to start tracing automatically, as well as cache the traces. Additionally, it would have to deal better with guards that fail a lot, attaching new traces to the failing guards. However, all that is "just" engineering on top of the ideas presented in this series of blog posts.

High-Level Conclusion

Some concluding high-level thoughts about the similarities of tracing and partial evaluation: Tracing and partial evaluation try to tackle a similar problem, that of automatically reducing the interpreter overhead, their approaches are slightly different though.

Tracing is very close to normal evaluation, only keeping some extra information in the process. But then, the optimizer that is used in a tracer is again very similar in structure to a partial evaluator. The task of the optimizer is much simpler though, because it does not need to deal with control flow at all, just a linear list of operations.

So in a sense tracing is taking those parts of partial evaluation that work (the "just evaluate those things that you can, and leave the others") and replacing the parts that don't (controlling unfolding) by a much more pragmatic mechanism. That mechanism observes actual execution runs of the program to choose control flow paths that are typical. At the same time, the tracer's focus is on loops, because they are where most programs spend significant amounts of time.

Another point of view of tracing is that it is a form of partial evaluation that replaces the control components of a partial evaluator with an oracle (the actual execution runs) that provide the information which paths to look at.

Already in the quite trivial interpreter here the effects of this are visible. The simple partial evaluator over-specializes the loop and produces two identical versions of it, that aren't different. The tracer doesn't, and it also generates only code for the loop itself, not for the initialization opcodes.

That's it for this series. To those that made it, thanks for following along. Also thanks to Samuele and Sven, who consistently gave me good feedback on the posts before I put them here.

Matt Harrison: Amazon KDP Non-Fiction Week 2 Update

Mon, 13 Feb 2012 13:03:00 +0000

I am two weeks into my Amazon KDP non-fiction experiment. Guide to: Learning Python Decorators is currently #15 on the Python list as I write this, but was up at 5 during the week. It is aso #1 on the Python “Hot New Releases”, which surprises me because #2 is a really strong title. Sales of both of my books have been slow and steady. I heard that the average “indie” author only sells 4 books/month. I’m happy to report that I’m slightly better than that and I will be receiving a check from Amazon this month ($100+ in sales).

What else have I done this week? I have wondered about Amazon’s lack of existence in BRIC (Brazil, Russia, India and China). When I had my own store front, I was able to sell to a few individuals from some of those countries. Now I am not because Amazon has no presence there. Again this is probably not something Indie fiction writers care about (with perhaps the exception of India) — I assume fiction is somewhat language dependent. But many programmers know English, so presumably I am losing some sales. How big would they be? I am not sure, but I have received at least one request from someone who claimed they did not have Kindle reader access. Currently my non-US sales are less than 10% of US sales. I am not sure if that is a function of my marketing, or how strong the Amazon presence is in those countries (UK, DE, ES, IT, FR), or some combination of the two.

I also signed up for Amazon Author Central. Perhaps this will help with SEO, I am not sure, but I now have an author page, which also aggregates this blog and my tweetstream.

While at the Utah Python group this week, the first presenter actually mentioned my decorator book in his talk. When the meeting ended, an attendee informed me that he thought my book was pretty good (which he had purchased and was reading during the second presentation). Also he gave me some more feedback, which is always nice.

I also spent some time looking into pbook generation. My calculation is around 4% of people would prefer dead tree books (very unscientific, consists of requests divided by current sales at some point in time). KDP only requires exclusive digital rights to your sales, not physical, so this is a very real possibility. But, my current pdf generation has issues and I starting learning LaTex to solve those. I hope to have some sort of dead tree versions released soon. I’m thinking about using CreateSpace, but would be open to using other POD services. If you have experience with POD feel free to drop me some notes.

My book was submitted to the January 2012 Ebook Cover Design Awards, which was released this week. I was hoping to get some feedback on my cover, but did not. I interpret that response as an indication of the “blah”-ness of my cover. Not sufficiently interesting to warrant any praise, nor blatantly bad enough to expose the horrible design elements.

Finally, a huge advantage to KDP, is I can easily push updates and Amazon handles everything else. A disadvantage is anonymity of purchases. With sales from my site, I had emails for all purchases and could interact with buyers, but pushing updates was a bit more of a pain. Some food for thought if you are considering one or the other. I really like to think that ebooks are somehow “alive” and that I can incorporate reader feedback very quickly and actually push it out, though this is not something that “real” publishers seem to be doing yet. I’m thinking that I might stay on KDP with my more expensive/longer Treading on Python which receives a higher percentage of “rentals”, and make distribution of smaller ebooks more widely available — Kobo, my own site, and perhaps BN again.