Planet Python

Thomas Vander Stichele: ubuntuone on Fedora

Mon, 23 Nov 2009 23:19:10 +0000

I hacked some more this evening on getting UbuntuOne running on my Fedora 11 desktop.

Now, obviously, trying to get something with ‘Ubuntu’ in the name on Fedora was going to be an exercise in masochism, so I pretty much knew what I was in for.

The good thing though is that the desktopcouch and ubuntuone hackers are obviously enthusiastic at someone getting this to run on Fedora, and as I often find the right motivation is 75% of the work. If these guys are going to be receptive to my feedback, then it is worth spending my time getting this to run.

I needed to first figure out order of packages and software. ubuntuone-storage-protocol goes underneath everything. For now I settled on creating a bdist_rpm out of the setup.py, which I should repackage properly later.

On top of that goes the ubuntuone-client stuff.

Here’s a bread crumb trail of bugs I ran into with possible patches I made:

ubuntuone-storage-protocol 1.0 release does not have pem file in MANIFEST
ubuntuone poking into httplib.HTTPSConnection private variables which weren’t there before 2.6.3 (Fedora 11 has 2.6, which I assume is the release right before 2.6.1 :))
a bug where ssl authentication fails, because the ca-certs.crt of Ubuntu is a completely different path on Fedora – but that took a while to figure out because of the obtuse error message

Now I got it to the point where the client applet actually starts up without errors, and loads a UbuntuOne page into my Firefox window:

And there are no further tracebacks on my console.

Sadly, I get this puzzling notification message straight after:

I’m not sure yet how my client can be newer – I’m sure the ubuntuone guys will tell me what this means. Enough hacking for one day, time to catch some sleep for tomorrow.

UPDATE: apparently I ran into this bug, where apparently due to some bug the ubuntuone guys decided to add a capability to make sure no one would be using the old client. I understand the logic but I think that should be handled better – the message is not obvious, and I don’t think it’s easy to figure out what’s wrong.

In any case, the patch worked for me, and I just synced my first test file to the cloud ! Whee ! Not sure why syncing a 22 byte text file took roughly half a minute to sync, but it’s a start.

Logilab: pylint bug day next wednesday!

Mon, 23 Nov 2009 18:01:00 +0000

Remember that the first pylint bug day will be held on wednesday, november 25, from around 8am to 8pm in the Paris (France) time zone.

We'll be a few people at Logilab and hopefuly a lot of other guys all around the world, trying to make pylint better.

Join us on the #public conference room of conference.jabber.logilab.org, or if you prefer using an IRC client, join #public on irc.logilab.org which is a gateway to the jabber forum. And if you're in Paris, come to work with us in our office.

People willing to help but without knowledge of pylint internals are welcome, it's the perfect occasion to learn a lot about it, and to be able to hack on pylint in the future!

Greg Wilson: Reverse Engineering a Bibliography

Mon, 23 Nov 2009 17:32:39 +0000

Dear lazyweb,

I’ve been handed a file with the authors and titles of approximately 140 scientific papers, and would like to construct a proper bibliography (complete with journal titles, publication dates, DOIs, etc.). If there were just 20 entries, I’d do it by hand, but with 140, I’d like some kind of script. How would you do this? It can be as ugly as it needs to be…

Thanks in advance.

Simon Willison: django-batch-select

Mon, 23 Nov 2009 16:19:52 +0000

django-batch-select (via). A smart attempt at solving select_related for many-to-many relationships in Django. Add a custom manager to your model and call e.g. Entry.objects.all()[:10].batch_select(“tags”) to execute two queries—one pulling back the first ten entries and another using an “IN” query against the tags table to pull back all of the tags for those entries in one go.

Brian Jones: A Couple of Python Tools for Tornado or AMQP Users

Mon, 23 Nov 2009 14:20:30 +0000

Hi all,

Not lots of time to blog since starting my new gig as Senior Operations Developer at MyYearbook.com (which is pretty awesome, btw), but wanted to let anyone who’s interested know about a couple of tools I’ve released on github:

bunny is a CLI tool for doing quick tests and simple operations on an AMQP server (I use RabbitMQ for testing, but it should work with any AMQP server). It’ll do your basic create/delete of exchanges, queues, and bindings. It’s only capable of doing things that are part of AMQP, and I still need to finish the feature that tests functionality via a ’round trip’ test, where it sends a message, then gets it, then sends the body to stdout. If you find time before me to finish that, send a pull request or patch.

curtain is a mixin for use with the Tornado web server that provides HTTP Digest Authentication. I actually started with this recipe and altered/updated it in various ways. It’s still not complete according to RFC 2617 — I need to code for a few edge cases like getting a stale nonce, but what’s there does work, so if you (like me) were wondering what the performance comparison would look like between tornado doing digest auth and having a front-end proxy do it, have at it

Grig Gheorghiu: Compiling Python 2.6 with sqlite3 support

Mon, 23 Nov 2009 13:54:02 +0000

Quick note to self, hopefully useful to others too:

If you compile Python 2.6 (or 2.5) from source, and you want to enable sqlite3 support (which is included in the stdlib for 2.5 and above), then you need to pass a special USE flag to the configuration command line, like this:

./configure USE="sqlite"

(note "sqlite" and not "sqlite3")

Catherine Devlin: Viva Tortuga

Mon, 23 Nov 2009 13:07:54 +0000

Joseph Lisee, author of the upcoming Python submarine robot PyCon talk, left a comment on my last post. I think he was a little shy about me highlighting him.

I'm sorry, Joseph. You really left me no choice.

Completely Unfounded Rumors About "An Underwater Python: Tortuga the Python Powered Robot"

Roll 1d6 for each hour spent in the Atlanta Hyatt bar.

1. Joseph will announce the release of asimov.py, a pure-Python implementation of the Three Laws of Robotics.
2. Bring a swimsuit and snorkel. One lucky audience member will be picked to join Tortuga in the hotel pool, where Tortuga will take a fish from their hand.*
3. Jozeph 'az been practeeseeng 'eez reedeeculous Jacques Cousteau accent for months and weel uze eet to deeleever zee eentire talk.
4. Several minutes into the presentation, Tortuga will overpower Joseph, throw him from the stage, and announce that humankind is obsolete and has been deprecated.
5. Attendees will be asked to pour out a libation to Poseidon. Any caffeinated beverage may be used.
6. There will be a sprint to construct a tall, dapper companion to Tortuga for communication and protocol purposes.

* - No, Tortuga won't be physically present at PyCon. It's not that portable. Believe me, the program committee asked!

P. S. Blogger, don't you know what an <ol> is? You know, like an <ul> with numbers.

Brett Cannon: realStorage 2.0.1 released

Mon, 23 Nov 2009 12:17:07 +0000

Just made a quick micro release of 2.0.1 for realStorage. There was a ReferenceError under browsers lacking Gears. It didn't manifest itself as an error in my unit tests as it was in the last executed line of the file.

S. Lott: On Risk and Estimating and Agile Methods

Mon, 23 Nov 2009 12:01:22 +0000

See The Question of Risk.

Also, see Lean Projects -- Not Deficient Projects.

And Keeping the Customer Satisfied.

These are notes for a long, detailed rant on the value of Agile methods.

One specious argument against an Agile approach is the "risk management" question. In this case, however, it becomes a "how much of a contingency budget should be write into the contract." Which isn't really risk management.

Doug Hellmann: PyMOTW: sys, Part 7: Modules and Imports

Mon, 23 Nov 2009 08:43:40 +0000

Modules and Imports

Most Python programs end up as a combination of several modules with a main application importing them. Whether
you are using the features of the standard library, or organizing your own code in separate files to make it
easier to maintain, understanding and managing the dependencies for your program is an important aspect of
development. sys includes information about the modules available to your application, either as built-ins
or after being imported. It also defines hooks for overriding the standard import behavior for special cases.

Imported Modules

sys.modules is a dictionary mapping the names of imported modules to the module object holding the code.

import sys
import textwrap

names = sorted(sys.modules.keys())
name_text = ', '.join(names)

print textwrap.fill(name_text)

The contents of sys.modules change as new modules are imported.

$ python sys_modules.py
UserDict, __builtin__, __main__, _abcoll, _codecs, _sre, _warnings,
abc, codecs, copy_reg, encodings, encodings.__builtin__,
encodings.aliases, encodings.codecs, encodings.encodings,
encodings.utf_8, errno, exceptions, genericpath, linecache, new, os,
os.path, posix, posixpath, re, signal, site, sphinxcontrib,
sre_compile, sre_constants, sre_parse, stat, string, strop, sys,
textwrap, types, warnings, zipimport

Built-in Modules

The Python interpreter can be compiled with some C modules built right in, so you don’t need to distribute them
as separate shared libraries. These modules don’t appear in the list of imported modules managed in
sys.modules because they weren’t technically imported. The only way to find the available built-in modules is
through sys.builtin_module_names.

import sys

for name in sys.builtin_module_names:
    print name

Note

Your results may vary, especially if you have built a custom version of the interpreter.
This script was run using a copy of the interpreter installed from the standard python.org
installer for the platform.

$ python sys_builtins.py
__builtin__
__main__
_ast
_codecs
_sre
_symtable
_warnings
errno
exceptions
gc
imp
marshal
posix
pwd
signal
sys
thread
xxsubtype
zipimport

Import Path

The search path for modules is managed as a Python list saved in sys.path. The default contents of the path
include the directory of the script used to start the application and the current working directory.

import sys

for d in sys.path:
    print d

As you can see here, the first directory in the search path is the home for the sample script itself. That is
followed by a series of platform-specific paths where compiled extension modules (written in C) might be
installed, and then the global site-packages directory is listed last.

$ python sys_path_show.py
/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

The import search path list can be modified before starting the interpreter by setting the shell variable
PYTHONPATH to a colon-separated list of directories.

$ PYTHONPATH=/my/private/site-packages:/my/shared/site-packages python sys_path_show.py
/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
/my/private/site-packages
/my/shared/site-packages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

A program can also modify its path by adding elements to sys.path directly.

import sys
import os

base_dir = os.path.dirname(__file__) or '.'
print 'Base directory:', base_dir

# Insert the package_dir_a directory at the front of the path.
package_dir_a = os.path.join(base_dir, 'package_dir_a')
sys.path.insert(0, package_dir_a)

# Import the example module
import example
print 'Imported example from:', example.__file__
print '\t', example.DATA

# Make package_dir_b the first directory in the search path
package_dir_b = os.path.join(base_dir, 'package_dir_b')
sys.path.insert(0, package_dir_b)

# Reload the module to get the other version
reload(example)
print 'Reloaded example from:', example.__file__
print '\t', example.DATA

$ python sys_path_modify.py
Base directory: .
Imported example from: ./package_dir_a/example.pyc
        This is example A
Reloaded example from: ./package_dir_b/example.pyc
        This is example B

Custom Importers

Modifying the search path lets you control how standard Python modules are found, but what if you need to import
code from somewhere other than the usual .py or .pyc files on the filesystem? PEP 302 solves this
problem by introducing the idea of import hooks that let you trap an attempt to find a module on the search
path and take alternative measures to load the code from somewhere else or apply pre-processing to it.

Finders

Custom importers are implemented in two separate phases. The finder is responsible for locating a module and
providing a loader to manage the actual import. Adding a custom module finder is as simple as appending a
factory to the sys.path_hooks list. On import, each part of the path is given to a finder until one claims
support (by not raising ImportError). That finder is then responsible for searching data storage represented by
its path entry for named modules.

import sys

class NoisyImportFinder(object):
    
    PATH_TRIGGER = 'NoisyImportFinder_PATH_TRIGGER'
    
    def __init__(self, path_entry):
        print 'Checking NoisyImportFinder support for %s' % path_entry
        if path_entry != self.PATH_TRIGGER:
            print 'NoisyImportFinder does not work for %s' % path_entry
            raise ImportError()
        return
    
    def find_module(self, fullname, path=None):
        print 'NoisyImportFinder looking for "%s"' % fullname
        return None

sys.path_hooks.append(NoisyImportFinder)

sys.path.insert(0, NoisyImportFinder.PATH_TRIGGER)

try:
    import target_module
except Exception, e:
    print 'Import failed:', e

This example illustrates how the finders are instantiated and queried. The NoisyImportFinder raises ImportError
when instantiated with a path entry that does not match its special trigger value, which is obviously not a real
path on the filesystem. This test prevents the NoisyImportFinder from breaking imports of real modules.

$ python sys_path_hooks_noisy.py
Checking NoisyImportFinder support for NoisyImportFinder_PATH_TRIGGER
NoisyImportFinder looking for "target_module"
Checking NoisyImportFinder support for /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
NoisyImportFinder does not work for /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
Import failed: No module named target_module

Importing from a Shelve

When the finder locates a module, it is responsible for returning a loader capable of importing that module.
This example illustrates a custom importer that saves its module contents in a database created by shelve.

The first step is to create a script to populate the shelf with a package containing a sub-module and
sub-package.

import sys
import shelve
import os

filename = '/tmp/pymotw_import_example.shelve'
if os.path.exists(filename):
    os.unlink(filename)
db = shelve.open(filename)
try:
    db['data:README'] = """
==============
package README
==============

This is the README for ``package``.
"""
    db['package.__init__'] = """
print 'package imported'
message = 'This message is in package.__init__'
"""
    db['package.module1'] = """
print 'package.module1 imported'
message = 'This message is in package.module1'
"""
    db['package.subpackage.__init__'] = """
print 'package.subpackage imported'
message = 'This message is in package.subpackage.__init__'
"""
    db['package.subpackage.module2'] = """
print 'package.subpackage.module2 imported'
message = 'This message is in package.subpackage.module2'
"""
    print 'Created %s with:' % filename
    for key in sorted(db.keys()):
        print '\t', key
finally:
    db.close()

A real packaging script would read the contents from the filesystem, but using hard-coded values is sufficient
for a simple example like this.

$ python sys_shelve_importer_create.py
Created /tmp/pymotw_import_example.shelve with:
        data:README
        package.__init__
        package.module1
        package.subpackage.__init__
        package.subpackage.module2

Next, we need to provide finder and loader classes that know how to look in a shelf for the source of a module or
package:

import contextlib
import imp
import os
import shelve
import sys


@contextlib.contextmanager
def shelve_context(filename, flag='r'):
    """Context manager to make shelves work with 'with' statement."""
    db = shelve.open(filename, flag)
    try:
        yield db
    finally:
        db.close()

    
def _mk_init_name(fullname):
    """Return the name of the __init__ module for a given package name."""
    if fullname.endswith('.__init__'):
        return fullname
    return fullname + '.__init__'


def _get_key_name(fullname, db):
    """Look in an open shelf for fullname or fullname.__init__, return the name found."""
    if fullname in db:
        return fullname
    init_name = _mk_init_name(fullname)
    if init_name in db:
        return init_name
    return None
    

class ShelveFinder(object):
    """Find modules collected in a shelve archive."""
    
    def __init__(self, path_entry):
        if not os.path.isfile(path_entry):
            raise ImportError
        try:
            # Test the path_entry to see if it is a valid shelf
            with shelve_context(path_entry):
                pass
        except Exception, e:
            raise ImportError(str(e))
        else:
            print 'new shelf added to import path:', path_entry
            self.path_entry = path_entry
        return
        
    def __str__(self):
        return '<%s for "%s">' % (self.__class__.__name__, self.path_entry)
        
    def find_module(self, fullname, path=None):
        path = path or self.path_entry
        print 'looking for "%s" in %s ...' % (fullname, path),
        with shelve_context(path) as db:
            key_name = _get_key_name(fullname, db)
            if key_name:
                print 'found it as %s' % key_name
                return ShelveLoader(path)
        print 'not found'
        return None


class ShelveLoader(object):
    """Load source for modules from shelve databases."""
    
    def __init__(self, path_entry):
        self.path_entry = path_entry
        return
        
    def _get_filename(self, fullname):
        return '<%s "%s"[%s]>' % (self.__class__.__name__, self.path_entry, fullname)
        
    def get_source(self, fullname):
        print 'loading source for "%s" from shelf' % fullname
        try:
            with shelve_context(self.path_entry) as db:
                key_name = _get_key_name(fullname, db)
                if key_name:
                    return db[key_name]
                raise ImportError('could not find source for %s' % fullname)
        except Exception, e:
            print 'could not load source:', e
            raise ImportError(str(e))
            
    def get_code(self, fullname):
        source = self.get_source(fullname)
        print 'compiling code for "%s"' % fullname
        return compile(source, self._get_filename(fullname), 'exec', dont_inherit=True)
    
    def get_data(self, path):
        print 'looking for data for "%s"' % path
        if not path.startswith(self.path_entry):
            raise IOError
        path = path[len(self.path_entry)+1:]
        key_name = 'data:' + path
        try:
            with shelve_context(self.path_entry) as db:
                return db[key_name]
        except Exception, e:
            # Convert all errors to IOError
            raise IOError
        
    def is_package(self, fullname):
        init_name = _mk_init_name(fullname)
        with shelve_context(self.path_entry) as db:
            return init_name in db

    def load_module(self, fullname):
        source = self.get_source(fullname)

        if fullname in sys.modules:
            print 'reusing existing module from previous import of "%s"' % fullname
            mod = sys.modules[fullname]
        else:
            print 'creating a new module object for "%s"' % fullname
            mod = sys.modules.setdefault(fullname, imp.new_module(fullname))

        # Set a few properties required by PEP 302
        mod.__file__ = self._get_filename(fullname)
        mod.__name__ = fullname
        mod.__path__ = self.path_entry
        mod.__loader__ = self
        mod.__package__ = '.'.join(fullname.split('.')[:-1])
        
        if self.is_package(fullname):
            print 'adding path for package'
            # Set __path__ for packages
            # so we can find the sub-modules.
            mod.__path__ = [ self.path_entry ]
        else:
            print 'imported as regular module'
        
        print 'execing source...'
        exec source in mod.__dict__
        print 'done'
        return mod
        

Now we can use ShelveFinder and ShelveLoader to import code from a shelf. For example, importing the
package created above:

import sys
import sys_shelve_importer

def show_module_details(module):
    print '  message:', module.message
    print '  __name__:', module.__name__
    print '  __package__:', module.__package__
    print '  __file__:', module.__file__
    print '  __path__:', module.__path__
    print '  __loader__:', module.__loader__

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

print 'Import of "package":'
import package

print
print 'Examine package details:'
show_module_details(package)

print
print 'Global settings:'
print 'sys.modules entry:', sys.modules['package']

The shelf is added to the import path the first time an import occurs after the path is modified. The finder
recognizes the shelf and returns a loader, which is used for all imports from that shelf. The initial
package-level import creates a new module object and then execs the source loaded from the shelf, using the new
module as the namespace so that names defined in the source are preserved as module-level attributes.

$ python sys_shelve_importer_package.py
Import of "package":
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found it as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done

Examine package details:
  message: This message is in package.__init__
  __name__: package
  __package__:
  __file__: <ShelveLoader "/tmp/pymotw_import_example.shelve"[package]>
  __path__: ['/tmp/pymotw_import_example.shelve']
  __loader__: <sys_shelve_importer.ShelveLoader object at 0xddf50>

Global settings:
sys.modules entry: <module 'package' from '<ShelveLoader "/tmp/pymotw_import_example.shelve"[package]>'>

Packages

The loading of other modules and sub-packages proceeds in the same way.

import sys
import sys_shelve_importer

def show_module_details(module):
    print '  message:', module.message
    print '  __name__:', module.__name__
    print '  __package__:', module.__package__
    print '  __file__:', module.__file__
    print '  __path__:', module.__path__
    print '  __loader__:', module.__loader__

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

print
print 'Import of "package.module1":'
import package.module1

print
print 'Examine package.module1 details:'
show_module_details(package.module1)

print
print 'Import of "package.subpackage.module2":'
import package.subpackage.module2

print
print 'Examine package.subpackage.module2 details:'
show_module_details(package.subpackage.module2)

$ python sys_shelve_importer_module.py

Import of "package.module1":
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found it as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done
looking for "package.module1" in /tmp/pymotw_import_example.shelve ... found it as package.module1
loading source for "package.module1" from shelf
creating a new module object for "package.module1"
imported as regular module
execing source...
package.module1 imported
done

Examine package.module1 details:
  message: This message is in package.module1
  __name__: package.module1
  __package__: package
  __file__: <ShelveLoader "/tmp/pymotw_import_example.shelve"[package.module1]>
  __path__: /tmp/pymotw_import_example.shelve
  __loader__: <sys_shelve_importer.ShelveLoader object at 0xe31b0>

Import of "package.subpackage.module2":
looking for "package.subpackage" in /tmp/pymotw_import_example.shelve ... found it as package.subpackage.__init__
loading source for "package.subpackage" from shelf
creating a new module object for "package.subpackage"
adding path for package
execing source...
package.subpackage imported
done
looking for "package.subpackage.module2" in /tmp/pymotw_import_example.shelve ... found it as package.subpackage.module2
loading source for "package.subpackage.module2" from shelf
creating a new module object for "package.subpackage.module2"
imported as regular module
execing source...
package.subpackage.module2 imported
done

Examine package.subpackage.module2 details:
  message: This message is in package.subpackage.module2
  __name__: package.subpackage.module2
  __package__: package.subpackage
  __file__: <ShelveLoader "/tmp/pymotw_import_example.shelve"[package.subpackage.module2]>
  __path__: /tmp/pymotw_import_example.shelve
  __loader__: <sys_shelve_importer.ShelveLoader object at 0xe32d0>

Reloading

Reloading a module is handled slightly differently. Instead of creating a new module object, the existing module
is re-used.

import sys
import sys_shelve_importer

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

print 'First import of "package":'
import package

print
print 'Reloading "package":'
reload(package)

By re-using the same object, existing references to the module are preserved even if class or function
definitions are modified by the reload.

$ python sys_shelve_importer_reload.py
First import of "package":
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found it as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done

Reloading "package":
looking for "package" in /tmp/pymotw_import_example.shelve ... found it as package.__init__
loading source for "package" from shelf
reusing existing module from previous import of "package"
adding path for package
execing source...
package imported
done

Import Errors

When a module cannot be imported, ImportError is raised.

import sys
import sys_shelve_importer

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

try:
    import package.module3
except ImportError, e:
    print 'Failed to import:', e

$ python sys_shelve_importer_missing.py
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found it as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done
looking for "package.module3" in /tmp/pymotw_import_example.shelve ... not found
Failed to import: No module named module3

Package Data

In addition to defining the API loading executable Python code, PEP 302 defines an optional API for retrieving
package data intended for distributing data files, documentation, and other non-code resources used by a package. By implementing get_data(), a loader can allow calling applications to support retrieval of data associated with the package without considering how the package is actually installed (especially without assuming that the package is stored as files on a filesystem).

import sys
import sys_shelve_importer
import os

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

import package

readme_path = os.path.join(package.__path__[0], 'README')

readme = package.__loader__.get_data(readme_path)
print readme

foo_path = os.path.join(package.__path__[0], 'foo')
foo = package.__loader__.get_data(foo_path)
print foo

get_data() takes a path based on the module or package that owns the data, and returns the contents of the
resource “file” as a string, or raises IOError if the resource does not exist.

$ python sys_shelve_importer_get_data.py
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found it as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done
looking for data for "/tmp/pymotw_import_example.shelve/README"

==============
package README
==============

This is the README for ``package``.

looking for data for "/tmp/pymotw_import_example.shelve/foo"
Traceback (most recent call last):
  File "sys_shelve_importer_get_data.py", line 26, in <module>
    foo = package.__loader__.get_data(foo_path)
  File "/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys/sys_shelve_importer.py", line 114, in get_data
    raise IOError
IOError

Importer Cache

Searching through all of the hooks each time a module is imported can become expensive. To save time,
sys.path_importer_cache is maintained as a mapping between a path entry and the loader that can use the
value to find modules.

import sys
import pprint

print 'PATH:',
pprint.pprint(sys.path)
print
print 'IMPORTERS:'
pprint.pprint(sys.path_importer_cache)

A cache value of None means to use the default filesystem loader. Each missing directory is associated with
an imp.NullImporter instance, since modules cannot be imported from directories that do not exist. In the
example output below, several zipimport.zipimporter instances are used to manage EGG files found on the path.

$ python sys_path_importer_cache.py
PATH:['/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/pip-0.3.1-py2.6.egg',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python26.zip',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/plat-darwin',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/plat-mac',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/plat-mac/lib-scriptpackages',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/lib-tk',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/lib-old',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages',
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages',
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages']

IMPORTERS:
{'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6': None,
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk': None,
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin': None,
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac': None,
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages': None,
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages': None,
 '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg': <zipimporter object "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg">,
 '/Users/dhellmann/.virtualenvs/pymotw/bin/../lib/python2.6/': None,
 '/Users/dhellmann/.virtualenvs/pymotw/bin/../lib/python26.zip': <imp.NullImporter object at 0x16028>,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6': None,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/encodings': None,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/lib-dynload': None,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/lib-old': <imp.NullImporter object at 0x164a8>,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/lib-tk': <imp.NullImporter object at 0x164a0>,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/plat-darwin': <imp.NullImporter object at 0x16488>,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/plat-mac': <imp.NullImporter object at 0x16490>,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/plat-mac/lib-scriptpackages': <imp.NullImporter object at 0x16498>,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages': None,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg': None,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/pip-0.3.1-py2.6.egg': None,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg': <zipimporter object "/Users/dhellmann/.virtualenvs/pymotw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg">,
 '/Users/dhellmann/.virtualenvs/pymotw/lib/python26.zip': <imp.NullImporter object at 0x16480>,
 '/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys': None,
 'sys_path_importer_cache.py': <imp.NullImporter object at 0x164b0>}

Meta Path

The sys.meta_path further extends the sources of potential imports by allowing a finder to be searched
before the regular sys.path is scanned. The API for a finder on the meta-path is the same as for a regular
path. The difference is that the meta-finder is not limited to a single entry in sys.path, it can search
anywhere at all.

import sys
import sys_shelve_importer
import imp


class NoisyMetaImportFinder(object):
    
    def __init__(self, prefix):
        print 'Creating NoisyMetaImportFinder for %s' % prefix
        self.prefix = prefix
        return
    
    def find_module(self, fullname, path=None):
        print 'NoisyMetaImportFinder looking for "%s" with path "%s"' % (fullname, path)
        name_parts = fullname.split('.')
        if name_parts and name_parts[0] == self.prefix:
            print ' ... found prefix, returning loader'
            return NoisyMetaImportLoader(path)
        else:
            print ' ... not the right prefix, cannot load'
        return None


class NoisyMetaImportLoader(object):
    
    def __init__(self, path_entry):
        self.path_entry = path_entry
        return
        
    def load_module(self, fullname):
        print 'loading %s' % fullname
        if fullname in sys.modules:
            mod = sys.modules[fullname]
        else:
            mod = sys.modules.setdefault(fullname, imp.new_module(fullname))

        # Set a few properties required by PEP 302
        mod.__file__ = fullname
        mod.__name__ = fullname
        # always looks like a package
        mod.__path__ = [ 'path-entry-goes-here' ]
        mod.__loader__ = self
        mod.__package__ = '.'.join(fullname.split('.')[:-1])
        
        return mod


# Install the meta-path finder
sys.meta_path.append(NoisyMetaImportFinder('foo'))

# Import some modules that are "found" by the meta-path finder
print
import foo

print
import foo.bar

# Import a module that is not found
print
try:
    import bar
except ImportError, e:
    pass

Each finder on the meta-path is interrogated before sys.path is searched, so there is always an opportunity to have a central importer load modules without explicitly modifying sys.path. Once the module is “found”, the loader API works in the same way as for regular loaders (although this example is truncated for simplicity).

$ python sys_meta_path.py
Creating NoisyMetaImportFinder for foo

NoisyMetaImportFinder looking for "foo" with path "None"
 ... found prefix, returning loader
loading foo

NoisyMetaImportFinder looking for "foo.bar" with path "['path-entry-goes-here']"
 ... found prefix, returning loader
loading foo.bar

NoisyMetaImportFinder looking for "bar" with path "None"
 ... not the right prefix, cannot load

Steve Holden: Links for 2009-11-22 [del.icio.us]

Mon, 23 Nov 2009 08:00:00 +0000

Mandleson, ethics, culture, commerce and copyright law
Huh. The UK will soon be even more draconian than the US in copyright.

Eli Bendersky: Visualizing binary trees with Graphviz

Mon, 23 Nov 2009 04:08:32 +0000

When implementing binary trees of some kind, one of the first utilities one writes is a visualization function that given a tree prints it to the screen.

The basic printing of a binary tree is almost always a variation of:

That is, 6 is the left (first) child of 15, 18 is its right child. 6 has ho children, 18 has only a left child (- are NULL nodes). Given a typical tree node declaration:

typedef struct bst_node_t bst_node;

struct bst_node_t
{
    int key;
    bst_node* left;
    bst_node* right;
};

The printing code is easy to write:

/* Auxiliary for bst_print_ascii
*/
void print_offset(FILE* stream, int offset)
{
    int i;
    for (i = 0; i < offset; ++i)
    {
        fprintf(stream, " ");
    }
}

/* Prints the BST horizontally as ASCII
*/
void bst_print_ascii(bst_node* tree, FILE* stream)
{
    static int offset = 0;

    print_offset(stream, offset);

    if (tree == NULL)
    {
        fprintf(stream, "-\n");
        return;
    }
    fprintf(stream, "%d\n", tree->key);

    offset += 3;
    bst_print_ascii(tree->left, stream);
    bst_print_ascii(tree->right, stream);
    offset -= 3;
}

The problem with this representation is that it isn’t particularly helpful, because (especially for larger trees) it’s quite difficult to understand. Printing trees properly in ASCII, level by level is a much more difficult job.

But there’s a better way!

Graphviz - Graph Visualization Software - is a language (called DOT) and a set of tools for automatically generating visualizations of graphs. Graphviz is used heavily in academy to supply publication-quality visualizations for papers. It’s also used by the Doxygen documentation tool for generating class hierarchies.

The power of Graphviz is in its powerful layout algorithms. You provide a textual description of the graph - which edges are there, what is connected to what, and so on, and Graphviz automagically lays out the graph in a visually pleasant way. The DOT language is a great example of a "mini-language" or an external DSL, and is very easy to use.

It isn’t very difficult to craft the C code that auto-generates the DOT file for a given binary tree:

void bst_print_dot_null(int key, int nullcount, FILE* stream)
{
    fprintf(stream, "    null%d [shape=point];\n", nullcount);
    fprintf(stream, "    %d -> null%d;\n", key, nullcount);
}

void bst_print_dot_aux(bst_node* node, FILE* stream)
{
    static int nullcount = 0;

    if (node->left)
    {
        fprintf(stream, "    %d -> %d;\n", node->key, node->left->key);
        bst_print_dot_aux(node->left, stream);
    }
    else
        bst_print_dot_null(node->key, nullcount++, stream);

    if (node->right)
    {
        fprintf(stream, "    %d -> %d;\n", node->key, node->right->key);
        bst_print_dot_aux(node->right, stream);
    }
    else
        bst_print_dot_null(node->key, nullcount++, stream);
}

void bst_print_dot(bst_node* tree, FILE* stream)
{
    fprintf(stream, "digraph BST {\n");
    fprintf(stream, "    node [fontname=\"Arial\"];\n");

    if (!tree)
        fprintf(stream, "\n");
    else if (!tree->right && !tree->left)
        fprintf(stream, "    %d;\n", tree->key);
    else
        bst_print_dot_aux(tree, stream);

    fprintf(stream, "}\n");
}

For the tree shown in ASCII in the beginning of this post, the generated DOT file is:

digraph BST {
    node [fontname="Arial"];
    15 -> 6;
    null0 [shape=point];
    6 -> null0;
    null1 [shape=point];
    6 -> null1;
    15 -> 18;
    18 -> 17;
    null2 [shape=point];
    17 -> null2;
    null3 [shape=point];
    17 -> null3;
    null4 [shape=point];
    18 -> null4;
}

And here is the result (running the dot tool with PNG output):

Much nicer, isn’t?

Graphviz is a tool for drawing graphs, not trees, so there’s some tiny tweaking needed for trees. Particularly, to differentiate left from right pointers, I always draw both. The NULL children are drawn as empty dots. There are alternative ideas for drawing trees with Graphviz, but this one is IMHO both easy to implement and looks most familiar.

Abstract vs. Concrete Syntax Trees CSTs - Concrete Syntax Trees (a.k.a. Parse Trees) and ASTs...

Alex Gaynor: Using PLY for Parsing Without Using it for Lexing

Mon, 23 Nov 2009 00:57:12 +0000

Over the past week or so I've been struggling with attempting to write my own parser (or parser generator) by hand. A few days ago I finally decided to give up on this notion (after all the parser isn't my end goal) as it was draining my time from the interesting work to be done. However, I wanted to keep my existing lexer. I wrote the lexer by hand in the method I described in a previous post, it's fast, easy to read, and I rather like my handiwork, so I wanted to keep it if possible. I've used PLY before (as I described last year) so I set out to see if it would be possible to use it for parsing without using it for lexing.

As it turns out PLY expects only a very minimal interface from it's lexer. In fact it only needs one method, token(), which returns a new token (or None at the end). Tokens are expected to have just 4 attributes. Having this knowledge I now set out to write a pair of compatibility classes for my existing lexer and token classes, I wanted to do this without altering the lexer/token API so that if and when I finally write my own parser I don't have to remove legacy compatibility stuff. My compatibility classes are very small, just this:


    class PLYCompatLexer(object):
        def __init__(self, text):
            self.text = text
            self.token_stream = Lexer(text).parse()
        
        def token(self):
            try:
                return PLYCompatToken(self.token_stream.next())
            except StopIteration:
                return None


    class PLYCompatToken(object):
        def __init__(self, token):
            self.type = token.name
            self.value = token.value
            self.lineno = None
            self.lexpos = None
        
        def __repr__(self):
            return "<Token: %r %r>" % (self.type, self.value)

This is the entirety of the API that PLY needs. Now I can write my parser exactly as I would normally with PLY.

IronPython-URLs: Hello GTK# from IronPython and F#

Sun, 22 Nov 2009 22:31:44 +0000

GTK# is a managed interface to the GTK user interface library. GTK# is part of the Mono project, but it can also be used from .NET on Windows.

In this blog entry, another one from Steve Gilham, he illustrates using GTK# from both IronPython and C# by porting some C# examples.

“Hello GTK#” from the latest IronPython and F#

A little post to record a short bit of spiking with GTK# as UI toolkit, porting the simple C# examples from here to my preferred .net languages. Neither of these explorations are totally novel -- there are examples in either language to Google, but not all of them recorded all the details of the slight rough spots that needed a little working (and they were often not at all recent, either).

For IronPython 2.6 latest with GTK# 2.12.9-2, running the programs as ipy Program.py.

The dynamic nature of the language means we can lose a lot of the declaration clutter. We just have to explicitly reference the main GTK# assembly (which is GAC'd by the GTK# installer), and away we go.

Steve Holden: Starting 2010 With a Bang

Sun, 22 Nov 2009 22:21:45 +0000

Holden Web's first one-day workshop was, thanks to Jacob Kaplan Moss, a sell-out success. As a result, and partially due to some excellent feedback from the New York City Python Meetup group, we will be running the same workshop in New York on January 22, again with Jacob presenting. We are also offering a one-day IronPython workshop presented by Michael Foord on January 21.

Since the three-day Introduction to Python classes have been well-received in Virginia we are also offering that class in New York on January 18-20.

To try and make things easier for those attending and smooth out our administration we are using Eventbrite for the first time. I would really like to know how easy people find it to get information about our classes and to enroll for them. Anyone wanting specific information not mentioned in the course outlines is, of course, welcome to contact us for further details.

If you would like to take one of these classes simply follow the links above (or click here for a list of all our current offerings, then just go to the ones you are interested in) and click the Order Now button which should be clearly visible. Once you have entered the details click the Review Your Order button, and you have fifteen minutes to check that you have entered the correct information before you click the Pay Now button. It really couldn't be much easier, I hope.

We are also very interested to know what other event you would like us to run. This is the front end of a new venture for Holden Web, and your opinions and requirements (places you'd like to attend presentations as well as other topics) will help us to move in the right direction. So feel free to contact us with your suggestions, or make them in comments below. Thanks in advance for the feedback.

Thomas Vander Stichele: desktopcouch on Fedora

Sun, 22 Nov 2009 21:24:05 +0000

Time to unravel some tufts of the yak hair lying around from this weekend.

I wanted to work on making my GTD application (using CouchDB) a little more friendly.

I started out by learning GreaseMonkey so I could modify Trac pages to include a link converting the ticket into a thing in my application.

The basics of that were easy, but I need to learn a bunch more Javascript before it will actually work well. So, I switched to finally figuring out if desktopcouch might be a good solution for the general problem for users to set up a couchdb database.

desktopcouch’s tagline describes it nicely ‘A CouchDB for every desktop.’ From what I can tell, it has:

a way to automatically start a per-user couchdb instance
oauth enabled by default for it, so only you can get to it
some gnomekeyring integration
a pairing tool that makes it easy to pair with other desktopcouch instances on your network
a python module on top of python-couchdb that layers a ‘record’ concept over CouchDB documents using record_type and record_revision keys
some pre-defined record types for notes, contacts, and bookmarks
possible integration with the UbuntuOne service

At this point I don’t know how intertwined this record concept is with the rest of desktopcouch. As at the moment I want to use desktopcouch simply as a reliable way to get a secure, per-user, replicated couchdb database, I’d prefer to not deal with records at all at this point.

In the process of getting acquainted with desktopcouch and packaging it for Fedora, I ran into a few little niggles:

First off, bdist_rpm didn’t work for desktopcouch because of the man pages being compressed.
My first run failed. Furthermore, the unhandled exception left my $HOME/.local/share/desktopcouch in a half-done state; I ran it again, and it worked, but it didn’t generate a couchdb.html as it should have because of this.
Some rpmlint warnings

As for the package, if you’re on Fedora 11, just install tao-release for F-11 i386 then type ‘yum -y install desktopcouch’

After that, you can start it by asking it for what port is running:

dbus-send --session --dest=org.desktopcouch.CouchDB --print-reply --type=method_call / org.desktopcouch.CouchDB.getPort

If all goes well that should come back with a number.

You can now access your desktopcouch through Futon with:
firefox $HOME/.local/share/desktop-couch/couchdb.html

Let me know if this worked for you or not !

I have a separate post about packaging ubuntuone stuff, which was much more complicated – and I’m still not sure if I will be able to use it as an online server for my couchdb app.

Now, for some questions to you desktopcouch-savvy people out there:

Looking at the generated couchdb.html, I see that I can now access the user-specific couchdb with a generated username and password. If I want to push my GTD app to this couchdb, I can do so using this username and password. So, obvious questions: a) should I push with couchapp this way to get my app in desktopcouch ? b) is there a better way to get this username/password ? I assume desktopcouch does this in the python modules using gnome-keyring c) is it an intended mode of operation to be able to push my app into the user couchdb ? I am working on the web interface part served by couchdb and I want it to be one of the ways in which to access the data.

Is it possible to pair with a different user on the same machine ? I was on a laptop with no other linux computer around, so I was not able to test the pairing. I did set it up for a different user, but neither account seemed to see the other’s couchdb running.

Will I be able to sync non-record databases to ubuntuone ? If not, what else could I do to have an online couchdb to sync between my various work machines ?

Greg Wilson: Serendipitous and Unexpected

Sun, 22 Nov 2009 21:14:34 +0000

Via Ryan Lilian:

Most research effort does not produce what is thought of as a traditionally publishable result. That doesn’t mean, however, that nothing was gained by conducting the research. These results, whether they are failures or merely perplexing, can provide valuable insights into open problems and prevent other researchers from duplicating work. We started a journal that focuses on serendipitous (I have no idea why this worked) and unexpected (it seems like this technique should work on this problem but it doesn’t) results. The goal of the journal is to provide a venue where ideas can flow and be debated.

The Journal of Serendipitous and Unexpected Results (JSUR) is an open-access forum for researchers seeking to further scientific discovery by sharing surprising or unexpected results. These results should provide guidance toward the verification (or negation) of extant hypotheses. JSUR has two branches, one focusing on Computational Sciences and the other on the Life Sciences. JSUR submissions include, but are not limited to, short communications of recent research results, full-length papers, review articles, and opinion pieces.

Recently, we launched the beta version of the journal site at http://jsur.org. We would love to get your feedback and even better, a submission for the first issue.

To get the journal started, we’re looking to collect a large number of short (2-4 page) reports. I know you have something to publish. Please help us spread the word and forward this information to interested colleagues.

Sincerely,
The JSUR Editorial Board

Thomas Vander Stichele: python setup.py bdist_rpm and man pages

Sun, 22 Nov 2009 21:10:46 +0000

Today while using bdist_rpm to build a quick package to test desktopcouch, the build failed on man pages. It took a while to figure out what to google for, but it turns out it’s this bug which will turn 7 years old come Friday.

That is a little embarassing. I don’t know if it’s because bdist_rpm hardly ever gets used or because it’s hard to figure out what the bug is or how to report it.

In any case, I attached a patch that worked for me today against python 2.6. I punted on actually rebuilding Python for Fedora for my repo since it’s a little bit overkill (the plan is to build a proper .spec file anyway for the package), but if you need the patch it’s in the bug report.

Malthe Borch: Don't look back in anger

Sun, 22 Nov 2009 21:08:00 +0000

Next Thursday it will be a year since the release of Python 3. You've got to be out early in Christmas time which is my excuse for the untimeliness of this rant.

I thought I'd try to see what the fuss was about, i.e. does anything work? WSGI? It doesn't. While very informative, the roadmap for the WSGI specification is also very unreadable. Let's see if we can extract some knowledge out of it!

It seems to me that people sit with a copy of Essential Python 2 and try to see how to interpret the WSGI specification against an imaginary Essential Python 3, e.g. "In Python 2 this was a string; what should–right, so bytes. Next, in Python 2 this was..." – to no end. Let's cut to the chase:

Rule 1: Strings should be strings, chunks should be bytes.

The roadmap then mentions that on Jython, sockets will return unicode. That's wrong, so that can be conveniently filed as a bug. It continues with another string prediction: "The status line specified by the WSGI application must be a byte string." Why must this be a byte string? It is obviously an actual string. I think it's even a Freudian slip to call it a "byte string". Please see rule 1.

Then it gets a little unclear, but skipping ahead a handful of paragraphs, we come to this bullet point: "For the WSGI variable 'wsgi.url_scheme' contained in the WSGI environment, the value of the variable should be a native string." I think this is what the rest of the world calls the server protocol. If I can make a humble suggestion:

Rule 2: Please remove all proprietary keys from the HTTP environment.

Incidentally, the presence of these keys is probably what inspires developers to put all sorts of other proprietary keys in the HTTP environment. I have previously argued why this is wrong. It occurs to me that when the WSGI specification says environ, it means request. What the application needs is the request which holds the HTTP environment and the content body. Everybody uses the term request for this; an HTTP request. It's convenient to split a request up in an environment, which is the first few lines of the socket input and the body, which is the rest. This body might be quite large, so the interface must be an iterable of binary chunks.

Note that if we had a request object instead of this augmented environment, middlewares and frameworks that wanted to add touches of magic could wrap it or even change its class. Today they have to insert special string keys into a dictionary and explain how this isn't that wrong in their documentation (e.g. "it's better than a thread-local").

Okay, this has turned sour. All I wanted was to try out Python 3. I'm not sure the current approach is going to make this any fun. The 2to3 solution is not appealing. It's a bit like an in-law staying temporarily in the house. When does it end? When is the try-import-except dance over? Am I ever going to import anything from the functools module and feel comfortable about it? Here's a moratorium for you:

Rule 3: Forget Python 2.

From now on it's on the paying clients only list. Join the movement of full incompatibility and never look back! The downside is that we have to rewrite everything. I think it's worth it. Microsoft should have done it, Apple did it.

Who got the better uptake?

Anna Martelli Ravenscroft: Cooking and nostalgia

Sun, 22 Nov 2009 20:11:55 +0000

Today, I'm cooking wild boar ragu in the slow cooker, and polenta. And it reminds me of when we lived in Italia.
Wild boar is something we ate with my step-son's girlfriend - her family had local hunters give them wild boar. The flavor is rich and gamey in a good way.
But Polenta will always remind me of our honeymoon - up in the Italian Alps. The little hotel we stayed at invited all its guests to lunch at their Malga (a traditional farmhouse up slope from the village). To get there, we all walked along a path, through woods and flowers, along a creek, and then emerging to a pasture, high in the mountains, surrounded by grass, and rock, and flowers, with the sound of the creek audible below. Children played, running around with the dog that had accompanied us up, a happy little dog who was all energy and bounce. We sat and drank wine and chatted and explored while the hotel folk cooked over an open fire. The meat had been roasting since the morning, but the polenta - oh, the polenta pot was a thing to behold. A HUGE copper pot, over a wood fire built into the side of the farmhouse, at just the right height for us to take turns stirring with a wooden spoon that was at least 2 feet long. The polenta bubbled and boiled like the proverbial cauldron for hours, getting thicker, and creamier. Meanwhile, they plied us with coldcuts and bread and wine, but then came the grilled pork sausage and then the roasted venison and the polenta. All the picnic tables were set with bottles of red wine, as we sat at picnic tables under the summer sun, and ate until we couldn't eat any more. And as the evening began, we all walked together, down the long path, (very glad it was downhill), back to the hotel, before the next day's adventures.

IronPython-URLs: A Good Mix 32: Alpha Encoding Files, Embedding IronPython in Russian, IronSharePoint, IronRuby, World's Worst Paint Program

Sun, 22 Nov 2009 20:06:16 +0000

More IronPython and DLR related projects, articles and news from around the web.

Alpha-encoding file versions

Steve Gilham shows how to generated encoded version strings for installers using IronPython:

When building installers the UpgradeVersion must have a unique property value that is an installer public property (upper-case alpha). So, what better way of adding uniqueness than making it have the form "product name + product version" with the version suitably encoded...

So, a script for turning a file version (4 x 16bit ints) encoded as a System.Version into a short alpha string, assuming that Major and Minor will be small, and that common approaches are to step Build, to use a stepped Build plus date-stamped Revision, or a timestamp Build and Revision.

C# + IronPython: вызов методов

An article on IronPython, and in particular using IronPython from C#, in Russian.

IronSharePoint Source Code Available

A while ago Christian Glessner announced the start of the IronSharePoint project: showing how to develop SharePoint applications with IronPython. Well, he hasn't released anything but there is now code available for download from the Codeplex site. Could be worth exploring if you work with SharePoint...

Passing the Torch

Since early 2007 John Lam has been working for Microsoft as program manager of IronRuby, the implementation of Ruby on the Dynamic Language Runtime. Up until almost eleven months ago that is. John quietly transferred to a 'secret' project within Microsoft, a project that has also claimed Curt Hagenlocher (IronCurt). As they are *close* to being able to announce the secret project John has made his departure from the IronRuby team official and handed over the reigns to Jimmy Schementi who was previously only in charge of the integration between dynamic languages and Silverlight. Now Jimmy is doing both.

World's Worst Paint Program

Dino Veihland presented on IronPython and the Dynamic Language Runtime at the recent PDC conference. Part of his talk featured the "World's Worst Paint Program", which from Dino's description includes both IronPython and IronRuby code. He has made this available for download from the IronPython Codeplex Site.

This is built using Visual Studio 2010 Beta 2 and the IronPython CTP for that release - so you'll need to download both of those. I've also included the Python and Ruby code that I used during the demonstration in a file called code.txt. The paint program it's self is in the final form at the end of the demo.

Rene Dudfield: 2to3c: an implementation of Python's 2to3 for C code

Sun, 22 Nov 2009 18:55:12 +0000

"2to3c: an implementation of Python's 2to3 for C code"

See http://dmalcolm.livejournal.com/3935.html for a description of this tool.

It works on C modules. So it should be easier for people to do ports to py3k for their C modules.

The 2to3c tool uses Coccinelle for transformations on the C code. This program has been used for linux, and other software... for updating code when their APIs change.

The perfect fit for the python C API changes!

Ned Batchelder's blog: Memory leak mystery

Sun, 22 Nov 2009 16:57:06 +0000

Here's a detailed story of finding and fixing a memory leak in a Python C extension, complete with testing mysteries.

A user of coverage.py reported that running it consumed all his memory. No one else had mentioned anything like this, but it's a beta version, and those don't get nearly the attention as regular releases.

He helpfully had reproduced the problem on a large public test suite by running Django's test suite under coverage. I tried running the Django test suite with the latest coverage code, and sure enough, the memory ballooned. A plain run consumes about 100Mb, but with coverage it was up to 500Mb. I expect running with coverage.py to use a little more memory, since it has to collect its data in memory, but the amount of data should be very small, proportional not to the execution time, but to the number of distinct lines of code executed.

Coverage.py has two different trace modules, one in C and one in Python. Forcing the Python tracer (by using the --timid flag) showed no memory leak, confirming my suspicion that the C code was at fault.

I made a small test file to run for a long time:

def once(x): if x: return 10 else: return 11 for i in xrange(10**9): once(i)

Running it, nothing untoward happened, no memory leak. OK, so something was different between Django and my test file. Of course, lots of things are different between them, the task was to figure out what was the important difference.

Poring over the C tracer code, there are plenty of interesting program events that cause stuff to happen internally: calling a function, returning from a function, the first execution in a file, raising an exception, deep recursion, and so on.

I added some statistics gathering in the tracer code to see which of these might be happening enough to correlate with the memory leak. Exceptions and deep recursion seemed like likely candidates since my test code had neither.

It turned out that the Django code didn't recur deeply: it never got past the internally-interesting milestone of 100 stack frames. It did throw a lot of exceptions: the full test suite raises 1,862,318 of them. Examining the code, though, there's not much happening there that could leak, and disabling that code proved that it wasn't the issue.

How about first executions in a file? The Django test suite touches 9300 files (or 9300 different ways to refer to files). This is certainly more than my test file so maybe it was the source of the leak. When a new file is visited, the tracer calls back to a Python function to determine if the file is interesting enough to trace. One of the arguments to the callback is the stack frame. This seemed like a good candidate for a leak: mishandling the reference count on a frame could keep it around forever, and frames refer to lots of other things, so tons of memory could leak each of those 9300 times.

But a few experiments, ending with completely disabling the callback, proved that this code was not to blame either. I was running out of places to look.

The per-line code was about all that was left. It didn't make any sense for this to be the problem, since my test file runs many many lines just like Django does. But just to rule it out, I commented out the line of C code that adds the line number to the data dictionary. To my surprise, the test suite no longer leaked memory!

Here's the buggy line (simplified):

PyDict_SetItem(file_dict, PyInt_FromLong(frame->f_lineno), Py_None);

The problem is that PyInt_FromLong returns a new reference, and PyDict_SetItem doesn't steal the reference. This is a memory leak. (If you don't know what I'm talking about with this reference stuff, take a peek at A Whirlwind Excursion through Python C Extensions.)

The fix was straightforward; I had to explicitly release my reference on the integer object:

PyObject * this_line = PyInt_FromLong(frame->f_lineno); PyDict_SetItem(file_dict, this_line, Py_None); Py_DECREF(this_line);

Running this on the Django tests showed that the problem was fixed! Great!

But back to the remaining mystery: why did my test file, which executed that same buggy line billions of times, not leak? The docs for PyInt_FromLong provided the answer:

The current implementation keeps an array of integer objects for all integers between -5 and 256, when you create an int in that range you actually just get back a reference to the existing object.

Aha! Because my test file was only 10 lines long, the line numbers were all less than 256, so the integer objects returned by PyInt_FromLong were all pre-allocated and held for the lifetime of the process. The buggy code was mishandling the reference count, but it didn't matter, since no new objects were allocated.

Finally I had the answer to my question: the difference between my file and the Django source is that Django has files longer than 256 lines.

And in fact, using the buggy tracer code, and adding 300 blank lines to my test file proved the point: once my simple code had line numbers above 256, it leaked like a sieve!

If there's a lesson in this, it's that complex systems introduce leaky abstractions in unexpected places. This has unfortunate implications for testing: how do I know if I've got test cases that can ferret out potential problems like this? Does white-box testing extend to reading up on the internal details of Python integers? If I add test cases with 300 lines of source, how do I know there isn't some other effect that only kicks in at 5000?

Be careful out there...

Brett Cannon: realStorage 2.0 is out

Sun, 22 Nov 2009 16:32:18 +0000

realStorage 2.0 is now officially released. For those that don't know, realStorage is a compatibility library that handles browser incompatibilities for the W3C Web Storage API as of August. What's new in this release and why I am specifically supporting the August version of the spec and not the newest one, read on.

The biggest deal in realStorage 2 is support for Chrome 4 dev and IE 8. I don't know when Chrome's dev channel flipped the bit to let web pages access localStorage, but I have it under OS X so I was able to verify support.

As for IE 8, supporting that browser required some API tweaking. Turns out the browser has no support for accessors on non-DOM objects. That meant I had to make the length property optional and add a getLength() function.

The other major addition is Gears support. I needed to benchmark some of my PhD work on Chrome before it had localStorage support, so I had to add support to use Gears to get any results. This back-end is provided mostly for transition purposes as it looks like Gears might be on its way out in favor of the Web Storage API.

Otherwise I took this new major release as a chance to rename some things and continue to try to optimize the code as much as possible.

One thing to keep in mind, though, is that realStorage currently only supports the Web Storage spec as of August. The reason I am using an old version is that after the August draft the API switched to accepting structured clones instead of only strings. This makes implementation exceedingly difficult as it would mean coming up with my own serialization format for things such as Regexp and ImageData objects. I don't want to deal with that, so realStorage sticks to the old spec.

I emailed the WHATWG about this and voiced my concerns. Some supported me in wanting to roll back to the strings-only version. Others wanted to drop the spec entirely, but since all current browsers support the API that was deemed unreasonable. Otherwise wanted to tweak it to make it asynchronous to avoid lockup edge cases that exist. In the end the thread kind of died out and it seems the Web Database spec will become the more focused one, especially since databases can be accessed in web workers and they standardized what SQL dialect to support (sqlite 3.16.9 specifically).

At this point I consider realStorage done until the next round of major browser updates comes around. If they support the new version of the spec I will update realStorage to 3.0 to accept only strings. Otherwise there is nothing critical left to add to this code.

Python 411 Podcast: Programming in Python 3

Sun, 22 Nov 2009 15:42:07 +0000

This podcast reviews "Programming in Python 3" by Mark Summerfield and then talks about the transition from Python 2 to Python 3 as a critical juncture in the community. It also contains a ringing endorsement of the MIT Python video lectures.