Planet Python
Last update: May 22, 2013 07:47 PM
May 22, 2013
Juho Vepsäläinen
Thoughts on Kasvu Open Forum and Djangocon Finland 2012
Two days, two conferences. The first was a local, business oriented one, Kasvu Open Forum. The second, Djangocon Finland 2012, focused mainly on technology. I gave two talks in the latter one. It was very nice to experience both events and meet some new people and a few old acquaintances.
Kasvu Open Forum
We didn't make it to the finals and weren't impressed by the quality of the feedback given. This event totally made up for it. This is definitely something they can improve on the next year. The last thing you want to do is to discourage some potential idea or company. After all Kasvu Open is in the business of creating new business.
Djangocon Finland 2012
Thoughts on "Kaleva.fi - how we replaced 10 years of legacy code in one year"
It was particularly interesting to see what Kaleva.fi looks like from "outside" in terms of DevOps. I participated in the project as a software designer during the past year for a period of a few months. So I got to know certain bits of it quite well. I never really looked into the overall infrastructure (too busy staring at my code :) ), though.There were many interesting tidbits in Markus' talk. Especially the bits on scaling the service were interesting. It's quite different to develop a service used by hundreds of thousands than something that has only a few users. You get a lot of new problems to solve.
Thoughts on other talks
bongaus.fi - Spotting Service Powered by Django
In development of bongaus.fi we noticed Pareto principle applies quite well here. It takes only perhaps 20% or so time to get the relevant bits done. The rest is just tweaking and dealing the corner cases. And boy that sure can take time.
Another important thing to pick out is the value of pivots. Even if you are doing something and going to a certain direction, doesn't mean you should be going there indefinitely. It can pay off to adjust the trajectory and try something perhaps a bit different. I believe the willingness to pivot is one of the key attributes of startups that become successful.
Speccer
Conclusion
Vasudev Ram
Zato, an open source ESB in Python
Zato is an ESB (Enterprise Service Bus) which is written in Python. It is free and open source (LGPL).
http://en.m.wikipedia.org/wiki/Enterprise_service_bus
Tibco was one of the first ESB products and Mule is an open source one.
According to the Zato site:
It supports HTTP, JSON, SOAP, Redis, JMS WebSphere MQ, ZeroMQ, FTP, SQL.
It has a web admin GUI, a CLI and an API.
Documentation and commercial support are available.
I got to know about Zato recently from the main developer, Dariusz Suchojad, who had earlier written to me regarding my blog post about PyMQI:
PyMQI, Python interface to IBM WebSphereMQ (formerly IBM MQSeries):
http://jugad2.blogspot.com/2013/02/pymqi-python-interface-to-ibm.html
Dariusz was a maintainer of PyMQI and also a developer on Spring Python, which is sort of a port of the Java Spring framework to Python.
Zato docs (quite detailed):
https://zato.io/docs/index.html
Part 1 of a basic Zato tutorial:
https://zato.io/docs/tutorial/01.html
I took a look at the tutorial. Broadly, it shows how to install Zato, create a simple Zato service in Python, that talks to PostgreSQL and Redis, and deploy it. Two servers get created, behind a load-balancer, and the service gets hot-deployed to the servers. Then curl is used to access the service. (This tutorial does not create a real client; curl is used to simulate one.)
Zato looks interesting and powerful (and somewhat complex, but that is to be expected for a product like an ESB).
I will check it out more and then report on my findings.
- Vasudev Ram
dancingbison.com
May 21, 2013
PyPy Development
PyPy 2.0.2 - Fermi Panini
We're pleased to announce PyPy 2.0.2. This is a stable bugfix release over 2.0 and 2.0.1. You can download it here:
http://pypy.org/download.html
It fixes a crash in the JIT when calling external C functions (with ctypes/cffi) in a multithreaded context.
What is PyPy?
PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (pypy 2.0 and cpython 2.7.3 performance comparison) due to its integrated tracing JIT compiler.
This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Support for ARM is progressing but not bug-free yet.
Highlights
This release contains only the fix described above. A crash (or wrong results) used to occur if all these conditions were true:
- your program is multithreaded;
- it runs on a single-core machine or a heavily-loaded multi-core one;
- it uses ctypes or cffi to issue external calls to C functions.
This was fixed in the branch emit-call-x86 (see the example file bug1.py).
Cheers, arigo et. al. for the PyPy team
Brian Okken
How not to test, part 1
or Complete coverage testing or More is Better testing Update 5/22/13: I’ve received several criticisms of this post. Most of them constructive. :) I’d like to take some time to re-read the post, re-read the feedback, and compose a more coherent statement about my thoughts on the matter. I agree with some points of my [...]
The post How not to test, part 1 appeared first on Python Testing.
Reinout van Rees
Advanced Python through Django: metaclasses - Peter Inglesby
Metaclasses are a handy feature of Python and Django makes good use of them.
When you create certain kinds of classes in Django, a metaclass will do something to the class before it is created. For forms, the various attributes of the class are converted into a base_fields dictionary on the class.
Similarly, a subclass of Model also fires up a metaclass that does some registering. A foreignkey to another model adds a relation back on that other model, for instance.
As a recap, a class is something that can be instantiated into an object. It can have an __init__() method that does something upon instantiation. type(your_instance) will return the class.
Did you know that you can create classes dynamically? See for yourself:
>>> name = 'ExampleClass'
>>> bases = (object,)
>>> attrs = {'__init__': lambda self: print('Hello from __init__')}
>>> ExampleClass = type(name, bases, attrs)
>>> example = ExampleClass()
Hello From __init__
>>> type(example)
<class '__console__.ExampleClass'>
So... we can actually control how classes are created! You could create a create_class() method that calls type but that modifies, for instance, the name. Or we could take all the attributes and add them to a base_fields dictionary on the instance. Hey, that's what we saw in the first Django form example!
Now, what is type exactly? It is a class that creates classes.
This also means we can subclass it! The most useful thing to override in our subclass is the __new__() method. The __init__() method creates instances from the class, the __new__() creates classes. (Update: it is a little bit different, actually, see Venelin's comment below). So again we can modify the name and/or the attributes.
How do you use it in practice? Normally you'd set a __metaclass__ attribute on a class. This tells python to use that metaclass for creating the class. The same for subclasses. This is how our Django form classes use the metaclass specified in Django's base Form class.
Django uses metaclasses in five places: admin media, models, forms, formfields, form widgets. Grep for metaclass in your local django source code once to get a better feel for how Django uses it.
Note on python 3: it uses a slightly different syntax for specifying metaclasses. So Django 1.5 uses six to support both ways in a single codebase.
Warning: don't overuse metaclasses. They can make code difficult to debug and follow. Use Django as a good example of how to use metaclasses. Django saves you a lot of work by using metaclasses in a few locations.
See https://github.com/inglesp/Metaclasses
Nice way of giving a presentation, btw. Some sort of semi-interactive python prompt. The software is online at https://github.com/inglesp/prescons
Python 4 Kids
Comprehending Lists and Tuples
The last couple of tutorials have been a bit heavy, so this week’s tutorials are going to balance that out by being a little light. We’re having a look at a couple of different aspects of the Python language.
List Comprehensions
The first is comprehensions:
>>> a_list = [1,2,3,4,5] >>> b_list= [element*2 for element in a_list] >>> b_list [2, 4, 6, 8, 10]
In this example, we have automatically generated a new list b_list from an existing list* a_list by using a list comprehension. The structure of the comprehension is, I hope comprehensible. If not, it is a little hard to explain in steps. The comprehension above is equivalent to:
>>> b_list = [] >>> for element in a_list: ... b_list.append(element*2) ... >>> b_list [2, 4, 6, 8, 10]
We could call the variable element anything we liked – it is just another variable:
>>> b_list = [baloney*2 for baloney in a_list] >>> b_list [2, 4, 6, 8, 10] >>>
You can also add conditions. Let’s say you only wanted the even elements in a_list:
>>> b_list = [element for element in a_list if element%2 ==0] #ie remainder is 0 when dividing by two >>> b_list [2, 4] >>>
The condition here is if element%2 ==0, but you can substitute other conditions (as long as they resolve to either True or False). You can have a comprehension which relies on multiple variables (this example is from the Python documentation):
>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] [(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
Unwinding this is a little tricky – for each element in [1,2,3] it runs through each element in [3,1,4] (ie 9 comparisons in all) and appends the tuple if the elements are not equal.
List comprehensions provide an easy shorthand for constructing lists. Moreover, they are often more readable than writing out the for loops explicitly.
Tuples
The second topic is tuples.
The other thing we’re going to look at is tuples (pronounced, variously, “tupp-lls”, “two-pls” and, to some people, “tyou-pls” – my preference is tupp-lls, rhymes with couples). Tuples are a little like lists, except that they are immutable. That is, once they are made they cannot be changed. Tuples are made by putting the elements in round braces [actually, adding commas between them and, sometimes, also adding braces]:
>>> my_tuple = (1,2,3) >>> my_tuple (1, 2, 3) >>> my_list = [1,2,3] >>> my_list [1, 2, 3]
Elements of the tuple are referenced in the same way as you’d reference a list but, unlike lists, these elements cannot be changed (tuples are “immutable”):
>>>my_tuple[0] 1 >>>my_tuple[0]=2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'tuple' object does not support item assignment
The TypeError is telling us that you can’t change the elements of a tuple. To make a tuple with a single element is a little difficult because the interpreter could just interpret the parentheses as determining order of operation (think (1+2)*3). Instead, we put a comma after the single element to indicate that we are creating a tuple:
>>> my_tuple = (1) # no comma, so Python just thinks it's a number >>> my_tuple 1 >>> my_tuple[0] # just a number, so has no elements Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'int' object has no attribute '__getitem__' >>> my_tuple= (1,) >>> my_tuple (1,) >>>my_tuple[0] 1
So, if a tuple does what a list can do, only less, why bother with a tuple? Why not use a list? Well, tuples are easier for the Python interpreter to deal with and therefore might end up being faster. Tuples might also indicate that you have an ordered list where the ordering has some meaning (for example, a date tuple might store year, month and day (in that order) in tuple). The fact that you’re using a tuple then flags that each of the entries has a distinct meaning or use and that their order is significant. Another pragmatic reason to use a tuple is when you have data which you know shouldn’t change (like a constant). So, if you accidentally try to change one of the tuple’s entries, you will have an error thrown.
Finally, since tuples are immutable they can be used as keys in dictionaries, unlike lists:
>>> my_dict={}
>>> my_tuple=("A Name",23,"A location")
>>> my_list= list(my_tuple)
>>> my_dict[my_tuple]="Client 1"
>>> my_dict[my_list]="Client 2"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
Notes:
* as per Don’s comment, you can use any iterator, but we haven’t talked about them yet, so they don’t yet exist.
Reinout van Rees
The imaginative programmer - Zed Shaw
His goal: teach programmers to be more creative.
He's got a love/hate relationship with creativity. The first part of his talk was impossible to summarize. You'll have to watch the video later on :-)
- Artists tell him he's not artistic because he works on developing technical skills.
- Guitarists tell him he's not a real guitarist as he doesn't play in a band. And 'cause he builds his own guitars he's a programmer, not a Real Guitarist.
- Writers tell him he's not a writer because he writes technical books.
- Programmers tell him he's not a programmer because he doesn't work on their project. And by the way, he's a (technical) writer now, so he's not a programmer.
He's not creative enough. Or so the others say. Or he's not acceptable. How to deal with creativity? In a way, you can re-phrase creativity. Programmers are always making something from nothing, right? Isn't that the pinnacle of creativity?
Here are four hypothetical persons:
- Technique, no imagination: a stereotypical programmer.
- Imagination, no technique: stereotypical biz dude.
- No imagination, no technique. Probably doesn't exist.
- Both imagination and technique. Zed's goal.
Zed's imaginative programmer process. Everyone has a process (if they're good), here's the one he proposes to help you be more creative:
- You start with an idea.
- You establish a concept that helps form the idea. It gives your idea clothes.
- Research techniques or tools. Do some research or you'll pay for it later on.
- Refine the concept through composition. So put a box around the vast world of available possibilities. Just mark which features are inside and outside the concept.
- Explore through prototypes. Throw away code or use paper prototypes for instance. This saves you so much time later.
- You make it real.
We are programmers, so we should iterate this process.
He tried this process with http://projectzorn.com/, going through all the stages. And he's probably going to work on it during the sprints this weekend, so join him if you want.
Vasudev Ram
A partial crossword solver in Python
A Cryptic Crossword Clue Solver ←
Saw this via Twitter.
It is a partial crossword solver, because it only helps solve a particular category of crossword clues - those in which the clue (which is usually a sentence or phrase) contains both a "definition" of the answer as well a hint of some kind that leads to the same answer. This solver tries to compute the answer using both the definition and the hint, and checks whether the results match. Ingenious.
I found it interesting because this is a somewhat difficult problem, and yet the author managed to create a solution (involving NLTK and parsing) that works in many, if not all cases.
Also, long ago, in college days, I had written another kind of partial crossword solver (in BASIC); it was much simpler, using a brute force method - what it did was help solve the kind of crossword clues in which the answer is a permutation of a substring of the characters comprising the clue sentence or phrase. The program would generate and display on the screen, all possible permutations of all possible substrings of the sentence, that were of the same length as the answer. Then you had to view those permutations and guess whether any of them was the right answer, based on the clue.
I wrote the permutation-generation code by hand, but saw recently that the Python itertools module has methods to generate permutations (as well as combinations) from sequences:
http://docs.python.org/2/library/itertools.html
http://en.m.wikipedia.org/wiki/Permutation
http://en.wikipedia.org/wiki/Crossword
- Vasudev Ram
dancingbison.com
Montreal Python User Group
Python Projects Night V
Montréal-Python would like to invite you all to the next Python Project Night, on Thursday, the 30th of May, 2013 at the offices of Caravan.
Like on previous nights, it’s an informal meetup where people work on different projects and generally mess around with Python code. Everyone is welcome, from the grizzled python hacker to the absolute beginner who just finished their first workshop. We will encourage people to help each other and we will also have dedicated helpers to help people get started.
As per usual, beer and pizza are provided, so just bring your laptop computer.
If you have any projects you would like to work on, please post them on the mailing list: montrealpython@googlegroups.com. If you don’t have any ideas, don’t worry, we will find you a project that needs help.
When: Thursday, May 30th 2013 from 7 PM to 9:30 PM
Where: Caravan, 5334 de Gaspé, office #1204 (Montreal)
Where to sign up: Please sign up on our Eventbright event
We would like to thank 
for hosting our event!
Armin Ronacher
Porting to Python 3 Redux
After a very painful porting experience with Jinja2 to Python 3 I basically left the project idling around for a while because I was too afraid of breaking Python 3 support. The approach I used was one codebase that was written in Python 2 and translated with 2to3 to Python 3 on installation. The unfortunate side-effect of that is that any change you do requires about a minute of translation which destroys all your iteration speeds. Thankfully it turns out that if you target the right versions of Python you can do much better.
Thomas Waldmann from the MoinMoin project started running Jinja2 through my python-modernize with the right parameters and ended up with a unified codebase that runs on 2.6, 2.7 and 3.3. With a bit of cleanup afterwards we were able to come up with a nice codebase that runs in all versions of Python and also looks like regular Python code for the most time.
Motivated by the results from this I went through the code a few more times and also started migrating some other code over to experiment more with unified codebases.
This is a selection of some tips and tricks I can now share in regards to accomplishing something similar.
Drop 2.5, 3.1 and 3.2
This is the most important tip. Dropping 2.5 by now is more than possible since very few people are still on it and dropping 3.1 and 3.2 are no brainers anyways considering the low adoption of Python 3 so far. But why would you drop those versions? The basic answer is that 2.6 and 3.3 have a lot of overlapping syntax and features that allow for code that works well with both:
Compatible string literals. 2.6 and 3.3 support the same syntax for strings. You can use 'foo' to refer to a native string (byte string on 2.x and a Unicode string on 3.x), u'foo' to refer to a Unicode string and b'foo' to refer to a bytestring or bytes object.
Compatible print syntax. In case you have a few print statements sitting around you can from __future__ import print_function and start using the print function without having to bind it to a different name or suffering from other inconsistencies.
Compatible exception catching syntax. Python 2.6 introduced except Exception as e which is also the syntax used on 3.x to catch down exceptions.
Class decorators are available. They are incredible useful to automatically correct moved interfaces without leaving a footprint in the class structure. For instance they can be used to automatically rename the iteration method from next to __next__ or __str__ to __unicode__ for Python 2.x.
Builtin next() function to invoke __next__ or next. This is helpful because they are performing about the same speed as calling the method directly so you don't pay much of a performance penalty compared to putting runtime checks into places or making a wrapper function yourself.
Python 2.6 added the bytearray type which has the same interface in that version of Python as the one in 3.3. This is useful because while Python 2.6 lacks the Python 3.3 bytes object it does have a builtin with that name but that's just another name for str which has vastly different behavior.
Python 3.3 reintroduces bytes-to-bytes and string-to-string codecs that were broken in 3.1 and 3.2. Unfortunately the interface for them is clunkier now and the aliases are missing, but it's much closer to what we had in 2.x than before.
This is particularly useful if you did stream based encoding and decoding. That functionality was plain missing between 3.0 up until 3.3.
Yes, the six module can get you quite far, but don't underestimate the impact of looking at nice code. With the Python 3 port I basically lost interest in maintaining Jinja2 because the codebase started to frustrate me. Back then a unified codebase was looking ugly and had a performance impact (six.b('foo') and six.u('foo') everywhere) or was plagued under the bad iteration speeds of 2to3. Not having to deal with any of that brings the joy back. Jinja2's codebase now looks like very clean and you have to find the Python 2/3 compatibility support. Very few paths in the code actually do something like if PY2:.
The rest of this article assumes that these are the Python versions you want to support. Also attempting to support Python 2.5 is a very painful thing to do and I strongly recommend against it. Supporting 3.2 is possible if you are willing to wrap all your strings in function calls which I don't recommend doing for aesthetic and performance reasons.
Skip Six
Six is a pretty neat library and this is where the Jinja2 port started out with. There however at the end of the day there is not much in six that actually is required for getting a Python 3 port running and a few things are missing. Six is definitely required if you want to support Python 2.5 but from 2.6 or later there really is not much of a reason to use six. Jinja2 ships a _compat module that contains the few helpers required. Including a few lines of non Python 3 code the whole compatibility module is less than 80 lines of code.
This saves you from the troubles where users might expect a different version of six because of another library or pulling in another dependency into your project.
Start with Modernize
To start with the port the python-modernize library is a good start. It is a version of 2to3 that produces code that runs in either. While it's pretty buggy still and the default options are not particularly great, it can get you quite far with regards to doing the boring parts for you. You will still need to go over the result and clean up some imports and ugly results.
Fix your Tests
Before you do anything else walk through your tests and make sure that they still make sense. A lot of the problems in the Python standard library in 3.0 and 3.1 came from the fact that the behavior of the testsuite changed through the conversion to Python 3 in unintended ways.
Write a Compatibility Module
So you're going to skip six, can you live without helpers? The answer is “no”. You will still need a small compatibility module but that is small enough that you can just keep it in your package. Here are some basic examples of what such a compatibility module could look like:
import sys
PY2 = sys.version_info[0] == 2
if not PY2:
text_type = str
string_types = (str,)
unichr = chr
else:
text_type = unicode
string_types = (str, unicode)
unichr = unichr
The exact contents of that module will depend on how much actually changed for you. In case of Jinja2 I put a whole bunch of functions in there. For instance it contains ifilter, imap and similar itertools functions that became builtins in 3.x. (I stuck with the Python 2.x functions to make it clear for the reader of the code that the iterator behavior is intended and not a bug).
Test for 2.x not 3.x
At one point there will be the requirement to check if you are executing on 2.x or 3.x. In that cases I would recommend checking for Python 2 first and putting Python 3 into your else branch instead of the other way round. That way you will have less ugly surprises when a Python 4 comes around at one point.
Good:
if PY2:
def __str__(self):
return self.__unicode__().encode('utf-8')
Less ideal:
if not PY3:
def __str__(self):
return self.__unicode__().encode('utf-8')
String Handling
The biggest change in Python 3 is without doubt the changes on the Unicode interface. Unfortunately these changes are very painful in some places and also inconsistently handled throughout the standard library. The majority of the time porting will clearly be wasted on this topic. This topic is a whole article by itself but here is a quick cheat sheet for porting that Jinja2 and Werkzeug follow:
'foo' always refers to what we call the native string of the implementation. This is the string used for identifiers, sourcecode, filenames and other low-level functions. Additionally in 2.x it's permissible as a literal in Unicode strings for as long as it's limited to ASCII only characters.
This property is very useful for unified codebases because the general trend with Python 3 is to introduce Unicode in some interfaces that previously did not support it, but never the inverse. Since native string literals “upgrade” to Unicode but still somewhat support Unicode in 2.x this string literal is very flexible.
For instance the datetime.strftime function strictly does not support Unicode in Python 2 but is Unicode only in 3.x. Because in most cases the return value on 2.x however was ASCII only things like this work really well in 2.x and 3.x:
>>> u'<p>Current time: %s' % datetime.datetime.utcnow().strftime('%H:%M') u'<p>Current time: 23:52'
The string passed to strftime is native (so bytes in 2.x and Unicode in 3.x). The return value is a native string again and ASCII only. As such both on 2.x and 3.x it will be a Unicode string once string formatted.
u'foo' always refers to a Unicode string. Many libraries already had pretty excellent Unicode support in 2.x so that literal should not be surprising to many.
b'foo' always refers to something that can hold arbitrary bytes. Since 2.6 does not actually have a bytes object like Python 3.3 has and Python 3.3 lacks an actual bytestring the usefulness of this literal is indeed a bit limited. It becomes immediately more useful when paired with the bytearray object which has the same interface on 2.x and 3.x:
>>> bytearray(b' foo ').strip() bytearray(b'foo')
Since it's also mutable it's quite efficient at modifying raw bytes and you can trivially convert it to something more conventional by wrapping the final result in bytes() again.
In addition to these basic rules I also add text_type, unichr and string_types variables to my compatibility module as shown above. With those available the big changes are:
- isinstance(x, basestring) becomes isinstance(x, string_types).
- isinstance(x, unicode) becomes isinstance(x, text_type).
- isinstance(x, str) with the intention of catching bytes becomes isinstance(x, bytes) or isinstance(x, (bytes, bytearray)).
I also created a implements_to_string class decorator that helps implementing classes with __unicode__ or __str__ methods:
if PY2:
def implements_to_string(cls):
cls.__unicode__ = cls.__str__
cls.__str__ = lambda x: x.__unicode__().encode('utf-8')
return cls
else:
implements_to_string = lambda x: x
The idea is that you just implement __str__ on both 2.x and 3.x and let it return Unicode strings (yes, looks a bit odd in 2.x) and the decorator automatically renames it to __unicode__ for 2.x and adds a __str__ that invokes __unicode__ and encodes the return value to utf-8. This pattern has been pretty common in the past with 2.x modules. For instance Jinja2 and Django use it.
Here is an example for the usage:
@implements_to_string
class User(object):
def __init__(self, username):
self.username = username
def __str__(self):
return self.username
Metaclass Syntax Changes
Since Python 3 changed the syntax for defining the metaclass to use in an incompatible way this makes porting a bit harder than it should be. Six has a with_metaclass function that can work around this issue but it generates a dummy class that shows up in the inheritance tree. For Jinja2 I was not happy enough with that solution and modified it a bit. The external API is the same but the implementation uses a temporary class to hook in the metaclass. The benefit is that you don't have to pay a performance penalty for using it and your inheritance tree stays nice.
The code is a bit hard to understand. The basic idea is exploiting the idea that metaclasses can customize class creation and are picked by by the parent class. This particular implementation uses a metaclass to remove its own parent from the inheritance tree on subclassing. The end result is that the function creates a dummy class with a dummy metaclass. Once subclassed the dummy classes metaclass is used which has a constructor that basically instances a new class from the original parent and the actually intended metaclass. That way the dummy class and dummy metaclass never show up.
This is what it looks like:
def with_metaclass(meta, *bases):
class metaclass(meta):
__call__ = type.__call__
__init__ = type.__init__
def __new__(cls, name, this_bases, d):
if this_bases is None:
return type.__new__(cls, name, (), d)
return meta(name, bases, d)
return metaclass('temporary_class', None, {})
And here is how you use it:
class BaseForm(object):
pass
class FormType(type):
pass
class Form(with_metaclass(FormType, BaseForm)):
pass
Dictionaries
One of the more annoying changes in Python 3 are the changes on the dictionary iterator protocols. In Python 2 all dictionaries had keys(), values() and items() that returned lists and iterkeys(), itervalues() and iteritems() that returned iterators. In Python 3 none of that exists any more. Instead they were replaced with new methods that return view objects.
keys() returns a key view which behaves like some sort of read-only set, values() which returns a read-only container and iterable (not an iterator!) and items() which returns some sort of read-only set-like object. Unlike regular sets it however can also point to mutable objects in which case some methods will fail at runtime.
On the positive side a lot of people missed that views are not iterators so in many cases you can just ignore that. Werkzeug and Django implement a bunch of custom dictionary objects and in both cases the decision was made to just ignore the existence of view objects and let keys() and friends return iterators.
This is currently the only sensible thing to do due to limitations of the Python interpreter. There are a few problems with it:
- The fact that the views are not iterators by themselves mean that in the average case you create a temporary object for no good reason.
- The set-like behavior of the builtin dictionary views cannot be replicated in pure Python due to limitations in the interpreter.
- Implementing views for 3.x and iterators for 2.x would mean a lot of code duplication.
This is what the Jinja2 codebase went with for iterating over dictionaries:
if PY2:
iterkeys = lambda d: d.iterkeys()
itervalues = lambda d: d.itervalues()
iteritems = lambda d: d.iteritems()
else:
iterkeys = lambda d: iter(d.keys())
itervalues = lambda d: iter(d.values())
iteritems = lambda d: iter(d.items())
For implementing dictionary like objects a class decorator can become useful again:
if PY2:
def implements_dict_iteration(cls):
cls.iterkeys = cls.keys
cls.itervalues = cls.values
cls.iteritems = cls.items
cls.keys = lambda x: list(x.iterkeys())
cls.values = lambda x: list(x.itervalues())
cls.items = lambda x: list(x.iteritems())
return cls
else:
implements_dict_iteration = lambda x: x
In that case all you need to do is to implement the keys() and friends method as iterators and the rest happens automatically:
@implements_dict_iteration
class MyDict(object):
...
def keys(self):
for key, value in iteritems(self):
yield key
def values(self):
for key, value in iteritems(self):
yield value
def items(self):
...
General Iterator Changes
Since iterators changed in general a bit of help is needed to make this painless. The only change really is the transition from next() to __next__. Thankfully this is already transparently handled. The only thing you really need to change is to go from x.next() to next(x) and the language does the rest.
If you plan on defining iterators a class decorator again becomes helpful:
if PY2:
def implements_iterator(cls):
cls.next = cls.__next__
del cls.__next__
return cls
else:
implements_iterator = lambda x: x
For implementing this class just name the iteration step method __next__ in all versions:
@implements_iterator
class UppercasingIterator(object):
def __init__(self, iterable):
self._iter = iter(iterable)
def __iter__(self):
return self
def __next__(self):
return next(self._iter).upper()
Transformation Codecs
One of the nice features of the Python 2 encoding protocol was that it was independent of types. You could register an encoding that would transform a csv file into a numpy array if you would have preferred that. This feature however was not well known since the primary exposed interface of encodings was attached to string objects. Since they got stricter in 3.x a lot of that functionality was removed in 3.0 but later reintroduced in 3.3 again since it proved useful. Basically all codecs that did not convert between Unicode and bytes or the other way round were unavailable until 3.3. Among those codecs are the hex and base64 codec.
There are two use cases for those codecs: operations on strings and operations on streams. The former is well known as str.encode() in 2.x but now looks different if you want to support 2.x and 3.x due to the changed string API:
>>> import codecs
>>> codecs.encode(b'Hey!', 'base64_codec')
'SGV5IQ==\n'
You will also notice that the codecs are missing the aliases in 3.3 which requires you to write 'base64_codec' instead of 'base64'.
(These codecs are preferable over the functions in the binascii module because they support operations on streams through the incremental encoding and decoding support.)
Other Notes
There are still a few places where I don't have nice solutions for yet or are generally annoying to deal with but they are getting fewer. Some of them are unfortunately now part of the Python 3 API are hard to discover until you trigger an edge case.
Filesystem and file IO access continues to be annoying to deal with on Linux due to it not being based on Unicode. The open() function and the filesystem layer have dangerous platform specific defaults. If I SSH into a en_US machine from an de_AT one for instance Python loves falling back to ASCII encoding for both file system and file operations.
Generally I noticed the most reliable way to do text on Python 3 that also works okay on 2.x is just to open files in binary mode and explicitly decode. Alternatively you can use the codecs.open or io.open function on 2.x and the builtin open on Python 3 with an explicit encoding.
URLs in the standard library are represented incorrectly as Unicode which causes some URLs to not be dealt with correctly on 3.x.
Raising exceptions with a traceback object requires a helper function since the syntax changed. This is very uncommon in general and easy enough to wrap. Since the syntax changed this is one of the situations where you will have to move code into an exec block:
if PY2: exec('def reraise(tp, value, tb):\n raise tp, value, tb') else: def reraise(tp, value, tb): raise value.with_traceback(tb)
The previous exec trick is useful in general if you have some code that depends on different syntax. Since exec itself has a different syntax now you won't be able to use it to execute something against an arbitrary namespace. This is not a huge deal because eval with compile can be used as a drop-in that works on both versions. Alternatively you can bootstrap an exec_ function through exec itself.
exec_ = lambda s, *a: eval(compile(s, '<string>', 'exec'), *a)
If you have a C module written on top of the Python C API: shoot yourself. There is no tooling available for that yet from what I know and so much stuff changed. Take this as an opportunity to ditch the way you build modules and redo it on top of cffi or ctypes. If that's not an option because you're something like numpy then you will just have to accept the pain. Maybe try writing some abomination on top of the C-preprocessor that makes porting easier.
Use tox for local testing. Being able to run your tests against all python versions at once is very helpful and will find you a lot of issues.
Outlook
Unified codebases for 2.x and 3.x are definitely within reach now. The majority of the porting time will still be spend trying to figure out how APIs are going to behave with regards to Unicode and interoperability with other modules that might have changed their API. In any case if you want to consider porting libraries don't bother with versions outside below 2.5, 3.0-3.2 and it will not hurt as much.
May 20, 2013
Tarek Ziade
A step-by-step introduction to Circus
Note
Circus is a process & socket manager. See https://circus.readthedocs.org
Photo by kennethreitz
During Django Con, I was asked how to use Circus to run & monitor a Python web application. The documentation has no single page step-by-step tutorial yet, so here goes... this blog post will be integrated into the documentation for the next release.
Installation
Circus is tested under Mac OS X and Linux, on the latest Python 2.6 and 2.7. To run a full Circus, you will also need libzmq, libevent & virtualenv.
Under Debuntu:
$ sudo apt-get install libzmq-dev libevent python-virtualenv
Create a virtualenv and install circus, circus-web and chaussette in it
$ virtualenv /tmp/circus $ cd /tmp/circus $ bin/pip install circus $ bin/pip install circus-web $ bin/pip install chaussette
Once this is done, you'll find a plethora of commands in the local bin dir.
Usage
Chaussette comes with a default Hello world app, try to run it:
$ bin/chaussette
You should be able to visit http://localhost:8080 and see hello world.
Stop Chaussette and add a circus.ini file in the directory containing:
[circus] stats_endpoint = tcp://127.0.0.1:5557 httpd = 1 [watcher:webapp] cmd = bin/chaussette --fd $(circus.sockets.web) numprocesses = 3 use_sockets = True [socket:web] host = 127.0.0.1 port = 9999
This config file tells Circus to bind a socket on port 9999 and run 3 chaussettes workers against it. It also activates the Circus web dashboard and the statistics module.
Save it & run it using circusd:
$ bin/circusd --daemon circus.ini
Now visit http://127.0.0.1:9999, you should see the hello world app.
You can also visit http://localhost:8080/ and enjoy the Circus web dashboard.
Interaction
Let's use the circusctl shell while the system is running:
$ bin/circusctl circusctl 0.7.1 circusd-stats: active circushttpd: active webapp: active (circusctl)
You get into an interactive shell. Type help to get all commands:
(circusctl) help Documented commands (type help <topic>): ======================================== add get list numprocesses quit rm start stop decr globaloptions listen numwatchers reload set stats dstats incr listsockets options restart signal status Undocumented commands: ====================== EOF help
Let's try basic things. Let's list the web workers processes and add a new one:
(circusctl) list webapp 13712,13713,13714 (circusctl) incr webapp 4 (circusctl) list webapp 13712,13713,13714,13973
Congrats, you've interacted with your Circus! Get off the shell with Ctrl+D and now run circus-top:
$ bin/circus-top
This is a top-like command to watch all your processes' memory and CPU usage in real time.
Hit Ctrl+C and now let's quit Circus completely via circus-ctl:
$ bin/circusctl quit ok
Next steps
You can plug your own WSGI application instead of Chaussette's hello world simply by pointing the application callable.
Chaussette also comes with many backends like Gevent or Meinheld.
Read https://chaussette.readthedocs.org/ for all options.
Enthought
Enthought awarded $1M DOE SBIR grant to develop open-source Python HPC framework
We are excited to announce that Enthought is undertaking a multi-year project to bring the strengths of NumPy to high-performance distributed computing. The goal is to provide a more intuitive and user-friendly interface to both distributed array computing and to high-performance parallel libraries. We will release the project as open source, providing another tool in [...]
Morten W Petersen
Jep, Jython and CPython compared, all in a nice little script
So, in relation to the work with Jep ( http://www.nidelven-it.no/weblogs/hosting/blog_entry?id=1368... ) - I've been building a script that can build and setup Jep, Jython and CPython in a directory, and then have a compare.sh script that runs the same code on the 3 different systems.
The installation script is here:
https://raw.github.com/morphex/PythonCompare/master/install....
Here's the output from the generated compare.sh file:
morphex@copyleft-laptop:~/projects/self/jython_jepp_installer6$ ./compare.sh
Starting comparison of Jython, Jepp and CPython..
Starting with Jep..
0.39471411705
0.341079950333
0.333596944809
0.34021282196
0.322955131531
0.322613954544
0.322247982025
0.323844909668
0.319895029068
0.324920892715
Ran in milliseconds: 3878.0
Now Jython..
1.25800013542
0.579999923706
0.436999797821
0.361000061035
0.43799996376
0.287999868393
0.289000034332
0.910000085831
0.289000034332
0.290999889374
Ran in milliseconds: 7576
Now CPython..
0.321110010147
0.312225103378
0.312016010284
0.309517145157
0.30745100975
0.313014984131
0.312664985657
0.312491893768
0.311804056168
0.312137126923
Ran in milliseconds: 3165
As you can see, the Jep and CPython scripts are fairly stable in terms of execution speed while Jython varies a bit but gets faster (probably due to the HotSpot technology). If you want to change the test to something else, you can try "./jdk1.7.0_21/bin/java -jar usr/lib/jython-standalone-2.5.3.jar -m compileall -l ." and mytest.py will be recompiled, along with the other .py files.
Now, feel free to use the install.py script as you like, I think it's a good example of how to setup the whole thing in a specific directory.. there was some hair-pulling to get the entire build process setup so that linking and dependencies were setup the right way.
Fun to work with a mix if Python, Java and Bash scripting for a change. I think being able to use Python to script for example prototypes or even production code is a big win in terms of productivity.. weak typing has its place in rapid application development. :)
John Cook
Need a 12-digit prime?
You may have seen the joke “Enter any 12-digit prime number to continue.” I’ve seen it floating around as the punchline in several contexts.
So what do you do if you need a 12-digit prime? Here’s how to find the smallest one using Python.
>>> from sympy import nextprime >>> nextprime(10**11) 100000000003L
The function nextprime gives the smallest prime larger than its argument. (Note that the smallest 12-digit number is 1011, not 1012. Great opportunity for an off-by-one mistake.)
Optionally you can provide an addition argument to nextprime to get primes further down the list. For example, this gives the second prime larger than 1011.
>>> nextprime(10**11, 2) 100000000019L
What if you wanted the largest 12-digit prime rather than the smallest?
>>> from sympy import prevprime >>> prevprime(10**12) 999999999989L
Finally, suppose you want to know how many 12-digit primes there are. SymPy has a function primepi that returns the number of primes less than its argument. Unfortunately, it fails for large arguments. It works for arguments as big as 2**27 but throws a memory error for 2**28.
The number of primes less than n is approximately n / log n, so there are about 32 billion primes between 1011 and 1012. According to Wolfram Alpha, the exact number of 12-digit primes is 33,489,857,205. So if you try 12-digit numbers at random, your chances are about 1 in 30 of getting a prime. If you’re clever enough to just pick odd numbers, your chances go up to 1 in 15.
May 19, 2013
Python Diary
Looking for advertising proposals
As some of you may have noticed, the AdSense bar no longer exists on my blog, this is due to Google recently revoking my AdSense account, I am quite sure it is in regards to mentioning it on a page. Currently users who have an account are able to opt'd out of either being tracked by Analytics or have no ads served to them. I will be removing this feature soon, as I am planning on self-hosting ads from prospect publishers. If you have a Python or Django related project which you would like to adverse on this blog, please contact me. Having a way to fund this website will allow me to publish more quality articles and tutorials. When a new article is posted, this blog receives over 1,000 hits in that single day. These are 1,000 prospect users which use Python and maybe Django who will see your advertisement. These can also be users who are just learning Python, so books and courses are also welcome.
Currently the ad serving system has yet to be implemented, so at this time I am only asking prospect advertisers to provide me with a proposal on how the ads should be served to users and costs they might be willing to pay. For the record this blog has been online for well over a year now and receives many returning users due to the quality of the content which is provided. This website is also much more than a blog, as it has other features which bring users back for more.
Thank you for your time in reading this.
codeboje
Post to Tumblr with python
I have an art blog over at tumblr where i post my (almost) daily doodles. Usually i post them with the Tumblr UI, but lately i i got annoyed but that way and hacked something together to post images directly from my windows exlorer.
- Register an app at tumblr
- Install oauth2 and pytumblr
- Modify the "Twitter Three-legged OAuth Example" Script from python-oauth2 to use tumblr endpoints and insert your consumer key and secret
- Run the script and note the oauth token and secret the script outputs
-
My actual Poster Script (pretty less coe :-) ), add your keys and blog url here
import pytumblr import sys client = pytumblr.TumblrRestClient( '<consumer_key>', '<consumer_secret>', '<oauth_token>', '<oauth_secret>', ) client.create_photo("your blog url", state="published" , data=sys.argv[1])
-
Add Script to Windows Explorer context menu follwoing this tutorial
<path-to-python>python.exe <path_to_script>poster.py "%1"
-
Enjoy :-)
Python Diary
DVD Collection source code now available
For those who were wanting a copy of the DVD Collection software made in Python, I have now open sourced it and it is live on BitBucket!
Python Script to encode Django templates
Do you need to display raw Django template code in your Django 1.4 project? Look no further than this script! It's rather crude, but gets the job done. I haven't yet updated a few Django websites to Django 1.5, which has a new template tag to do this for you, so I created this script to use in legacy Django sites, and it works like a charm!
#!/usr/bin/python
import sys
try:
filename = sys.argv[1]
except IndexError:
print "This command needs exactly 1 parameter!"
sys.exit()
data = open(filename, 'r').read()
data = data.replace('{%', '{! templatetag openblock !}').replace('%}', '{! templatetag closeblock !}')
data = data.replace('{{', '{% templatetag openvariable %}').replace('}}', '{% templatetag closevariable %}')
print data.replace('{!', '{%').replace('!}', '%}')
You should Pygments to highlight the syntax like I do on this blog of course. If you are using Django 1.5 or greater, you should use the verbatim template tag over this.
PyCon
PyCon 2014 Begins! Call For Launch Day Sponsors
French translation: http://montrealpython.org/fr/2013/05/pycon-day-sponsors/
It's that time again! Planning for PyCon 2014 is well underway, and we're currently preparing for the launch of our new site. With the launch comes a unique opportunity: Is your organization interested in being a launch day sponsor?
PyCon 2013 launched on July 9, 2012 with 21 sponsors pledging support, leading the charge that drew over 150 organizations to pitch in to the biggest and best Python conference yet. We're planning a similar go-live date for the 2014 site, and we're building up our cadre of supporters for the April 9-17 conference taking place in Montreal, Quebec, Canada.
Your organization's support enables PyCon to do the awesome things that it does. 2013 introduced a number of new events that we've heard great feedback on, so we'd like to keep doing those things and more. For example, the inaugural Young Coders tutorial was such a hit that there are already plans around the world for user groups to replicate it, and we're looking forward to doing a PyCon 2014 rendition. Programs like Financial Aid, which saw its budget increased and then quickly doubled to reach $100,000 USD, are greatly enhanced by the giving of sponsors.
Along with benefiting the community, sponsorship of PyCon brings many benefits to its supporters. We hear it year after year that there is no better place to hire Python developers than at PyCon. We offer sponsors a place on our site to promote their open positions, and we run a job fair on-site that has been a huge success. The Expo Hall is a great place to market your latest projects and to network with 2,000 eager Python developers. The value is unparalleled in the conference scene, especially after considering our flexibility to work with each and every organization. We even offer a 50% discount to organizations under 25 people. See https://us.pycon.org/2013/sponsors/whysponsor/ for more thoughts.
While we're still finalizing the sponsorship prospectus, it will be very similar to the one we used in 2013 at https://us.pycon.org/2013/sponsors/prospectus/. We'll share the details as soon as we complete them, and any questions can be forwarded to our Sponsorship Chair, Jesse Noller.
For 2014, PyCon will have a maximum capacity of 2,000 attendees. We've sold out the last two conferences and we're expecting a third, so mark your calendars for April 9-17, 2014. Other dates to remember are our Call for Proposals in July, and we're looking forward to opening registration in September. We're planning for the conference schedule to be laid out in December, just in time for the holidays.
If you don't have a passport, don't forget that Canada requires one. US residents should see http://travel.state.gov/passport/ for details.
Holger Krekel
PEP438 is live: speed up python package installs now!
My “speed up pypi installs” PEP438 has been accepted and transition phase 1 is live: as a package maintainer you can speed up the installation for your packages for all your users now, with the click of a button: Login to https://pypi.python.org and then go to urls for each of your packages, and specify that all release files are hosted from pypi.python.org. Or add explicit download urls with an MD5. Tools such as pip or easy_install will thus avoid any slow crawling of third party sites.
Many thanks to Carl Meyer who helped me write the PEP, and Donald Stufft for implementing most of it, and Richard Jones who accepted it today! And thanks also to the distutils-sig discussion participants, in particular Phillip Eby and Marc-Andre Lemburg.
Mathieu Virbel
Kivy 1.7.0 is out
If you missed it, Kivy 1.7.0 is out last Monday. It’s quite a big release that include some big fixes we did during the :
- Android Emulator is now supported
- ScrollView has been re-factored with new physics calculations. To be exact, the scrollview doesn’t calculate anything now, it just pass the touch position to an Scroll effect. This class calculate the movement’s velocity, and the over-scroll’s distance (means how far you scrolled out of the bounds). Then, we implemented 2 visual effects that use the over-scroll to make the scrollview act as a Damped spring, and/or to fade out the scrollview if you over-scroll too much.
- Garden: a separated organization for centralize user’s widgets and addons. You can create garden packages very easily, and import it in the source code with just “kivy.garden.packagename“. Few garden packages are already available!
- And more, Check the announcement
If you like my work, tip me!
Twisted Matrix Labs
Migration Report
I have completed the migration scripts for deploying the services currently running on cube. They have been run against our new machine, dornkirk which is currently running with a snapshot of data.
It can currently be accessed by putting
66.35.39.66 twistedmatrix.com speed.twistedmatrix.com
in /etc/hosts. Please tests it, and verify that things appear to be
working, but be aware that any changes will be lost, when the
transition occurs.
At some point Monday or Tuesday, there will be some downtime for mail and the mailing lists, as mail-service is migrated to the new machine. For those that have accounts on cube, your data will be copied to the new machine at this point.
On Wednesday, at about 10 MDT (16 UTC), there will be downtime of all twisted services, as live data is transfered over. This may last up-to a couple of hours.
This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!
May 18, 2013
Dariusz Suchojad
Zato 1.0. The next generation ESB and application server. Open-source. In Python.
(This is a re-post of what I sent to python-announce@ but with several screenshots attached)
I’m very happy to announce the first release of Zato, the next generation ESB and application server, available under a commercial-friendly open-source LGPL license.
What can you expect out of the box?
- HTTP, JSON, SOAP, Redis, AMQP, JMS WebSphere MQ, ZeroMQ, FTP, SQL, hot-deployment, job scheduling, statistics, high-availability load balancing and more
- Incredible productivity with Python
- Painless rollouts with less downtime
- Slick web admin GUI, CLI and API
- Awesome documentation (several hundred A4 pages)
- 24×7 commercial support and training
Project’s site: https://zato.io
Download: https://zato.io/download/zato-1.0.tar.bz2
Support: https://zato.io/support
Docs: https://zato.io/docs
Architecture: https://zato.io/docs/architecture/overview.html
Tutorial: https://zato.io/docs/tutorial/01.html
GitHub: https://github.com/zatosource
Mailing list: https://mailman-mail5.webfaction.com/listinfo/zato-discuss
IRC: irc://irc.freenode.net/zato
Twitter: https://twitter.com/zatosource
LinkedIn: https://www.linkedin.com/groups?gid=5015554
Diversity statement: https://zato.io/docs/project/diversity.html
Spread the news and enjoy 
Cheers!
kdev-python
kdev-python 1.5.1 released
kdev-python provides Python language support for the KDevelop integrated development environment. For more information, please look at the 1.4 and 1.5 release announcements, and especially make sure to watch this short demonstration video!
The 1.5.1 release fixes a few issues with 1.5.0. Especially, it
- fixes a crash bug which could happen when PEP8 checking was enabled
- fixes capitalization in the coding line completion
- displays the correct plugin version in the About dialog
There's one thing I forgot to announce in the 1.5 release: PEP8 checking support (which was actually a good thing, since it was rather crashy until now ;)).
 |
| PEP8 checking suppot in kdev-python 1.5.1 |
By default it is disabled though, since it displays a lot of errors for people who don't follow PEP8, so if you want to use it, go to Settings -- Configure KDevelop -- PEP8 style checking (the change will only apply to newly opened or modified files).
If there's any issues, please make sure to let me know on the bug tracker.
The tarball can be found here, sha256 checksum is d9b68bd2dd9361961e264254d2acfebc9ce0ea4b47ea2689d2f01a3ed81f7c47
Vern Ceder
Another Quick Python Deal
Just a quick note that the Quick Python Book, 2nd edition will be Manning Publications deal of the day Saturday, May 18.
Here’s the official scoop:
Deal of the Day : Half off my book The Quick Python Book, Second Edition. Use code dotd0518au at http://www.manning.com/ceder/.
Also on the same code is Hello! Python (http://www.manning.com/briggs/) and Extending jQuery (http://www.manning.com/wood/)
And for anyone in Europe and the Americas, you should know that the deals usually run until past the end of day everywhere in the world. So if you don’t get it done on Saturday, the code might still work on Sunday morning. Just sayin’…
Filed under: Python
