Planet Python
Last update: January 28, 2012 03:44 AM
January 27, 2012
Shannon -jj Behrens
Dart at BayPiggies
I gave a talk on Dart last night at BayPiggies. We had a great turn out, and the crowd was super interactive! Never underestimate a Python programmer's ability to question every aspect of a new programming language, especially one that uses curly braces and semicolons ;)
If you're interested, here are the slides.
PyPy Development
Py3k and Numpy First Stage: Thanks to all who Gave
Last year was quite successful for PyPy fundraising through the Software Freedom Conservancy, and Conservancy and PyPy are very excited to announce that enough was raised to begin the first stages on the Py3k and Numpy grant proposals.
As of the end of 2011, 135 different individuals gave to the Py3k campaign, and 114 to the Numpy campaign. We thank each of you who donated to help make this work possible. Meanwhile, if you haven't given to support these projects, we do hope you'll give generously now to help fund their second stages later this year!
We're also particularly excited that a few donors gave particularly large donations to support this work; those big donations really filled in the gap to help us get started!
Specifically, we're pleased to announce that Google donated $35000 towards implementing Python 3 in PyPy. Google's general support of the Python community is well known, and their specific support of our grant proposal is much appreciated.
Meanwhile, Numpy was supported in part by contributions from Nate Lawson, Cantab Capital Partners, and Getco, as well as more than a hundred other contributors.
With these donations combined with many others, we're now starting work on both projects. This week, the Conservancy signed contracts with Antonio Cuni and Benjamin Peterson to work towards the Stage 1.1 goals in Py3k proposal (and is negotiating for another contractor as well), and with Maciej Fijałkowski to work towards the Stage 1 goals in the Numpy proposal.
In 2012, PyPy will continue regular sprint meetings, at which Py3K and Numpy efforts will certainly have a place. We have some limited funds to fund travels of contributors to those meetings.
We're very thankful for all who donated so far to support these efforts, and we hope that now that work has begun, even more donors will come forward to help us finish the job. In the meantime, watch for the commits showing up from these developers and other contributors in the PyPy repositories!
Cheers, The PyPy Team
Kenneth Reitz
How I Develop Things and Why
I've always considered myself a bit of a software junkie. Nothing excites me more than a great piece of new software.
Some of my best childhood memories are our trips to Grandma's house, where I'd have access to a computer with a dial-up connection that I'd use to obtain freeware and shareware. I'd bring 4 or 5 floppies with me and try to cram all the games, waveform editors, and utilities that I could sneaker-net home.
Luckily today, excellent software written with passion oozes out of the app ecosystem. OS X and the App Store really fuel an economy of software built for humans by people that care.
Unfortunately, this doesn't always hold true in developer software — text editors, modules, libraries, toolchains, &c. We are forced to deal with APIs on a daily basis that were not built with the user in mind. Over-engineering surrounds us as developers. Things that should be simple are often needlessly complex for the sake of being complex and "proper".
Why should consumer apps and developer APIs be treated differently?
Have an Issue
The first step to developing something great is to have a real problem. You can't solve a problem properly if you don't experience it firsthand.
On the consumer app side of things, a great example of this is Microsoft OneNote. Have you used OneNote? It's incredible.
Essentially, OneNote is a hierarchical freeform note-taking software that assumes nothing: you can type, use handwriting, embed files, cross-link notes, sync them online, &c.
Unfortunately, OneNote is only available on Windows. This kills me. I would love to think that Microsoft's would port this lovely peice of software to OS X, but I doubt it will ever happen.
If I ever decide to actually ship a consumer product, it will be something akin to OneNote for OS X. It would be incredible. It may not be for many, but for people that resonate with my problem, it will work wonderfully. It would be a reaction to a real problem, not an engineered app an entrepreneur thinks will fill a gap so he can make some fast cash.
GitHub wasn't built for the developer community at large; the founders built GitHub for themselves. The problem they solved simply happened to resonate with millions of developers because they themselves happen to be developers.
37Signals didn't build Basecamp for a world full of project managers and consultants; they built it for themselves. They also developed Ruby on Rails for themselves, as Ruby developers that were repeating themselves too often.
How pragmatic.
These companies didn't need to commission lengthy case studies and perform market analysis. They didn't setup faux AdWords to measure the effectiveness of various marketing copy. Yet, they are astronomically successful. How is this possible? They know exactly what they want to build, how it should function, and how it should look because they were building it for themselves and not for others.
Let's go back to the developer's side of things.
A great example is my Requests module. I was a heavy user of Convore at the time, and I wanted to interface with it programmatically. So, I set out to build a Python module that wrapped the Convore HTTP API. Unfortunately, this was easier said than done. Dealing with Python's standard library for HTTP was a complete and total nightmare. It was over-engineered.
I love Python because it's a language designed for Humans. Why should modern HTTP be so difficult? So, I sat out to discover what it was that I wanted, and built exactly what I needed. It resonated well with others.
Nothing is more satisfying than using your own tools to Get Things Done.
Respond with a README
Before I start writing a single line of code, I write the README and fill it with usage examples. I pretend that the module I want to build is already written and available, and I write some code with it.
This has an incredible effect: instead of engineering something that will only get the job done, you start to interact with the problem itself and build an interface that reacts to it.
You discover it. You respond to it.
Great sculptures aren't manufactured — they're discovered. The sculptor studies and listens to the slab of marble. He identifies with the stone. Then, he responds. He enables the marble to speak for itself, setting free something beautiful that hidden was inside all along.
He responds.
This is what responsive design is all about. It's not merely a method to engineer a web design that will function on a phone, tablet, and desktop.
Beware lest you lose the substance by grasping at the shadow.
Responsive design is about making a design that identifies and understands itself enough to respond to the environment it's placed in. It is about setting your design free from arbitrary constraints. It is setting free something beautiful that was inside all along.
This is known as Readme-Driven Development. I call it Responsive API Design.
Build
Now that you know what your API is: Build it. Make it happen. If there's a significant amount of complexity behind a simple call, make a layered API: a porcelain interface that sits on top of a verbose API that sits on top of an low level integration interface.
The user API is all that matters. Everything else is secondary.
Once your software is released, improve it! Add new features, better security, optimal performance, and rigidity. But never compromise the API.
Manifesto
Build things that you want. Build things that you need. Build things for you.
The Golden Rule™:
Do unto others as you would have them do to you.
Adapted to:
Build tools for others that you want to be built for you.
Eli Bendersky
pyelftools ported to Python 3
I’ve released version 0.20 of pyelftools, with support for Python 3. Now pyelftools supports Python 2.6, 2.7 and 3.2 in a single code-base. The new version is also available on PyPI.
This was surprisingly painless for a project that has close to 6KLOC of Python code, probably because I planned for eventual Python 3 support from the start, and the minimal required version of Python 2 is 2.6, which is much more compatible with Python 3 than earlier versions in the 2.x line.
If you’re curious, here’s a direct link to the py3compat module which pyelftools uses for various compatibility issues between Python 2 and 3. It’s partially based on the six library.
By far the hardest part of this port to Python 3 was porting the construct library on which pyelftools relies for the low-level binary stream parsing. I’ve forked the construct Github repository to perform the port – it’s available here. After construct was ported, it took around 2 hours to port the rest of pyelftools. construct is also being distributed with pyelftools, so there are no actual external dependencies.
No related posts.
Brett Cannon
Grab bag of tips when working with App Engine unit testing
[update: App Engine doc bug is being fixed and should be publicly visible in 1.6.2 or 1.6.3]
First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.
With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.
First thing you should do when testing an App Engine app is set up a testbed. This will allow you to stub out various parts of App Engine so your tests run in isolation without having stale state carrying over. You basically create a testbed which holds state. You then activate the system by setting some core environment variables with defaults. After that you do your own tweaks to env vars. From there you activate various stubs. And then finally you deactivate the whole thing to clean up...
But you need to realize there is an apparent error in the docs (which will be fixed no later than 1.6.3, possibly 1.6.2). The stated order of calling activate() and setup_env() are wrong; you should call activate() before calling setup_env() in order to set the default env vars before you begin to set your own.
Another thing you should know about is that setup_env() has an overwrite argument. This tells the method to overwrite any pre-existing env var; otherwise the call will skip changing any pre-existing env vars. This can call issues if your app presets some env vars that should be overridden during testing. I would honestly consider using that argument every time to make sure things work the way you expect.
And the last tip specific to setup_env()is make sure to pass all of your env var changes in a single call. Otherwise you can end up with some settings be changed as they have a default value.
In relation to handling users, to specify who the logged in user is you should set the USER_EMAIL env var. That will cause App Engine to return a user with the specified email address when calling users.get_current_user().
Random thing about email testing is that mail.send_mail_to_admins() does not write to the mail stub to verify any email was sent. This is because while testing there is no way to specify who admins are, so there is no way to know who to say the emails are sent to. So in order to test the function you will need to mock out the call yourself (personally I'm a fan of mock if you don't already have a preferred mocking library).
And finally, you can mock our the request and response objects for a handler. You can call a handler's initialize() method with the object to use -- for the request and response, respectively -- which allows you to mock and test how the handler does things.
Bonus for those of you using the Python 2.7 runtime: I would consider using unittest's discovery features to drive your unit tests. You can start from my driver code and add what you need. But it's nice to just be able to plop down a new test module and have it automatically picked up.
Ned Batchelder
Slim comparisons
Hanging out in the #python IRC channel today, I learned something new about Python comparisons. It isn't so much a new detail of the language, as a way to make use of a detail, a clever technique that I hadn't seen before.
When defining a class, it's often useful to define an equality comparison so that instances of your class can be considered equal. For example, in an object with three attributes, the typical way to define __eq__ is like this:
class Thing(object):
def __init__(self, a, b, c):
self.a = a
self.b = b
self.c = c
def __eq__(self, other):
print "Comparing %r and %r" % (self, other)
return (
self.a == other.a and
self.b == other.b and
self.c == other.c
)
When run, it shows what happens:
>>> x = Thing(1, 2, 3)
>>> y = Thing(1, 2, 3)
>>> print x == y
Comparing <Thing 37088896> and <Thing 37088952>
True
Here the __eq__ method compares the three attributes directly on the self and other objects, and returns a boolean, a simple direct comparison.
But on IRC, a different technique was proposed:
class Thing(object):
def __init__(self, a, b, c):
self.a = a
self.b = b
self.c = c
def __eq__(self, other):
print "Comparing %r and %r" % (self, other)
return (self.a, self.b, self.c) == other
Now when we run it, something unusual happens:
>>> x = Thing(1, 2, 3)
>>> y = Thing(1, 2, 3)
>>> print x == y
Comparing <Thing 37219968> and <Thing 37220024>
Comparing <Thing 37220024> and (1, 2, 3)
True
Our __eq__ is being called twice! The first time, it's called with two Thing objects, and it tries to compare a tuple of (1, 2, 3) to other, which is y, which is a Thing. Tuples don't support comparison to Thing's, so it returns NotImplemented. The == operator handles that case, and relying on the commutative nature of ==, tries swapping the two arguments. That means comparing y to (1, 2, 3), which calls our __eq__ again. Now it compares (1, 2, 3) to (1, 2, 3), which succeeds, producing the final True result.
This is an interesting technique, but I'm not sure I like it. For one thing, the code doesn't read clearly. It's comparing a tuple to an object, which isn't supported. It only makes sense when you keep in mind the argument-swapping dance.
For another, it make operations work that maybe shouldn't:
x == (1, 2, 3)
(1, 2, 3) == x
I don't know that I want these comparisons to succeed. It exposes internals that should be hidden. Of course, why would a caller who didn't know the internals try a comparison like this? But things like this have a way of creeping out to bite you.
I'm glad to have a better understanding of the workings of comparisons, but I'm not sure I'll write them like this.
January 26, 2012
Kay Hayen
Nuitka Release 0.3.19
This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things are in this.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
Bug fixes
- The manpages of “nuitka” and “nuitka-python” had no special layout for the option groups and broken whitespace for “–recurse-to” option. Also “–g++-only” was only partially bold. Released as 0.3.18.1 hotfix already.
- The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hotfix already.
- Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of “/” instead of using a “joinpath” call. Released as 0.3.18.3 already.
- A call to the range built-in with no arguments would crash the compiler, see Issue#29. Released as 0.3.18.4 already.
- Compatibility Fix: When rich comparison operators returned false value other “False”, for comparison chains, these would not be used, but “False” instead, see .
The support for “__import__” didn’t cover keyword arguments, these were simply ignored. See Issue#28. Fixed, but no warning is given yet.
New Features
- A new option has been added, one can now specify “–recurse-directory” and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements.
- Added support for the “exec” built-in of Python3, this enables us to run one more basic test, “GlobalStatement.py” with Python3. The test “ExecEval.py” nearly works now.
New Optimizations
- The no arguments “range()” call now optimized into the static CPython exception it raises.
- Parts of comparison chains with constant arguments are now optimized away.
Cleanups
- Simplified the “CPythonExpressionComparison” node, it now always has only 2 operands. If there are more, the so called comparison chain, it’s done via “and” with assignments to temporary variables, which are expressed by a new node type “CPythonExpressionTempVariableRef”. This allowed to remove “expression_temps” from code templates and generation, reducing the overall complexity.
- When executing a module (“–execute” but not “–exe”), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment.
- The calls to the variadic “MAKE_TUPLE” were replaced with calls the MAKE_TUPLExx, that are generated on a as-needed basis. This gives more readable code, because no EVAL_ORDERED_xx is needed at call site anymore.
- Many node classes have moved to new modules in “nuitka.nodes” and grouped by theme. That should make them more accessible.
- The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the “sys.abiflags” and works with Python3.
- The replacing of “.py” in filenames was made more robust. No longer is “str.replace” used, but instead proper means to assure that having “.py” as other parts of the filenames won’t be a trouble.
- Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements.
- As always, some PyLint work, and some minor TODOs were solved.
Organizational
- Added more information to the “Developer Manual”, e.g. documenting the tree changes for “assert” to become a conditional statement with a raise statement, etc.
- The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise. I expect to make a separate announcement with the links to packages soonish.
- Added support to specify the binary under test with a “NUITKA” environment, so the test framework can run with installed version of Nuitka too.
- Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for “os.execl” not propagating exit codes under Windows. See Issue#26 for more information.
- For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the “PATH” or even installed to execute the binary.
New Tests
- The “basic”, “programs”, “syntax”, and “reflected” were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations.
- Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains.
- The “Referencing.py” test was made portable to Python3.
- Cover no arguments “range()” exception as well.
- Added test to demonstrate that “–recurse-directory” actually works. This is using an “__import__” that cannot be predicted at run time (yet).
- The created source package is now tested on pbuilder chroots to be installable and capable of the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn’t break without notice.
Numbers
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
Summary
This releases contains many changes. The “temporary variable ref” and “assignment expression” work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. inplace assignments can be reduced to assignments to temporary variables.
While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with “value propagation” working.
These are the real focus. The old comparison chains were a wart. Working, but no way understood by any analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimizations to see through them.
Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of inplace assignments. With an “assignment expression” to a “temporary variable ref”, these become the same as user code using such a variable. There will be more of these to find.
So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the “__import__” related one, were not suitable for hotfix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already.
As always you will find its latest version here.
Yours,
Kay Hayen
Holger Krekel
Why privacy matters and State Secrecy not
Today i saw a woman’s german article on how she wouldn’t want reports about her sex life spread through wikileaks. I wouldn’t like that either. But i see it as a misconception and confusion about two very different issues, namely personal privacy and state/government secrecy. This confusion is used and increased by many politicians for their own purposes.
Privacy means our rights to have a private life, private flat, private actions and private communications. We don’t want the state or the public to surveil us or intrude our private world unless they can show evidence to an independent court that there is something criminal going on.
State secrecy denotes making secret deals, performing secret communication with and secret actions against people or other states. Secret actions can naturally not be discussed in the public and are exempt from our judgement when electing officials. Increasing state secrecy very quickly leads to inner circles wielding great power. The 20th century has tons of bad examples.
Therefore I refuse the notion that if am positive about privacy i must also be ok with state secrecy. Or if i want a transparent government that i also must be ok with total surveillance of my private life. No way. Likely it’s rather true that the more transparent a government is the more secure i can feel with respect to my privacy.
On a sidenote, this all relates to a point in the hacker ethics made by Wau Holland and the Chaos Computer Club a long time ago: “Make public data available, protect private data.”
PyPy Development
Comparing Partial Evaluation and Tracing, Part 1
As part of writing my PhD I am currently thinking about the relationship between PyPy's meta-tracing approach with various previous ideas to automatically get a (JIT-)compiler from only an interpreter of a language. One of the most-researched ideas along these lines is that of partial evaluation. Partial evaluation has basically the same goals as PyPy when it comes to compilers: Write an interpreter, and get a compiler for free. The methods for reaching that goal are a bit different. In this series of blog posts, I am trying to explore the similarities and differences of partial evaluation and PyPy's meta-tracing.
A Flowgraph Language
To be able to clearly understand what "partial evaluation" is and what "meta-tracing" is I will show an "executable model" of both. To that end, I am defining a small imperative language and will then show what a partial evaluator and a tracer for that language look like. All this code will be implemented in Prolog. (Any pattern-matching functional language would do, but I happen to know Prolog best. Backtracking is not used, so you can read things simply as functional programs.) In this post I will start with the definition of the language, and a partial evaluator for it. The code written in this blog post can be found fully here: http://paste.pocoo.org/show/541004/
The language is conceptionally similar to PyPy's flow graphs, but a bit more restricted. It does not have function calls, only labelled basic blocks that consist of a series of linearly executed operations, followed by a conditional or an unconditional jump. Every operation is assigning a value to a variable, which is computed by applying some operation to some arguments.
A simple program to raise x to the yth power in that language looks like this:
power:
res = 1
if y goto power_rec else goto power_done
power_rec:
res = res * x
y = y - 1
if y goto power_rec else goto power_done
power_done:
print_and_stop(res)
To represent the same program as Prolog data structures, we use the following Prolog code:
block(power, op1(res, same, const(1),
if(y, power_rec, power_done))).
block(power_rec, op2(res, mul, var(res), var(x),
op2(y, sub, var(y), const(1),
if(y, power_rec, power_done)))).
block(power_done, print_and_stop(var(res))).
Every rule of block declares one block by first giving the label of the block, followed by the code. Code is a series of op1 or op2 statements terminated by a jump, an if or a print_and_stop. op1 statements are operations with one argument of the form op1(res_variable, operation_name, argument, next_statement). Arguments can be either variables in the form var(name) or constants in the form const(value).
To run programs in this flowgraph language, we first need some helper functionality. The first few helper functions are concerned with the handling of environments, the data structures the interpreter uses to map variable names occuring in the program to the variables' current values. In Python dictionaries would be used for this purpose, but in Prolog we have to emulate these by lists of key/value pairs (not very efficient, but good enough):
lookup(X, [], _) :- throw(key_not_found(X)).
lookup(Key, [Key/Value | _], Value) :- !.
lookup(Key, [_ | Rest], Value) :- lookup(Key, Rest, Value).
write_env([], X, V, [X/V]).
write_env([Key/_ | Rest], Key, Value, [Key/Value | Rest]) :- !.
write_env([Pair | Rest], Key, Value, [Pair | NewRest]) :- write_env(Rest, Key, Value, NewRest).
remove_env([], _, []).
remove_env([Key/_ | Rest], Key, Rest) :- !.
remove_env([Pair | Rest], Key, [Pair | NewRest]) :- remove_env(Rest, Key, NewRest).
resolve(const(X), _, X).
resolve(var(X), Env, Y) :- lookup(X, Env, Y).
The implementation of these functions is not too important. The lookup function finds a key in an environment list, the write_env function adds a new key/value pair to an environment, remove_env removes a key. The resolve function is used to take either a constant or a variable and return a value. If it's a constant, the value of that constant is returned, if it's a variable it is looked up in the environment. Note how the last argument of lookup and resolve is actually a return value, which is the typical approach in Prolog.
So far we have not specified what the primitive operations that can occur in the program actually mean. For that we define a do_op function which executes primitive operations:
do_op(same, X, X).
do_op(mul, X, Y, Z) :- Z is X * Y.
do_op(add, X, Y, Z) :- Z is X + Y.
do_op(sub, X, Y, Z) :- Z is X - Y.
do_op(eq, X, Y, Z) :- X == Y -> Z = 1; Z = 0.
do_op(ge, X, Y, Z) :- X >= Y -> Z = 1; Z = 0.
do_op(readlist, L, I, X) :- nth0(I, L, X).
do_op(Op, _, _, _) :- throw(missing_op(Op)).
Again the last argument is an output variable.
Now we can start executing simple operations. For that an interp predicate is defined. It takes as its first argument the current environment and as the second argument the operation to execute. E.g. to execute primitive operations with one or two arguments:
interp(op1(ResultVar, Op, Arg, Rest), Env) :-
resolve(Arg, Env, RArg),
do_op(Op, RArg, Res),
write_env(Env, ResultVar, Res, NEnv),
interp(Rest, NEnv).
interp(op2(ResultVar, Op, Arg1, Arg2, Rest), Env) :-
resolve(Arg1, Env, RArg1),
resolve(Arg2, Env, RArg2),
do_op(Op, RArg1, RArg2, Res),
write_env(Env, ResultVar, Res, NEnv),
interp(Rest, NEnv).
First the arguments are resolved into values. Afterwards the operation is executed, and the result is written back into the environment. Then interp is called on the rest of the program. Similarly easy are the unconditional jump and print_and_stop:
interp(jump(L), Env) :-
block(L, Block),
interp(Block, Env).
interp(print_and_stop(Arg), Env) :-
resolve(Arg, Env, Val),
print(Val), nl.
In the unconditional jump we simply get the target block and continue executing that. To execute print_and_stop we resolve the argument, print the value and then are done.
The conditional jump is only slightly more difficult:
interp(if(V, L1, L2), Env) :-
lookup(V, Env, Val),
(Val == 0 ->
block(L2, Block)
;
block(L1, Block)
),
interp(Block, Env).
First the variable is looked up in the environment. If the variable is zero, execution continues at the second block, otherwise it continues at the first block.
Given this interpreter, we can execute the above example program like this, on a Prolog console:
$ swipl -s cfglang.pl
?- block(power, Block), interp(Block, [x/10, y/10]).
10000000000
Partial Evaluation of the Flowgraph Language
Let's look at what a partial evaluator for this simple flowgraph language would look like. Partial evaluation (PE), also called specialization, is a program manipuation technique. PE takes an input program and transforms it into a (hopefully) simpler and faster output program. It does this by assuming that some variables in the input program are constants. All operations that act only on such constants can be folded away. All other operations need to remain in the output program (called residual program). Thus the partial evaluator proceeds much like an interpreter, just that it cannot actually execute some operations. Also, its output is not just a value, but also list of remaining operations that could not be optimized away.
The partial evaluator cannot use normal environments, because unlike the interpreter not all variables' values are known to it. It will therefore work on partial environments, which store just the know variables. For these partial environments, some new helper functions are needed:
plookup(Key, [], var(Key)).
plookup(Key, [Key/Value | _], const(Value)) :- !.
plookup(Key, [_ | Rest], Value) :- plookup(Key, Rest, Value).
presolve(const(X), _, const(X)).
presolve(var(V), PEnv, X) :- plookup(V, PEnv, X).
The function plookup takes a variable and a partial environment and returns either const(Value) if the variable is found in the partial environment or var(Key) if it is not. Equivalently, presolve is like resolve, except that it uses plookup instead of lookup.
With these helpers we can start writing a partial evaluator. The following two rules are where the main optimization in the form of constant folding happens. The idea is that when the partial evaluator sees an operation that involves only constant arguments, it can constant-fold the operation, otherwise it can't:
pe(op1(ResultVar, Op, Arg, Rest), PEnv, NewOp) :-
presolve(Arg, PEnv, RArg),
(RArg = const(C) ->
do_op(Op, C, Res),
write_env(PEnv, ResultVar, Res, NEnv),
RestResidual = NewOp
;
remove_env(PEnv, ResultVar, NEnv),
NewOp = op1(ResultVar, Op, RArg, RestResidual)
),
pe(Rest, NEnv, RestResidual).
pe(op2(ResultVar, Op, Arg1, Arg2, Rest), PEnv, NewOp) :-
presolve(Arg1, PEnv, RArg1),
presolve(Arg2, PEnv, RArg2),
(RArg1 = const(C1), RArg2 = const(C2) ->
do_op(Op, C1, C2, Res),
write_env(PEnv, ResultVar, Res, NEnv),
RestResidual = NewOp
;
remove_env(PEnv, ResultVar, NEnv),
NewOp = op2(ResultVar, Op, RArg1, RArg2, RestResidual)
),
pe(Rest, NEnv, RestResidual).
The pe predicate takes a partial environment, the current operations and potentially returns a new operation. To partially evaluate a simple operation, its arguments are looked up in the partial environment. If all the arguments are constants, the operation can be executed, and no new operation is produced. Otherwise, we need to produce a new residual operation which is exactly like the one currently looked at. Also, the result variable needs to be removed from the partial environment, because it was just overwritten by an unknown value.
The potentially generated residual operation is stored into the output argument NewOp. The output argument of the recursive call is the last argument of the newly created residual operation, which will then be filled by the recursive call. This is a typical approach in Prolog, but may look strange if you are not familiar with it.
Note how the first case of these two rules is just like interpretation. The second case doesn't really do anything, it just produces a residual operation. This relationship between normal evaluation and partial evaluation is very typical.
The unconditional jump and print_and_stop are not much more complex:
pe(jump(L), PEnv, jump(LR)) :-
do_pe(L, PEnv, LR).
pe(print_and_stop(Arg), Env, print_and_stop(RArg)) :-
presolve(Arg, Env, RArg).
To partially evaluate an unconditional jump we again produce a jump. The target label of that residual jump is computed by asking the partial evaluator to produce residual code for the label L with the given partial environment. print_and_stop is simply turned into a print_and_stop. We will see the code for do_pe soon.
Conditional jumps are more interesting:
pe(if(V, L1, L2), PEnv, NewOp) :-
plookup(V, PEnv, Val),
(Val = const(C) ->
(C = 0 ->
L = L2
;
L = L1
),
do_pe(L, PEnv, LR),
NewOp = jump(LR)
;
do_pe(L1, PEnv, L1R),
do_pe(L2, PEnv, L2R),
NewOp = if(V, L1R, L2R)
).
First we look up the value of the condition variable. If it is a constant, we can produce better code, because we know statically that only one path is reachable. Thus we produce code for that path, and then emit an unconditional jump there. If the condition variable is not known at partial evaluation time, we need to partially evaluate both paths and produce a conditional jump in the residual code.
This rule is the one that causes the partial evaluator to potentially do much more work than the interpreter, because after an if sometimes both paths need to be explored. In the worst case this process never stops, so a real partial evaluator would need to ensure somehow that it terminates. There are many algorithms for doing that, but I will ignore this problem here.
Now we need to understand what the do_pe predicate is doing. Its most important task is to make sure that we don't do the same work twice by memoizing code that was already partially evaluated in the past. For that it keeps a mapping of Label, Partial Environment to Label of the residual code:
do_pe(L, PEnv, LR) :-
(code_cache(L, PEnv, LR) ->
true
;
gensym(L, LR),
assert(code_cache(L, PEnv, LR)),
block(L, Code),
pe(Code, PEnv, Residual),
assert(block(LR, Residual))
).
If the code cache indicates that label L was already partially evaluated with partial environment PEnv, then the previous residual code label LPrevious is returned. Otherwise, a new label is generated with gensym, the code cache is informed of that new label with assert, then the block is partially evaluated and the residual code is added to the database.
For those who know partial evaluation terminology: This partial evaluator is a polyvariant online partial evaluator. "Polyvariant" means that for every label, several specialized version of the block can be generated. "Online" means that no preprocessing is done before the partial evaluator runs.
Partial Evaluation Example
With this code we can look at the classical example of partial evaluation (it's probably the "Hello World" of partial evaluation). We can ask the partial evaluator to compute a power function, where the exponent y is a fixed number, e.g. 5, and the base x is unknown:
?- do_pe(power, [y/5], LR).
LR = power1.
To find out which code was produced, we can use listing:
?- listing(code_cache)
code_cache(power, [y/5], power1).
code_cache(power_rec, [y/5, res/1], power_rec1).
code_cache(power_rec, [y/4], power_rec2).
code_cache(power_rec, [y/3], power_rec3).
code_cache(power_rec, [y/2], power_rec4).
code_cache(power_rec, [y/1], power_rec5).
code_cache(power_done, [y/0], power_done1).
?- listing(block)
.... the block definition of the user program ....
block(power_done1, print_and_stop(var(res))).
block(power_rec5, op2(res, mul, var(res), var(x), jump(power_done1))).
block(power_rec4, op2(res, mul, var(res), var(x), jump(power_rec5))).
block(power_rec3, op2(res, mul, var(res), var(x), jump(power_rec4))).
block(power_rec2, op2(res, mul, var(res), var(x), jump(power_rec3))).
block(power_rec1, op2(res, mul, const(1), var(x), jump(power_rec2))).
block(power1, jump(power_rec1)).
The code_cache tells which residual labels correspond to which original labels under which partial environments. Thus, power1 contains the code of power under the assumption that y is 5. Looking at the block listing, the label power1 corresponds to code that simply multiplies res by x five times without using the variable x at all. The loop that was present in the original program has been fully unrolled, the loop variable y has disappeared. Hopefully this is faster than the original program.
Conclusion
In this blog post we saw an interpreter for a simple flow graph language in Prolog, together with a partial evaluator for it. The partial evaluator essentially duplicates every rule of the interpreter. If all the arguments of the current operation are known, it acts like the interpreter, otherwise it simply copies the operation into the residual code.
Partial evaluation can be used for a variety of applications, but the most commonly cited one is that of applying it to an interpreter. To do that, the program that the interpreter runs is assumed to be constant by the partial evaluator. Thus a specialized version of the interpreter is produced that does not use the input program at all. That residual code can be seen as a compiled version of the input program.
In the next blog post in this series we will look at writing a simple tracer for the same flowgraph language.
Calvin Spealman
A little trick for wide pages
We have wide monitors and our reading doesn't tend to like wide text very well. This is why newspapers have lots of narrow columns, rather than stretch each story across the entire width of the paper.
Not all websites follow this tip, so drag this to your bookmark toolbar and squeeze the margin in 100px at a time, until you can read more naturally.
Yes, I could resize my window, but the same width isn't right for all pages, and most are padded or have sidebars. This is good when you only need some of the pages you have open to be narrower than the rest.
Ondřej Čertík
When double precision is not enough
I was doing some finite element (FE) calculation and I needed the sum of the lowest 7 eigenvalues of a symmetric matrix (that comes from the FE assembly) to converge to at least 1e-8 accuracy (so that I can check calculation done by some other solver of mine, that calculates the same but doesn't use FE). In reality I wanted the rounded value to 8 decimal digits to be correct, so I really needed 1e-9 accuracy (but it's ok if it is let's say 2e-9, but not ok if it is 9e-9). With my FE solver, I couldn't get it to converge more than to roughly 5e-7 no matter how hard I tried. Now what?
When doing the convergence, I take a good mesh and keep increasing "p" (the polynomial order) until it converges. For my particular problem, it is fully converged for about p=25 (the solver supports the order up to 64). Increasing "p" further will not increase the accuracy anymore, and the accuracy stays at the level 5e-7 for the sum of the lowest 7 eigenvalues. For optimal meshes, it converges at p=25, for not optimal meshes, it converges for higher "p", but in all cases, it doesn't get below 5e-7.
I know from experience, that for simpler problems, the FE solver can easily converge to 1e-10 or more using double precision. So I know it is doable, now the question is what the problem is: there
are a few possible reasons:
- The FE quadrature is not accurate enough
- The condition number of the matrix is high, thus LAPACK doesn't return very accurate eigenvalues
- Bug in the assembly/solver (like single/double corruption in Fortran, or some other subtle bug)
Obviously by making the matrix less ill conditioned, which is caused by the mesh for the problem (the ratio of the longest/shortest elements is 1e9) but for my problem I really needed such a mesh. So the other option is to increase the real number accuracy.
In Fortran all real variables are defined as real(dp), where dp is an integer defined at a single place in the project. There are several ways to define it, but it's value is 8 for gfortran and it means double precision.So I increased it to 16 (quadruple precision), recompiled. Now the whole program calculates in quadruple precision (more than 30 significant digits). I had to recompile LAPACK using the "-fdefault-real-8" gfortran option, that promotes all double precision numbers to quadruple precision, and I used the "d" versions (double precision, now promoted to quadruple) of LAPACK routines.
I rerun the calculation ---- and suddenly LAPACK residuals are around 1e-13, and the solver converges to 1e-10 easily (for the sum of the lowest 7 eigenvalues). Problem solved.
Turning my Fortran program to quadruple precision is as easy as changing one variable and recompiling. Turning LAPACK to quadruple precision is easy with a single gfortran flag (LAPACK uses the old f77 syntax for double precision, if it used real(dp), then I would simply change it as for my program). The whole calculation got at least 10x slower with quadruple. The reason is that gfortran runtime uses the libquadmath library, that simulates quadruple precision (as current CPUs only support double precision natively).
I actually discovered a few bugs in my program (typically some constants in older code didn't use the "dp" syntax, but had the double precision hardwired). Fortran warns about all such cases, when the real variables have incompatible precision.
It is amazing how easy it is to work with different precision in Fortran (literally just one change and recompile). How could this be done with C++? This wikipedia page suggests, that "long double" is only 80bit in most cases (quadruple is 128bit), but gcc offers __float128, so it seems I would have to manually change all "double" to "__float128" in the whole C++ program (this could be done with a single "sed" command).
Brett Cannon
Asynchronous XML-RPC in Python
Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)
In my case I needed to communicate with an XML-RPC server that supported multi-calls, but in an asynchronous fashion with a non-standard communication object (i.e. not sockets but App Engine's urlfetch). So I had to piece together how to make a XML-RPC client call that used multi-calls so that I could do it asynchronously myself.
The first thing to know is that a multi-call is a system.multicall function call to the server. It takes a sequence of arguments that specify what each individual function call should be. Each sequence item contains a dict with two keys: "methodName" and "params". The "methodName" key has a string value of the function name to call on the XML-RPC server. The "params" key holds the tuple of arguments (and it must be the tuple type and you must use a tuple even if it is just a single argument --think of what it would take for the call to work with *args) to pass to the function named in "methodName".
Knowing what function you are calling on the other end (system.multicall) and having a sequence of dicts specifying what you are calling in hand, you can make your call to the server. First you need to encode everything into XML: xmlrpc.client.dumps((seq_of_args,), 'system.multicall') (notice how I passed the sequence of dicts in a tuple -- and I mean a tuple, not a list). With the XML now in hand, you make the actual call to the server by making a POST with the XML as the body and set the Content-Type header to text/xml using whatever asynchronous mechanism you want to use to make the call.
What you get back from the server is XML that you can load with xmlrpc.client.loads(data, use_datetime=True)(chances are you will want the [0][0] value as that contains a sequence of results from each function call you requested). Do realize, though, that there is no indication of what function returned what, so you will need to correlate the index of the return value with what call you made (zip() makes this trivial).
And that's how you construct an XML-RPC multi-call from scratch in Python so you can make an asynchronous call. If you want to see an example of all this, see this code.
Simeon Franklin
Announcing the Modesto Scripting Language Meetup
Announcing the Modesto Scripting Languages Meetup. Central Valley Pythonistas, Rubyists, PHPers, etc all welcome.
Read MoreKay Hayen
Static Site Generator
Hello,
somehow triggered by reading about Mynth, and the re-post of the last release announcement on my feed, just because I added a missing category to the post, I am wondering what it takes to achieve the following:
- Edit the pages site as ReST (reStructured text) document with links inside
- Edit posts as single ReST files
- Have it look decent and provide feeds, proper caching headers, etc.
- Generate a static site from with, with more control over what kind of updates makes it into the feed.
I am currently very annoyed, because:
- I would normally generate code examples now with “rst2man” and replace the existing WordPress plugin that renders in JavaScript. Except I cannot without re-posting all these articles, or removing the “python” tag from them.
- The documentation of Nuitka is in ReST and I even intend to expand it with UML diagrams generated from text. Making the cross to HTML content is difficult.
- Some pages only replicate content from the User or Developer Manual, keeping those updated, in sync, etc. is a dull thing to do.
- Extending WordPress in Python is not feasible. And in php, I doubt I want to master.
Other stuff in favor of it, is that WordPress or any CMS needs more memory and more time to process than static pages of course. I can’t right now let anyone fork and edit the pages in a practical manner. So while for user manual and developer manual of Nuitka I will get corrections, for the website it’s not happening.
Is Mynth an appropiate intermediate step? Do I really want to learn one of these strange templating languages or even consider software that has no Debian package. I never considered “virtualenv” and “pip” much of options to run software on a website. I guess I am more tied to Debian than to Python still.
Over the last months I have occasionally looked at Pyramids, but it seems overly complex, if all you want is to simplify the authoring. I have seen that e.g. Trac supports ReST as an exception, but that feels too little for a hefty software like that.
I may end up with using ReST to generate HTML which is then uploaded via XMLRPC to WordPress. Won’t give me the “static” benefits. Won’t solve the post update problem. But at least will let me edit more easily.
Yours,
Kay
January 25, 2012
Kay Hayen
Nuitka Release 0.3.18
This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via “apt-get install nuitka”.
Please see the page “What is Nuitka?” for clarification of what it is now and what it wants to be.
The release cycle was too short to have much focus. It merely includes fixes, which were available as hotfixes, and some additional optimizations and node tree cleanups, as well as source cleanups. But
Bug fixes
- Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. Issue#16. Released as 0.3.17a hotfix already.
- Nuitka was considering directories to contain packages that had no “__init__.py” which could lead to errors when it couldn’t find the package later in the compilation process. Released as 0.3.17a hotfix already.
- When providing “locals()” to “exec” statements, this was not making the “locals()” writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn’t consider the actual value “locals()”. Released as 0.3.17b hotfix already.
- Compatibility Fix: When no defaults are given, CPython uses “None” for “func.func_defaults”, but Nuitka had been using “None”.
New Optimizations
- If the condition of assert statements can be predicted, these are now optimized in a static raise or removed.
- For built-in name references, there is now dedicated code to look them up, that doesn’t check the module level at all. Currently these are used in only a few cases though.
- Cleaner code is generated for the simple case of “print” statements. This is not only faster code, it’s also more readable.
Cleanups
- Removed the “CPythonStatementAssert” node. It’s not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with “AssertionError” and the assertion argument. This allowed to remove code and complexity from the subsequent stetps of Nuitka, and enabled existing optimizations to work on assert statements as well.
- Moved builtin exception names and builtin names to a new module “nuitka.Builtins” instead of having in other places. This was previously a bit spread-out and misplaced.
- Added cumulative “tags” to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks.
- Enhanced the “visitor” interface to provide more kinds of callbacks, enhanced the way “each scope” visiting is achieved by generalizing is as “child has not tag ‘closure_taker’” and that for every “node that has tag ‘closure_taker’”.
- Moved “SyntaxHighlighting” module to “nuitka.gui” package where it belongs.
- More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that should be white listed.
- Consistently use “nuitka” in test scripts, as there isn’t a “Nuitka.py” on all platforms. The later is scheduled for removal.
- New node for builtin name loopups, which allowed to remove tricks played with adding module variable lookups for “staticmethod” when adding them for “__new__” or module variable lookups for “str” when predicting the result of “type( ‘a’ )”, which was unlikely to cause a problem, but an important TODO item still.
Organizational
- The “Download” page is now finally updated for releases automatically. This closes Issue#7 completely. Up to this release, I had to manually edit that page, but I now mastered the art of upload via XMLRCP and a Python script, so that I don’t loose as much time with editing, checking it, etc.
- Added a “Developer Manual” to the release. It’s incomplete, but it details some of the existing stuff, coding rules, plans for “type inference”, etc.
- The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages soonish.
- Made sure the test runners worth with bare “python2.6″ as well.
New Tests
- Added some tests intended for type inference development.
Numbers
There are no new numbers. Nuitka should be as fast as it was, 258% speedup:
python 2.6:
Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second
Nuitka 0.3.11 (driven by python 2.6):
Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second
Summary
This releases contains not as much changes as others. It’s going to be the stable one for a while. In the mean time, we are going to work on the branch “feature/ctypes_annotation” and rebasing it often while things are still flowing.
The “exec” fix was detected by continued work on the branch “feature/minimize_CPython26_tests_diff” branch, but that work is now complete.
It is being made pretty (many git rebases) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made.
But of course, the real excitement is the the “type inference” work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient. So when I learn something. e.g. “assert” is not special, I apply it to the “develop” branch immediately, to keep the differences as small as possible.
Overall, this is a great day. I would appreciate it, if you give Nuitka a whirl and let me know how it works for you.
As always you will find its latest version here.
Yours,
Kay Hayen
United Coders
facebook Hacker Cup 2012 Qualification - solution for billboards
The second problem from this year facebook hackercup sound like the classic knapsack problem. But the solution is much easier (words are in fixed sort order). Feel free to find a smarter algorithm or comment my python solution.
problem description
We are starting preparations for Hacker Cup 2013 really early. Our first step is to prepare billboards to advertise the contest. We have text for hundreds of billboards, but we need your help to design them.Read more
PyCharm
PyCharm 2.0.2 Release Candidate
It’s been more than a month since the previous PyCharm update, and although the team is working actively on PyCharm 2.1, we’ve got a bunch of bugfixes for 2.0 as well, both in the platform and in the PyCharm specific code. To make sure we’ve got everything right, we’re releasing the update as Early Access Preview first, and if no major problems are discovered, will release a production 2.0.2 update next week.
So please download PyCharm 2.0.2 Release Candidate, check out the release notes, and let us know if anything goes wrong.
Daniel Greenfeld
How to justify attending PyCon sprints
Sad that the PyCon sprints fall on business days? Wishing you could stay but the boss/client won't let you and demands you back so 'you can work'? This is how you make it so that the sprints are something your management is demanding you attend every sprint ever.
- Make it foremost in your mind that the wonderful thing about the PyCon sprints is that the odds are that anyone who knows anything about whatever you are doing in Python will be there.
- Write up a list of the things that you are finding challenging, hard, or impossible to do with Python.
- Now go to the boss and say something like:
"Because the experts and leaders of the open source tools we are using are going to be there, I want to attend PyCon sprints. All my time at the sprints will be focused on sitting around them and working on our tools. I'll focus on things that directly impact our agency / company / organization, specifically things I wrote down on this list."
- If the boss says,
"Why not just use IRC or email?"
Then you say something like,"Well, IRC/email is not the same as sitting next to these people. I'll be so much more productive there!"
Rinse and repeat.
Then, when you attend the PyCon sprints, follow through on what you said you were going to do. Sure, it might be more fun to work on project 'spam' even though your company uses project 'lumberjack', but if you prove how much you got done during the sprints, next year the boss will be much more encouraging. Even if a good boss says to go do what you want, at least spend some time sprinting on work related technology.
Also, once you get approval to go, consider sending your boss to this old rant of mine.
Don't forget to register for PyCon! Early bird rates end today (January 25th, 2012) which means today is the last day to get involved in the extremely unofficial PyCon Early Birds program!
ShiningPanda
Selenium, Python and Jenkins on Debian - 1/3
This series is dedicated to the use of Selenium with Python. Selenium is a convenient way to automate your web application for testing purposes.
In this first post, we'll see how to configure your continuous integration environment to run Selenium tests. In a second one, how to add Selenium tests to your web project, and finally how to integrate all this within Jenkins.
This post assumes a Debian GNU/Linux environment, but you should not have much trouble adapting it to another Linux distribution.
Browsers installation
Selenium requires at least one browser in order to run tests. The following three are a good start:
# Opera: see http://wiki.debian.org/Opera (replace squeeze accordingly)Then run:
deb http://deb.opera.com/opera squeeze non-free
# Chrome
deb http://dl.google.com/linux/chrome/deb/ stable main
$ aptitude updateAdd the keys to trust the new repositories:
$ wget -q -O - http://deb.opera.com/archive.key | apt-key add -Finally re-update and install the packages:
$ wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
$ aptitude updateSelenium also requires ChromeDriver to be able to interact with Chrome. Download chromedriver_linux64_X.zip or chromedriver_linux32_X.zip here (depending on your system). This zip contains an executable named chromedriver which should be in the PATH. Copy it in /usr/local/bin for example.
$ aptitude install iceweasel opera google-chrome
Display
If you are using slaves, it is likely that your Jenkins will run headlessly. If this is the case, you will need to provide a display when executing tests: this role is filled by vnc4server. To install it, just type:
$ aptitude install vnc4server imagemagickThen, on the account running Jenkins, execute the following command to avoid prompts (password must be at least 6 characters):
$ vnc4passwdPassword:Your environment is now ready. In our next blog post, we'll see how to write Selenium tests for a Python web project based on Django.
Verify:
Roberto Alsina
People doing useful stuff with my toys
About a year ago, I wrote a small web browser, called De Vicenzo just for fun.
But hey, someone went and madeit useful for something! Specifically, to provide previews when doing sphix docs
That's cool :)
Daniel Greenfeld
Join the PyCon Early Birds program!
First off, I want to say that me and my fiancee will be attending PyCon US this year! Hooray! Can't wait to see old friends and make new ones. I'll be chairing one of the Panels at the PyWeb Summit on March 8th. We're absolutely delighted to see all the great talks, hang out in the hallway, and just be in the middle of Python for well over a week.
Now on to the extremely unofficial PyCon Early Birds program!
PyCon early registration ends on January 25th. If you register at the early bird rate that gets you the benefit of joining the elite PyCon Early Birds group. Being a member of the PyCon Early Birds gets you all sorts of incredible rewards and benefits.
- Most importantly, you get some serious bragging rights.
- A custom ribbon that says 'Early Bird' that you get to attach to your conference badge.
- A discounted rate from the regular ticket rate as according to the registration page.
- The confidence of knowing you have a ticket before they sell out.
- A tasty and rather edible store-bought cookie provided by myself and Audrey Roy.
- If the PyCon Early Birds program gets enough members, I'm going to challenge PyCon chair Jesse Noller to stump me with Yoga poses! There's no way he'll even consider accepting a challenge like this unless the PyCon Early Birds membership roster is big enough. So join and help me find out if his Bikram will beat my Capoeira!
- Other incredible things that are in the works!
Of course, PyCon has tons of other reasons to sign up besides the PyCon Early Birds program. Amazing tutorials, talks, and sprints, plus great hallway tracks, a vendor room filled with great schwag, poster sessions, and startup row. Sponsorship levels are unbelievably high, and since the event is non-profit that means the money just goes right back into the community - starting with PyCon itself. This year is going to be AWESOME!!!
So what are you waiting for? Sign up for the Pycon Early Birds before it's too late!
Senthil Kumaran
Generators in Python and Icon
I was glancing through what's new in 2.2 and found an interesting historical titbit about generators in Python.
The idea of generators comes from other programming languages, especially Icon (http://www.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from “An Overview of the Icon Programming Language” at http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like.
sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)
In Icon the find() function returns the indexes at which the substring “or” is found: 3, 23, 33. In the if statement, i is first assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon retries it with the second value of 23. 23 is greater than 5, so the comparison now succeeds, and the code prints the value 23 to the screen.
It further goes on to say that Generators are not central concept in Python, as it is for Icon, but has gotten new properties like you can store the state in a iterator object and pass around that.
Alex Clark
Hello Plone theming
In this post, we introduce a new tutorial aimed at bridging the gap between the incredibly easy to use and enormously powerful Diazo theming engine for Plone, and bootstrapping your first new theme. You do not need to have any Python skills whatsoever to perform these tasks.
New Plone themes are:
- HTML/CSS/JavaScript/images (Sound familiar? You don’t need to know any Plone to understand this part.)
- A set of rules that map your HTML/CSS/JavaScript/images to Plone content. (This part is Diazo-specific; and while Diazo is not part of Plone, integration is provided by a Python package called plone.app.theming.)
The Diazo rules can get complex, but it’s very easy to get your first rule done and working and then off you go. You may find this new tutorial on pythonpackages.com. Enjoy, and get busy theming Plone.
Kenneth Reitz
mynt - Python-powered static site generator
mynt - Python-powered static site generator:
There’s no shortage of static site generators lately, but most are very blog-focused. Mynt from Andrew Fricke aims to deliver true CMS flexibility in a Python package. Mynt supports custom date and tag-based archives, Jinja templates, Markdown posts, and YAML front matter.
Check out the nicely done docs site for usage and options, or drop into #mynt on IRC.
January 24, 2012
Rob Galanakis
Large initializers/ctors?
With closures (and to some extent with runtime attribute assignments), I find the signatures of my UI types shrink and shrink. A lot of times we have code like this (python, but the same would apply to C#):
class FooControl(Control):
def __init__(self, value):
super(FooControl).__init__()
self.value = value
self._InitButtons()def _InitButtons(self):
self.button = Button('Press Me!')
btn.clicked.addListener(self._OnButtonClick)def _OnButtonClick(self):
print id(self.button), self.value
However we can easily rewrite this like so:
class FooControl(Control):
def __init__(self, value):
super(FooControl).__init__()
btn = Button('Press Me!')
def onClick():
print value
btn.clicked.addListener(onClick)
Now this is a trivial example. But I find that many types, UI types in particular, can have most or all of these callback methods (like self._OnButtonClick) removed by turning them into inner functions. And then as you turn them into inner functions in init, you can get rid of stored state (self.value and self.button).
But as we take this to the extreme, we end up with very simple classes (and in fact I could replace FooControl with a function, it doesn’t need to be a class at all), but very long init methods (imagine doing all your sub-control creation, layout, AND all callback functionality, inside of one method!).
I’ve decided I’d rather have a long init method, usually broken up into several inner functions, rather than a larger signature on the class with layout, callbacks, and stored state. In my mind, it is easier to pull something out into a type attribute, rather than remove it, as anything on the type is liable to be used externally. And breaking up your layout into instance methods that can really only be called once (_InitButtons), from the init, adds a cognitive burden for me.
So I can justify this decision to eliminate extra attributes rationally, but what seals the deal is, I’m not unit testing any of this code anyway. So whether it is in one long method, or broken up into several methods, it isn’t getting tested.
I started out as very much in the ‘break into small methods’ camp but have wholesale moved into the ‘one giant __init__ with inner functions’ camp. I’m curious what you all prefer and why?