skip to navigation
skip to content

Planet Python

Last update: April 23, 2014 07:47 PM

April 23, 2014


End Point

Dictionary Comprehensions in Python

Python has many features which usually stay unknown to many programmers. This time I discovered the dictionary comprehensions. One of the nice features which can make code much cleaner. Let me start with a simple introduction about list comprehensions and collection uniqueness

List Comprehensions

List comprehensions are a much simpler way of creating lists. This is one feature, rather widely used, and I saw this in many examples and source of many libraries.

Imagine you have a function which returns a list of data. A good example of this is xrange(start, end) function, which returns all numbers within the range [start, end). This is a generator, so it doesn't return all numbers at once, but you need to call this function many times, and each time it returns the next number.

Getting all numbers from the range [1, 10] using this function can be done like this:

numbers = []
for i in xrange(1, 11):
    numbers.append(i)

If you want to get only the even numbers, then you can write:

numbers = []
for i in xrange(1, 11):
    if i % 2 == 0:
        numbers.append(i)

List comprehensions can make the code much simpler. The below expression evaluates to a list:

[ expression for item in list if conditional ]

The first example can be then written as:

numbers = [i for i in xrange(1, 11)]

and the second:

numbers = [i for i in xrange(1, 11) if i % 2 == 0]

Of course this syntax can be a little bit strange at the very first moment, but you can get used to it, and then the code can be much simpler.

Removing Duplicates

Another common usage of collections is to remove duplicates. And again there are plenty of ways to do it.

Consider a collection like this:

numbers = [i for i in xrange(1,11)] + [i for i in xrange(1,6)]

The most complicated way of removing duplicates I've ever seen was:

unique_numbers = []
for n in numbers:
    if n not in unique_numbers:
        unique_numbers.append(n)

Of course it works, but there a much easier way. You can use a standard type like set. Set cannot have duplicates, so when converting a list to a set, all duplicates are removed. However at the end there will be set, not list, if you want to have a list, then you should convert it again:

unique_numbers = list(set(numbers))

Removing Object Duplicates

With objects, or dictionaries, the situation is a little bit different. You can have a list of dictionaries, where you use just one field for identity, this can look like:

data = [
  {'id': 10, 'data': '...'},
  {'id': 11, 'data': '...'},
  {'id': 12, 'data': '...'},
  {'id': 10, 'data': '...'},
  {'id': 11, 'data': '...'},
]

Removing duplicates, again, can be done using more or less code. Less is better, of course. With more code it can be:

unique_data = []
for d in data:
    data_exists = False
    for ud in unique_data:
        if ud['id'] == d['id']:
          data_exists = True
          break
    if not data_exists:
        unique_data.append(d)

And this can be done using a thing I discoverd a couple of days ago, this is the dictionary comprehension. It has a similar syntax to the list comprehension, however it evaluates to a dicionary:

{ key:value for item in list if conditional }

This can be used to make a list without all duplicates using a custom field a identity:

{ d['id']:d for d in data }.values()

The above code creates a dictionary with a key, which is the field I want to use for uniqueness, and the whole dictionary as the value. The dictionary then contains only one entry for each key. The values() function is used to get a list only with values, as I don't need the key:value mappings any more.

April 23, 2014 01:37 PM


Gael Varoquaux

Google summer of code projects for scikit-learn

I’d like to welcome the four students that were accepted for the GSoC this year:

Welcome to all of you. Your submissions were excellent, and you demonstrated a good will to integrate in the project, with its social and coding dynamics. It is a privilege to work with you.

I’d also like to thank all the mentors, Alex, Arnaud, Daniel, James, Jaidev, Olivier, Robert and Vlad. It is a lot of work to mentor and mentors are not only making it possible for great code to enter scikit-learn, but also shaping a future generation of scikit-learn contributors.

April 23, 2014 05:56 AM


Europython

Financial assistance, partner programme, diversity and schedule

Financial Assistance Programme

In the second round of the EuroPython Financial Assistance Programme we granted an additional sum of 3000 EUR for April which results in a total sum  of 8000 EUR so far. There will be a third last round in May. So if you are seeking for financial support for coming to the EuroPython 2014 in Berlin you can still apply for the last round. 

We also started the sale of Financial Assistance support tickets last month. You can buy one or more Financial Assistance support tickets for a value of 10, 20, 50 or 100 Euro. With the purchase of  Financial Assistance tickets you will support further Python enthusiasts to go to EuroPython 2014 in case they need financial support for a ticket or travel expenses. The EuroPython organization team granted already 5.000 Euro last month in the first of three  rounds. Please show your appreciation for the Python community and help us to bring more people to the conference which could not make it otherwise.

Partner programme

We are currently working on the details for a partner & family programme in case you want to bring your better half or your whole family to Berlin. Berlin offers a lot of options for spending the time outside the conference programme like visits to museums, boat trips or guided trips to the Berliner “Unterwelten” and many other places of interest. A survey will be announced and published shortly in order to involve you and your partner into the arrangement and selection of the partner programme.

Diversity at EuroPython

There has been a heated discussion over the last week about the representation of women as speakers at EuroPython 2014 and about diversity in general. You can find the official thread here and the official statement of the EuroPython organizers here. Starting point of the discussion was that PyCon 2014 in Montreal reached an outstanding attendance of about  30% women (both as speaker and attendees).

The EuroPython organizers are committed to diversity by gender, sexual orientation and location. Besides the existing Code of Conduct we are working on an explicit Diversity Statement in order to make the EuroPython 2014 enjoyable for everyone. Making EuroPython more diverse is a long-time process that involves the EuroPython Society, the local organizers and the European Python community as a whole. Diversity and outreach can not be dictated but must be lived in reality on all levels.

Schedule

Therehas been some confusion related to the published preliminary schedule (also under the aspects of diversity). The complete schedule will be announced shortly with roughly additional 50 talks and all trainings. The programme team is currently heavily working on the final version that will also take more diversity aspects into account for choosing the talks. We apologize for the confusion - specially to all persons that submitted a proposal keeping them in an uninformed state. The problem was caused by some internal miscommunication but also by the desire to publish a resilient schedule as early as possible in order to let the speakers make their travel arrangements in time.

April 23, 2014 05:39 AM


Eko S. Wibowo

Reviving Microsoft Agent using PyWin32 - Run the Agent Hourly

 
Now the agent can run in specified time interval

Now the agent can run in a specified time interval

If you inspect closely our last article, you will realize that the Agent although having the ability to speak the time when you need it, it still unable to speak the time in specified interval, say, hourly. In this part of the article we are going to inspect whether we can easily add that ability.

Let's find out!

Quick Note: COM Threading Models

Long story short: Microsoft Agent is COM Object having Single Threading Apartment Model. You won't be able using MS Agent instance that have been instantiated in a different thread. For example, consider our current Agent application in previous article. As the Agent was instantiated in the main thread, seen below, 

1
2
3
4
if __name__ == '__main__':
    ...
    agent = MsAgent()
    ...

You won't be able to use this agent object outside of this main thread. And why does it important? Because I want to display this Agent in a certain time interval (hourly, to be exact), meaning it must be run by a Python thread. The simple solution would be to instantiate and use this Agent entirely in another Thread, not separate its instantiation in main thread and use it in different thread. That's not going to work with COM Object having Single Threading Apartment Model as this MS Agent things.

How to run Python code in certain time interval?

To run Python code in certain time interval, you can easily use Python Timer class from threading module. The function below will run a thread that will wake up at exactly +1 o'clock from current time. 

1
2
3
4
5
6
7
8
9
10
    def wakeup_next_hour(self):
        '''
        Run a thread that will wake up exactly n-o'clock
        '''
        now = datetime.datetime.now()
        next_hour = now + datetime.timedelta(hours = 1)
        next_hour_oclock = datetime.datetime(next_hour.year, next_hour.month, next_hour.day, next_hour.hour, 0, 0)
        seconds = next_hour_oclock - now
        self.thread = threading.Timer(seconds.total_seconds(), self.say_the_time_hourly)
        self.thread.start()

Observe that, first, we get the value of what time is it now. And then, we calculate the next hour using timedelta from datetime module, later we calculate total seconds from current time until exactly next hour at H:0 o'clock. And finally, we run the thread using thread.start(). Don't forget to store this new thread object (using the statement self.thread = threading.Timer(..)) as object variable/property, for later dispose/cancellation of this running thread. 

The thread will run our new method say_the_time_hourly() that will call say_the_time() to  speak up the current time and schedule new thread for another hour. Below is the say_the_time_hourly() method:

1
2
3
4
5
6
    def say_the_time_hourly(self):
        '''
        say the time and then schedule for another hour at exactly n-th o'clock
        '''
        self.say_the_time(None)
        self.wakeup_next_hour()

Great. Now, what is the different between the previous say_the_time() method  with this new one? Let's inspect it in another section.

Safely Running MS Agent in Different Thread

As already been said, you can not use an instance of MsAgent that have been created from another thread. Previously, this is how we instantiate our MsAgent object:

1
2
3
4
5
6
7
8
9
10
11
12
13
class MsAgent(object):
    def __init__(self):
        self.agent=win32com.client.Dispatch('Agent.Control')
        self.charId = 'James'
        self.agent.Connected=1
        self.agent.Characters.Load(self.charId)
 
    def say_the_time(self, sysTrayIcon):
        now = datetime.datetime.now()
        str_now = '%s:%s:%s' % (now.hour, now.minute, now.second)
        self.agent.Characters(self.charId).Show()
        self.agent.Characters(self.charId).Speak('The time is %s' % str_now)
        self.agent.Characters(self.charId).Hide()

If you don't modify the above method, and forcefully try to run say_the_time() from another Thread (using threading.Timer() object), Python will gives this exception: CoInitialize has not been called. This is a clue to indicate that to use COM object in a new thread, except that the object support multi-threading concurrency model, we must instantiate it here. Actually, Pythoncom, which is the core module of PyWin32 that dealing with any sort of Python <-> COM communication, silently call CoInitialize() when first imported in main thread. Therefore, if we would like to use COM object in different thread, we have to initialize it ourselves.

Using the above explanation, below is the code that will schedule the Agent to run hourly at exactly x-o'clock and speak up the time. Got to say, it is quite a fun! Laughing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
class MsAgent(object):
    '''
    Construct an MS Agent object, display it and let it says the time exactly every hour
    '''
    def __init__(self):
        self.charId = 'James'
 
    def say_the_time(self, sysTrayIcon):
        '''
        Speak up the time!
        '''
        pythoncom.CoInitialize()
        agent = win32com.client.Dispatch('Agent.Control')
        charId = 'James'
        agent.Connected=1
        try:
            agent.Characters.Load(charId)
        except Exception as ex:
            print ex
 
        now = datetime.datetime.now()
 
        if now.hour == 0 and now.minute == 0:
            str_now = ' exactly %s o''clock' % now.hour
        else:
            str_now = '%s:%s:%s' % (now.hour, now.minute, now.second)
        agent.Characters(charId).Show()
 
        agent.Characters(charId).Speak('The time is %s' % str_now)
        time.sleep(3)
        agent.Characters(charId).Hide()
        time.sleep(5)
        agent.Characters.Unload(charId)
        pythoncom.CoUninitialize()

Observe from the code above, that we initialize our MS Agent object completely in the function say_the_time(), that will be run by a thread, making it as a one way to a thread safe COM application. We begin and end the function by calling pythoncom.CoInitialize() and pythoncom.CoUninitialize() respectively, in order to have a clean instantiation and cleanup of our MS Agent object. Observe that we also pause the running code several seconds after each MS Agent animation, to let the animation finished. Without doing so, you won't see anything!

What's Next?

As always, you can get the current source code here: oclockreminder-version-2.zip.

Or, follow its public Github repository here: pythonthusiast/oclockreminder.

You maybe wondering, why I keep pursuing this application. "It is a dead technology!", as RJ implied in the comment section of the previous article. Well, it is. But I visioned that it will be a great starting point for a new productivity application that integrated seamlessly in a Windows environment!

What is it? 

Stay tuned for my next article!

 

 

 

 

April 23, 2014 12:00 AM

April 22, 2014


Montreal Python User Group

Python Project Night XII

Python Project Night XII After a much needed break from PyCon. We at Montréal-Python are eager to get into coding. Caravan Coop will be hosting the twelfth edition of Python Nights.

We will also invite a special guest project for people interested in helping out our community.

Please sign up on our Eventbrite event as the seats are limited: https://python-project-night-xii.eventbrite.ca

We would like to thank Caravan for hosting us this evening. See you there!

April 22, 2014 11:42 PM


PyCharm

Announcing PyCharm 3.1.3

The PyCharm 3.1.3 build 133.1347 has been uploaded and is now available from the download page. It also will be available shortly as a patch update from within the IDE.

This minor bug update delivers a significant fix in the debugger to properly support Python 3.4 (PY-12317). This fix for core Python support effectively works for both free and open source PyCharm Community Edition and full-fledged PyCharm Professional Edition.

Should you have any problems or queries with this release please file them to our Issue Tracker.

So what’s coming up next? We’re still working on PyCharm 3.4. The first PyCharm 3.4 Early Access Preview build is almost ready to be revealed and we aim to release it on this week.

Develop with pleasure!
-PyCharm team

April 22, 2014 02:40 PM


Europython

You need some good reasons for going to EuroPython 2014?!

image

Some good reasons for coming to Berlin for EuroPython 2014:

Last but not least: mean, speak and discuss with hundreds of other Python enthusiats. Share your knowledge, ask questions, make connections, extend your network or just enjoy some days with other Pythonistas.

Get your ticket now!

April 22, 2014 01:54 PM


Mike C. Fletcher

Found the source of SegFaults on AMD + FreeGLUT

So the source of the segfaults that I'm seeing on fglrx and FreeGLUT on Kubuntu 14.04 has come to light. It's a known issue with the registration of FreeGLUT and fglrx at-exit handers (at the C level). You can work around it in your own code with PyOpenGL 3.1.0b3+ (which is still pending release) using:

        try:
            if fgDeinitialize: fgDeinitialize(False)
        except NameError as err:
            pass # Older PyOpenGL, you may see a seg-fault here...
        import sys
        sys.exit( 0 )

Where fgDeinitialize is available in the GLUT namespace.

April 22, 2014 01:27 PM


Logilab

Pylint 1.2 released!

Once again, a lot of work has been achieved since the latest 1.1 release. Claudiu, who joined the maintainer team (Torsten and me) did a great work in the past few months. Also lately Torsten has backported a lot of things from their internal G[oogle]Pylint. Last but not least, various people contributed by reporting issues and proposing pull requests. So thanks to everybody!

Notice Pylint 1.2 depends on astroid 1.1 which has been released at the same time. Currently, code is available on Pypi, and Debian/Ubuntu packages should be ready shortly on Logilab's acceptance repositories.

Below is the changes summary, check the changelog for more info.

New and improved checks:

Bug fixes:

Command line:

Other:

Astroid:

Nice easter egg, no?

April 22, 2014 11:39 AM

April 21, 2014


Tryton News

New Tryton release 3.2

We are happy to announce the 3.2 release of Tryton.

This release mainly consolidates many new functionalities added in last two years. Also it prepares the future migration to Python 3 by dropping the support for Python 2.6. But also as usual there are many bug-fixes, improvements and new modules (see below).

Of course, migration from previous series is fully supported.

Major changes in graphical user interface

  • The client uses the local timezone to display date time.

  • The copy/paste on editable list has been improved to add new lines if needed beside of updating existing records.

  • The buttons of the view are also available in the action menu. This allows fast access using keyboard shortcuts but also trigger the button for a list of selected records.

    button action menu
  • Buttons and wizards can now trigger actions from the client side. This means it behaves like if the user clicked on one of the tool bar buttons.

  • The client uses now a pool of connections, this allows to speed up the client on requests that can be parallelized.

  • The attachment button can now receive drag & drop of file to quickly create attachments.

  • There is a new widget multi-selection, it uses the Many2Many field as backend. It is very useful and more visual when there is a small number of selection.

    multiselection
  • The client allows to browse the revisions of a record if it is historized. It also works on a full list of records, in this case the client shows the result of the search as if it was done at the revision date.

    revisions

Major changes on the server side

  • The server runs internally always in UTC timezone.
  • The ModelStorage.write method receives the similar improvements as the ModelStorage.create in version 2.8. This means it can write different values to many sets of records in one call and so this improves the performance by validating all the records at once and also it will validate only the modified and dependant fields. Also the action values of relation fields have been updated with the same interface.
  • A new decorator fields.depends is introduced to replace the deprecated on_change, on_change_with, selection_change_with and autocomplete fields attributes. This decorator applies on the called methods and the result will be the sum of all depends of all the modules, this brings much more flexibility to the modularity.
  • Tryton uses bcrypt to hash password if the library is available.
  • All types of field can now have a domain to constraint its value and most of the domains are supported for pre-validation and inversion on the client side.
  • The on_change returned value of One2Many uses now an index for the add keyword. This allows to define the position of the new record in the list instead of being always at the bottom.
  • A new method ModelSQL.restore_history allows to restore the values of a record as they were at a specific date time.

Modules

Account

  • A new journal type write-off has been added to ease the creation of write-offs.
  • Taxes has now an optional start and end, this allows to manage the changes over time.

French Chart of Account

  • The French chart of account has been updated for the new tax rates of 2014.

Account Statement

  • The module prevents now to use an already paid invoice in draft statements.
  • It uses the new index of on_change to add the new split line under the original.

Account Stock Continental

  • The creation of account move for stock move is speed-up.

Bank

  • The IBAN numbers are now validated and formatted.

Company

  • A new timezone field is added to the company to get the right date for today.
  • The employee is also taken from the context just like the company. This allows to use many clients with the same user but different employees.

Production

  • It is now possible to define the effective date of a production. This allows to enter past productions.

Purchase

  • There is now a warning when trying to receive a supplier stock move without an origin. Normally, the origin should be a purchase order.
  • The purchase tries to create links between stock moves and invoice lines.

Sale

  • The same warning exists for customer move without origin.
  • The sale tries as the purchase to create links between stock moves and invoice lines.

Stock

  • Supplier Shipment Return can now have partial assignation
  • The computation of stock quantities has been reworked to allow easy customization and better search.
  • It is now possible to define the effective date for all shipments. This allows to enter past shipments.

Stock Lot

  • A new relate has been added from lot to moves.

New modules

  • The Party Relationship module defines different types of relations between parties.
  • The Account Payment module allows to generate grouped payments for receivable and payable account move lines.
  • The Account Payment SEPA module allows to generate SEPA files for payments.
  • The Stock Package module allows to store packaging information about customer and supplier return shipments.
  • The Sale Shipment Grouping module adds an option to define how stock moves generated from sales will be grouped.
  • The Account Credit Limit module manages credit limit of parties.
  • The Sale Credit Limit module adds confirmed sale to the credit amount of the party.

April 21, 2014 07:47 PM


Agendaless Consulting

PyCon 2014 Recap

PyCon 2014 in Montréal was a great conference! Check out the Agendaless PyCon 2014 talks:

Paul Everitt presented a half-day tutorial, "Python 3/2 Web Development with Pyramid ":

Tres Seaver gave a talk, "By Your Bootstraps: Porting Your Application to Python3":

We had lots of good conversations with folks, both in the "hallway track" and at the Pyramid booth. Great job by the Montréal Pythoneers organizing and running such a terrific conference! And for those who didn't get one there, the "Hacker" T-shirt is available online:

April 21, 2014 03:26 PM


Rob Galanakis

goless- Golang semantics in Python

The goless library https://github.com/rgalanakis/goless provides Go programming language semantics built on top of Stackless Python or gevent.* Here is a Go example using channels and Select converted to goless:

c1 = goless.chan()
c2 = goless.chan()

def func1():
    time.sleep(1)
    c1.send('one')
goless.go(func1)

def func2():
    time.sleep(2)
    c2.send('two')
goless.go(func2)

for i in range(2):
    case, val = goless.select([goless.rcase(c1), goless.rcase(c2)])
    print(val)

While I am not usually a Go programmer, I am a big fan of its style and patterns. goless provides the familiarity and practicality of Python while better enabling the asynchronous/concurrent programming style of Go. Right now it includes:

goless is pretty well documented and tested, but please take a look or give it a try and tell us what you think here or on GitHub’s issues. I’m especially interested in adding more Go examples converted to use goless, or other Go features replicated to create better asynchronous programs.**


*. goless was written at the PyCon 2014 sprints by myself, Carlos Knippschild, and Simon Konig, with help from Kristjan Valur Jonsson and Andrew Francis (sorry the lack of accents here, I am on an unfamiliar computer). Carlos and myself were both laid off while at PyCon- if you have an interesting job opportunity for either of us, please send me an email: rob.galanakis@gmail.com

**. We are close to getting PyPy support working through its stackless.py implementation. There are some lingering issues in the tests (though the examples and other ‘happy path’ code works fine under PyPy). I’ll post again and bump the version when it’s working.

April 21, 2014 01:34 PM


Will Kahn-Greene

Dennis v0.3.11 released! Fixes and Python 3 support

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.3.11 released!

v0.3.11 adds Python 3 support (there might be problems, but it's working for me) and adds error detection for the case where there's a } but no {.

Definitely worth updating!

8 out of 11 people who have heard of Dennis and continue to ignore its baby mews of wonderfulness also have a severe allergy to rainbows and kittens.

April 21, 2014 12:00 PM


Steve Holden

Neat Notebook Trick

I'm still trying to digest the little I saw of PyCon. Sadly I was pretty wiped out by the three-day recording session for Intermediate Python in California and then the intensive editing work that followed to help the amazing O'Reilly team get the whole thing ready a day before PyCon officially opened.

I also made the tactical (and, as it turned out, strategic) mistake of choosing to stay at the Hyatt in Montreal. This meant a considerable walk (for a gimpy old geezer such as myself) to the conference site, when the Palais des Congres is already intimidatingly large.

So the combination of exhaustion and knee pain meant I hardly got to see any talks (not totally unheard of) but that I also got very little time in the hallway track either. Probably the most upsetting absence was missing the presentation of Raymond Hettinger's Lifetime Achievement Award. As a PSF director I instituted the Community Service Awards, but these have never really been entirely appropriate for developers. This award makes it much clearer just how significant Raymond's contributions have been.

Because of the video releases I did spend some time of the O'Reilly stand, and signed away 25 free copies of the videos. I was also collecting names and addresses to distribute free copied of the Python Pocket Reference. If you filled out a form, you should receive your book within the next three weeks. We'll mail you with a more exact delivery date shortly.

But the real reason for this post is that I had the pleasure of meeting Fernando Perez, one of the leaders of the IPython project. He was excited to hear that the Intermediate Python notebooks are already available on Github, and when he realized the notebooks were all held in the same directory he showed me that if I dropped that URL into the Notebook Viewer site I would get a web page with links to viewable versions of the notebook. [Please note: they aren't currently optimally configured for reading, so it's still best to run the notebooks interactively, but in the absence of a local notebook server this will be a lot better than nothing. It will get better over time].

He also mentioned a couple of other wrinkles I hadn't picked up on, and we briefly discussed some of the interesting aspects of Notebooks being data structures.

The conversation was interesting enough that I plan to visit Berkeley soon to try and infiltrate my way into the documentation team and see if we can't make the whole system even easier to use and understand. One way or another, open source seems to be in my bloodstream.


April 21, 2014 03:39 AM

Intermediate Python: An Open Source Documentation Project

There is a huge demand for Python training materials, and there are many people who just don't have the spare cash to buy books or videos. That's one reason why, in conjunction with a new Intermediate Python video series I have just recorded for O'Reilly Media I am launching a new, open source, documentation project.

My intention in recording the videos was to produce a broad set of materials (the linked O'Reilly page contains a good summary of the contents). I figure that most Python programmers will receive excellent value for money even if they already know 75% of the content. Those to whom more is new will see a correspondingly greater benefit. But I was also keenly aware that many Python learners, particularly those in less-developed economies, would find the price of the videos prohibitive.

With O'Reilly's contractual approval the code that I used in the video modules, in IPython Notebooks, is going up on Github under a Creative Commons license [EDIT: The initial repository is now available and I very much look forward to hearing from readers and potential contributors - it's perfectly OK if you just want to read the notebooks, but any comments yuu have about your experiences will be read and responded to as time is available]. Some of it already contains markdown annotations among the code, other notebooks have little or no commentary. My intention is that ultimately the content will become more comprehensive than the videos, since I am using the video scripts as a starting point.

I hope that both learner programmers and experienced hands will help me turn it into a resource that groups and individuals all over the world can use to learn more about Python with no fees required. The current repository has to be brought up to date after a rapid spate of editing during the three-day recording session. It should go without saying that viewer input will be very welcome, since the most valuable opinions and information comes from those who have actually tried to use the videos to help them learn.

I hope this will also be a project that sees contributions from documentation professionals (and beginners they can help train), so I will be asking the WriteTheDocs NA team how we can lure some of those bright minds in.

Sadly it's unlikely I will be able to see their talented array of speakers as I will still be recovering from surgery. But a small party one evening or a brunch at the office might be possible. Knowing them it will likely involve sponsorship or beer. Or both. We shall see.

I think it's a worthwhile goal to have free intermediate-level Python sample code available, and I can't think of a better way for a relative beginner to get into an open source project. I also like the idea that two communities can come together over it and learn from each other. Suffice it to say, if there are enough people with a hundred bucks* in their pocket for a six-hour video training I am happy to use part of my share in the profits to support this project to some degree.

[DISCLOSURE: The author will receive a proportion of any profit from the O'Reilly Intermediate Python video series]

* This figure was plucked from the air before publication, and is still a good guideline, though as PyCon opened (Apr 11) a special deal was available on a package of both Jessica McKellar's Introduction to Python and my Intermediate Python.

April 21, 2014 03:06 AM


Real Python

Flask by Example - Part 1 - Project Setup

This tutorial shows you how to build an app using Flask with a simple text box that you enter text into and the app processes and displays a count of how many times each word appears on the page. In part one, we'll set up a local development environment and then deploy both a staging environment and a production environment on Heroku.

April 21, 2014 12:00 AM

April 20, 2014


Brett Cannon

Security of cash vs. bank accounts vs. Bitcoin

A co-worker recently gave a talk that explained how Bitcoin worked and it was interesting to hear how you should protect your bitcoins. When you think of cash, it’s basically some physical good which you own while you have it in your possession, and lose when you don’t have it. It’s very straight-forward and easy to comprehend. The biggest downside is there is no backup if you screw up, e.g. if you leave a $20 bill lying around and someone takes it there is no way to get it back. But this does mean you have complete control over money.

With a bank account, you have certain guarantees to protect your money. If someone steals your bank card information you have federal guarantees to get a refund for the fraudulent charges made against your bank account. This does mean, though, that you have to entrust your money to the bank and that they won’t go under, lose your money, etc. While the federal government makes certain guarantees about losing money from a bank, this assumes you follow the requirements and it isn’t some crazy systemic failure (e.g. the bank balance doesn’t go over a certain amount and your government isn’t collapsing).

Bitcoin is a lot like cash. What you do is make a public key and private key for your bitcoins. The public key is like an account number to send money to. The private key is like a password to access the bitcoins sent to the public key. If someone gets a hold of your private key they can easily transfer your bitcoins to another public key. And since anyone can generate a public key and not share who controls the keys it essentially becomes robbery by anonymous robbers. There is also no governmental backup in the case of theft like with your bank account. So just like cash, the security of your bitcoins are entirely up to you.

Unlike cash, though, you can keep your keys on your computer which exposes you to more potential theft than cash which is entirely offline. This is why the Bitcoin community has two pieces of advice to help mitigate the loss of bitcoins from someone breaking into the computer storing your private keys. First is to constantly be moving your bitcoins to different keys and to not keep all of your bitcoins in a single private key. The idea is that when you use the funds in a private key you transfer all of the funds out: transfer to the public key that instigated the transfer and then the remaining balance to another public key(s) you control.

The second piece of advice is that for long-term storage you should keep your private key entirely offline. In this scenario the idea is that if your private key never goes near the internet there is no chance that digital thieves can steal it (thus it ends up just like cash and only susceptible to physical theft). This becomes an interesting challenge in varying levels of paranoia to generate such a private key in such a way that it won’t be accessible ever to the internet until such a time that you want to withdraw from private key. Probably the simplest approach is to visit something like https://bitcoinpaperwallet.com in an incognito window in Chrome, take your computer offline, use the website to create a private/public key pair, print the keys using a printer that has no internet connection or buffer that saves what it prints, print the private key, quit Chrome, and then give your computer an internet connection again. This is probably good enough for most people.

But what if you want a proper guarantee that there is no chance your private key will ever touch the internet? For that you will want to take a Raspberry Pi which has only a physical connection to the internet, update its software with everything you need to print along with what I mentioned previously, disconnect your Raspberry Pi from the internet, generate the private key, print it, turn off your Raspberry Pi, and then destroy the SD card which you used to run your Pi. By never connecting your Pi to the internet once you begin the process of generating your private key you know it won’t leak online since there is no WiFi that might accidentally be turned on without you knowing. And by destroying the SD card immediately after you are done you guarantee that you will never accidentally have the private key make it on to the internet by reading the SD card on a computer with an internet connection. With that you can then do stuff like use tamper-resistant stickers to make sure no one snuck a peak at your private key and make copies that you distribute to trusted friends to protect against accidental destruction in e.g. a fire. This ability to have a single piece of paper represent any amount of money and to be able to physically make copies for safe keeping is what differentiates bitcoins from cash.

April 20, 2014 02:48 PM

Should my family get tablets or laptops?

With the assumption that everyone in my family has a smartphone of some kind, the question becomes whether members of my family should buy a laptop or a tablet as their primary computing device while at home. I think my general answer is choose one or the other depending on whether you need a keyboard regularly, but you will only want either a laptop or tablet and not both.

I did a blog post once about why I thought the tablet craze has died down. I basically said that the differentiator between a tablet and a small laptop like a Chromebook was the lack of a keyboard. If you type a lot then the lack of a keyboard on a laptop can be a hindrance. While you can get keyboards for tablets, they typically are structured such that you must have a flat surface to place the tablet and keyboard on, unlike a laptop which will work as-is in your lap.

I have continued to agree with this assessment. The point of this blog post, though, is to say a good-sized tablet – that will depend on you, so try to play with various tablet sizes to see which ones seem reasonable – can replace a laptop if you don’t need a keyboard regularly. If you write emails on a regular basis, then get a laptop. But if you can live without a keyboard and you still have access to a laptop for those times when you need one then a tablet can work. It’s worked for my father-in-law quite well and I don’t see why it couldn’t work for other family members. And if you so choose to buy a tablet, the recommendations I made in my mobile phone post hold for which tablet to buy.

April 20, 2014 12:23 AM

April 19, 2014


Yasoob Khalid

Misconceptions about Skype local database

Hi there guys. Recently I wrote an article with the name of “I didn’t know Skype stores your data in a local database without a password!. After publishing that article I got a lot of response from people like you and I came to know that it is not a vulnerability. It is so because the database is stored in the “appdata” directory which can only be accessed by the administrator which means that only an administrator account can open it. If you want someone else to use your computer just make a guest account which will restrict their level of access to the main directories only (this excludes the appdata directory). If you want to see your Skype logs then simply log in to your Skype account rather than going the complex way of accessing the local database.

However the tool (SkypeFreak) which I posted about in the previous post can be used as a post reconnaissance tool which means that if you hack into a computer then you can use the tool to access the Skype data without knowing the password.

At last I would like to apologize all of you about any misconceptions which my previous post might have made in your mind. You can safely discard those misconceptions as my mistake.

source: Previous post


April 19, 2014 12:42 PM


Invent with Python

Decimal, Binary, and Hexadecimal Odometers

It can be difficult to see how other number systems (such as binary and hexadecimal) work since they have a different amount of numerals than the ten numerals of decimal. But imagine that you are counting in these number systems using an old-fashioned analog odometer that has a different amount of numerals for each digit.

The following three odometers always show the same number, but they are written out differently in different number systems:

Decimal (Normal, base-10 with digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9):

Binary (base-2 with digits 0, 1):

Hexadecimal (base-16 with digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F):

UPDATE: Source code for Gavin Brock’s JavaScript odometers. Source code for this binary/decimal/hexadecimal demo all on a single page.


April 19, 2014 07:02 AM


Eko S. Wibowo

Reviving Microsoft Agent using PyWin32 - How to Make It Live!

 

I just found an MS Agent characters I haven't seen before, and though to share how to control them from within Python

I just found an MS Agent characters I haven't seen before, and though to share how to control them from within Python

I think this article was driven by the fact that this April, Microsoft discontinued Windows XP. And why is that related? Because if you remember Windows XP, somehow I think you will remember --maybe in annoyance-- this cute little dog called Rover from Windows XP Find Files bar:

Howdy, Rover?

Howdy, Rover? Not even you, now your master was also dead. So sad.. Frown

Long story short, Microsoft did not gain success with this MS Agent stuff. Without even realizing it, one of the first thing I do when re-installing Windows XP is ... get rid of Rover from Windows Explorer! And yeah, unfortunately, I am not the only one.

But still, in this article we are going to use it as a study case on how to properly use PyWin32, an amazing Python package from Mark Hammond, that let your Python application integrate tightly with Windows. For example, do you know that Dropbox client application was built with Python? Laughing

Great, lets dig more on this matter!

Three Different Routes for Windows Integration

There are three different routes you can take to integrate Python application seamlessly to Windows environment, which are:

  1. Using PyWin32 package from Mark Hammond.
    For casual application, this well-maintained and well-supported package is your first bet for a successful Windows integration. Download its latest build here (build 218 as the time of this writing), and chose the correct version for your Python distribution. When I say casual, it means that your application is satisfied with the ability to call Win32 API and automate IDispatch based COM Objects (COM/Active-X Objects that can be automated using scripting environment such as JavaScript/VBScript). A clear example would be this blog, that shows you how to use Python in Excel.
    PS: I love seeing people use Python in many diverse computing environment like that!
  2. Using comtypes package from Thomas Heller
    When you want to automate particular COM Objects that are derived from IUnknown instead of IDispatch, this will be the time for you to use comtypes. It really is amazing knowing what you can do with comtypes. For example, do you know that DropBox client application is a Python application? This article from DropBox Tech Blog may shed some light for you! Or, this codeproject article that demonstrate how to work with custom COM interface, will help you in your next step in Windows integration.
  3. Using ctypes package, also authored by Thomas Heller
    When your Windows integration necessity are beyond custom COM interface, in which you need to call specific DLLs either as part of Windows distribution (such as those reside in kernel32.dll) or wrap your own C function, you will going to need ctypes.

Quick note regarding COM/Active-X: Although it's not dead, COM (Component Object Model) is technically superseded by .NET Framework. In its essence, COM is a software component architecture that let diverse application reuse components from each others to build a working application. As you know that Windows exposed its functionality through Windows API (e.g creating system tray application using Shell_NotifyIcon function), it also exposed other parts of its system using COM object (e.g. INetCfg is an IUnknown based COM Object that let you configure Windows networking configuration). Later on .NET Framework was introduced and at the current moment a new API is come to live: Windows RT. I love hearing those jargon! Laughing

In this article, we are going to look on how to create a better Python application that integrate well in Windows, using PyWin32 packages, specifically its win32com package to automate Microsoft Agent component. We are going to make a talking application that will obediently sit low and still in System Tray, and will speak up the time every each hour. Lets get down to it!

Installing Microsoft Agent

I you are still using Windows XP (woops), then nothing else you should do. MS Agents was right there alright. But if you happen to use Windows 7, go ahead to this page from Microsoft and install a hot-fix to bring back MS Agent. Users who use Windows 8/8.1 may find this support page or this page useful. Another interesting collection of MS Agent characters can be found here.

Through out the rest of the article, I am using James characters as depicted in the above figure. You may want to use another characters of your liking. Be sure to check their supported animation though! For example, James supported animation can be found here

PS: Just found out this open source replacement for MS Agent also support Windows 7 (not sure about Windows 8 though). You may try it, although currently I am using that MS Agent Hot-Fix from Microsoft.

It's alive!

The next question is, "This MS Agent things is a COM object, right? But how do I use it in my Python application?". Thanks to great work by Mark Hammond, we got this awesome PyWin32 package that let us use native Windows API and COM services from within Python environment. Download an install an appropriate distribution for your Python environment from this Sourceforge page.

Let's test our newly installed James agents, and see if we truly bring it to life:

1
2
3
4
5
6
import win32com.client
ag=win32com.client.Dispatch('Agent.Control')
ag.Connected=1
ag.Characters.Load('James')
ag.Characters('James').Show()
ag.Characters('James').Speak('Hi there Pythonist. I see that you brought me back to this world. Thank you!')

A warning though: don't try to save the above code into a *.py file and run it with either python command line or your favorit e IDE. Why? It's because the main thread that start your application will exit immediately and the character won't have time to show itself. Paste the above code into an interactive python REPL session or even ipython if you like, and you'll see a character pop up and speak using a digital speech synthesizer. Pretty cool for a thrown away product, right?

Make it always run: stick it in a System Tray application

This is another example of how awesome PyWin32 package is: create a system tray icon for Python application. As the above code need a main thread that will keep the character showing, we are going to create a Python application that put an icon in the system tray. For the System Tray functionality, I am going to use Simon Brunning's SysTrayIcon class, which is a rip off from Mark Hammond's win32gui_taskbar.py and win32gui_menu.py demos from PyWin32. In this article, I use it without changes. So, credit goes to those guys..

Below is a new Python class that wrapped a single MS Agent characters of your liking. Observe that the code is pretty much straightforward to understand.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
__author__ = 'Eko Wibowo'
 
import systray
import win32com.client
import datetime
import threading
 
class MsAgent(object):
    '''
    Construct an MS Agent object, display it and let it says the time exactly every hour
    '''
    def __init__(self):
        self.agent=win32com.client.Dispatch('Agent.Control')
        self.charId = 'James'
        self.agent.Connected=1
        self.agent.Characters.Load(self.charId)
 
    def say_the_time(self, sysTrayIcon):
        '''
        Speak up the time!
        '''
        now = datetime.datetime.now()
        str_now = '%s:%s:%s' % (now.hour, now.minute, now.second)
        self.agent.Characters(self.charId).Show()
        self.agent.Characters(self.charId).Speak('The time is %s' % str_now)
        self.agent.Characters(self.charId).Hide()
 
    def bye(self, sysTrayIcon):
        '''
        Unload msagent object from memory
        '''
 
        self.agent.Characters.Unload(self.charId)
        self.thread.cancel()
 
if __name__ == '__main__':
    import itertools, glob
 
    icons = itertools.cycle(glob.glob('*.ico'))
    hover_text = "What can I do for you Sir?"
 
    agent = MsAgent()
    menu_options = (('Say the time', icons.next(), agent.say_the_time),)
 
    systray.SysTrayIcon(icons.next(), hover_text, menu_options, on_quit=agent.bye, default_menu_index=1)

Put an *.ico file in the same folder as the application (of course, together with systray.py module). Run it, and you will a see a new icon in the Windows System tray. Right click it, and choose "Say the time". James will obediently follow your command :) 

How to Run This Application?

I saved the main application in a file named oclockreminder.pyw. By using this extension, if this file was double clicked, it will be executed by pythonw.exe, making a non-console application (it's similar to javaw.exe). You can later create a shortcut for this file in Windows Start Menu, and  having it run automatically. Actually, the best way would be to prepare an *.exe installer for this application. We are going to explore this option later on this blog.

Conclusion

Realizing that Python can integrate well in a particular Operating System, bring a window to a whole new level of possibilities. The topic discuss in this article still only touch the surface of what you can do with Python. But it I hope it gives enough foundation to get you started. 

Download the application source directly from this link, or browse through Pythonthusiast public dropbox folder here.

Or, follow its Github Repository: pythonthusiast/oclockreminder.

Stay tuned for my next article!

April 19, 2014 12:00 AM

April 18, 2014


Bruno Rocha

Using Flask Cache

As a micro framework Flask does not have built-in cache functionality, however, there is werkzeug cache API and an excellent extension to provide its caching functionality to your Flask apps, that extension was created by @thadeusb and is very easy to implement and use.

installing

In your env install it via PyPI (recommended)

pip install Flask-Cache  

You can also install it directly from source code if you need recent changes or bugfixes

pip install https://github.com/thadeusb/flask-cache/tarball/master

Configuring

There is a set of configuration keys you can put in your app settings, but the most important is the cache backend defined by the CACHE_TYPE key.

The cache type resolves to an import string which needs to be an object implementing the werkzeug cache api, but there is some aliases to the werkzeug.contrib.cache implementations

By default the CACHE_TYPE is Null which means that your app will have no cache, so you need to choose of the options below:


Full options and config variables are in http://pythonhosted.org/Flask-Cache/#configuring-flask-cache

A Simple Flask app

a file named app.py

import time
from flask import Flask

app = Flask(__name__)

@app.route("/")
def view():
    return time.ctime()

if __name__ == "__main__":
    app.run(port=5000, debug=True, host='0.0.0.0')

Run the above with python app.py and open http://localhost:5000 in your browser and hit F5 (refresh) to see the current date and time.

Enabling Flask-Cache for views

Now we want to enable caching on that small application to avoid the refresh of the current time (for this example we are using current time as return but imagine that it could be a large dataset or huge calculations)

a file named cached_app.py

import time
from flask import Flask

# import the flask extension
from flask.ext.cache import Cache   

app = Flask(__name__)

# define the cache config keys, remember that it can be done in a settings file
app.config['CACHE_TYPE'] = 'simple'

# register the cache instance and binds it on to your app 
app.cache = Cache(app)   

@app.route("/")
@app.cache.cached(timeout=300)  # cache this view for 5 minutes
def cached_view():
    return time.ctime()

if __name__ == "__main__":
    app.run(port=5000, debug=True, host='0.0.0.0')

Run the above with python cached_app.py and open http://localhost:5000 in your browser and hit F5 (refresh) to see that the current date and time is now cached for 5 minutes.

@cache.cached decorator takes the request.path for that view and use this as cache key, if for any reason you need a different key you can pass a key_prefix argument to the decorator. In this case if you pass a key_prefix containing the %s placeholder it will be replaced by the current request.path

The above is the simplest and regular example of Flask app and the use of cache, but, if your app is designed using application factories, blueprints, class based views or views located in different modules you will need to use advanced approach.

Caching regular functions

The same cached decorator can be used to cache regular functions, but in this case you will need to specify the key_prefix argument, otherwise it will use the request.path which can lead to conflicts if you have many cached functions.

For this example we are going to use the this module and extract a random quote from the Zen of Python.

A file named cached_function_app.py

import time
import random

from this import s, d
from string import translate, maketrans

from flask.ext.cache import Cache
from flask import Flask

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

@app.cache.cached(timeout=10, key_prefix="current_time")
def get_current_time():
    return time.ctime()

def random_zen_quote():
    """Pick a random quote from the Zen of Python""" 
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@app.route("/")
def zen():
    return """
    <ul>
        <li><strong>It is cached:</strong> {cached}</li>
        <li><strong>It is not cached:</strong> {not_cached}</li>
    </ul>
    """.format(
        cached=get_current_time(),
        not_cached=random_zen_quote()
    )

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Now running python cached_function_app.py and opening http://localhost:5000 when hitting F5 to refresh you will see the current time cached for 5 minutes and the random quote updated, you can switch the cache just to see the efect.

def get_current_time():
    return time.ctime()

@app.cache.cached(timeout=10, key_prefix="zen_quote")
def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@app.route("/")
def zen():
    return """
    <ul>
        <li><strong>It is not cached:</strong> {cached}</li>
        <li><strong>It is cached:</strong> {not_cached}</li>
    </ul>
    """.format(
        cached=get_current_time(),
        not_cached=random_zen_quote()
    )

NOTE: Because we are importing the this module for the example, you will see the Zen quotes in your flask terminal, but there is no problem with this.

Caching modular views

Now an example when you have your app splitted in two or more files for better organization

in a folder called app put 3 files__init__.py, app.py and views.py

app/__init__.py is an empty file

app/views.py

import time
import random
from this import s, d
from string import translate, maketrans

def get_current_time():
    return time.ctime()

def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

def zen_view():
    return """
    <h1>Cached for 10 seconds!</h1>
    <ul>
        <li>{time}</li>
        <li>{quote}</li>
    </ul>
    """.format(
        time=get_current_time(),
        quote=random_zen_quote()
    )

as you can see the above file defined view functions, as it it a separated file, to avoid circular imports we are not recommended to use @app.route neither @app.cache so this views will be app agnostic and we are going to register its url rules and caching in the main app file.

That kind of structure is needed when your app has too many views and want a better organization.

NOTE: For better organization the mostly recommended pattern is Blueprints which I will explain further.

app/app.py

Now in the main app we need to import our views, explicitly decorate for caching and also register its urls.

from flask import Flask
from flask.ext.cache import Cache
from views import zen_view

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

# explicitly apply the cache in the old-style decoration way
cached_zen_view = app.cache.cached(timeout=10)(zen_view)

# explicitly register the cached view url mapping
app.add_url_rule("/", view_func=cached_zen_view)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

NOTE: You can also separate the cache instance in a different file for lazy initialization as we are going to see in the next example

Caching Blueprint views

As mentioned before, the best pattern to follow in Flask applications is the Blueprint pattern which is a way to create separated 'meta-apps' that will be connected to your main application in the time of initialization, the problem here is that Blueprints are meant to be reusable by many different applications, so the delegation of cache control should be dynamized.

In order to avoid circular imports you will want to create your cache instance separate from your application instance (you may want to consider switching to the app factory module if you are building something more complex).

Create a folder called blueprint_app with the following structure

cached_blueprint_app/
├── app.py
├── cache.py
├── blueprints
│   ├── __init__.py
│   └── zen_blueprint.py
└── __init__.py

The cache.py

from flask.ext.cache import Cache    
cache = Cache()

we can create a dummy lazy cache instance, that will be initialized in the future when the view will be called. For that in the app we are going to reimport the same cache instance and call init_app method.

The basic blueprints/zen_blueprint.py

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint
from cache import cache

zen = Blueprint('zen', __name__)

def get_current_time():
    return time.ctime()

def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@zen.route("/")
@cache.cached(timeout=20)
def zen_view():
    return """
    <h1>Cached for 20 seconds!</h1>
    <ul>
        <li>{time}</li>
        <li>{quote}</li>
    </ul>
    """.format(
        time=get_current_time(),
        quote=random_zen_quote()
    )

NOTE: In a real application you will want to modularize it separating the views, helpers etc and promoting your blueprint to a Python package.

The main app.py

from flask import Flask

from blueprints.zen_blueprint import zen
from cache import cache

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
cache.init_app(app)

app.register_blueprint(zen)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Notice that we created a dummy instance of cache in cache.py and then used that instance to decorate the blueprints views, then the cache was initialized in app.py with init_app method. That is possible because of the Flask initialization cycle and the excellent implementation in Flask-Cache extension that takes care of this case, if you plan to write yor own Flask extension take a look at the Flask-Cache source code.

Run the application by calling python cached_blueprint_app/app.py and open http://localhost:5000 to see the blueprint view cached for 20 seconds.

Caching MethodView

Lets use the same cached_blueprint_app example but turning the zen_view in to a MethodView

Change your zen_blueprint.py to:

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint
from flask.views import MethodView
from cache import cache

zen = Blueprint('zen', __name__)

class ZenView(MethodView):

    @cache.cached(30)
    def get(self):
        return """
        <h1>Cached for 30 seconds!</h1>
        <ul>
            <li>{time}</li>
            <li>{quote}</li>
        </ul>
        """.format(
            time=self.get_current_time(),
            quote=self.random_zen_quote()
        )

    @staticmethod
    def get_current_time():
        return time.ctime()

    @staticmethod
    def random_zen_quote():
        transtable = maketrans("".join(d.keys()), "".join(d.values()))
        return random.choice(translate(s, transtable).split("\n")[2:])


zen.add_url_rule("/", view_func=ZenView.as_view('zen'))

Method views maps HTTP method names as GET, POST, DELETE to the view methos as get, post, delete etc, So all we needed to do is to create a method called get and decorate it with @cache.cached decorator.

NOTE: Due to the implicit self from the caller’s perspective you cannot use regular view decorators on the individual methods of the view however, Flask-Cache is one exception because its implementation allow the use of cached decorator in individual methods. Keep this in mind.

Alternativelly you may want to cache all the methods in a view, for that you can cache the dispatch_request method or even better you can decorate the whole view.

Caching the dispatcher
class ZenView(MethodView):
    @cache.cached(timeout=30)
    def dispatch_request(self):
        return super(ZenView, self).dispatch_request()

    ...
Caching the whole view (recommended)
zen = Blueprint('zen', __name__)

class ZenView(MethodView):
    ...

cached_zen_view = cache.cached(timeout=50)(ZenView.as_view('zen'))
zen.add_url_rule("/", view_func=cached_zen_view)

Caching template blocks

Flask cache comes with a template tag able to cache template blocks, lets change our ZenView to use a Jinja2 template

in zen_blueprint.py

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint, render_template
from flask.views import MethodView

zen = Blueprint('zen', __name__)

class ZenView(MethodView):

    def get(self):
        return render_template(
            'zen.html',
            get_random_quote=self.random_zen_quote
        )

    @staticmethod
    def get_current_time():
        return time.ctime()

    @staticmethod
    def random_zen_quote():
        transtable = maketrans("".join(d.keys()), "".join(d.values()))
        return random.choice(translate(s, transtable).split("\n")[2:])

zen.add_url_rule("/", view_func=ZenView.as_view('zen'))

Now we need to create a template file in cached_blueprint_app/templates/zen.html

<h3> Random Zen of Python </h3>
<strong>{{get_random_quote()}}</strong>

Running the application with python cached_blueprint_app/app.py and opening http://localhost:5000 you will see a random quote refreshed every time you push F5, lets cache it for 30 second.

Change the zen.html template

{% cache 30 %}
<h3> Random Zen of Python </h3>
<strong>{{get_random_quote()}}</strong>
{% endcache %}

Now save the file and refresh to see the content cached for 30 seconds.

Caching functions and views with variant arguments using memoize decorator

Sometimes yout views and functions receives arguments which can come from url mapping or directly to the function call, yiou may want to cache the view or funtion and use the arguments as keys to cache its different results, Flask-Cache has a different decorator for doing that.

NOTE: With functions that do not receive arguments, cached() and memoize() are effectively the same.

Now with a simple application memoize_app.py

import time
from flask.ext.cache import Cache
from flask import Flask

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

@app.cache.memoize(timeout=5)
def get_current_time_and_name(name):
    return "%s - %s" % (name, time.ctime())

@app.route("/<name>")
def view(name):
    return get_current_time_and_name(name)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Now run python memoize_app.py and open http://localhost:5000/yourname and note that the function will be cached for each different name you pass as argument in the url.

Caching arbitrary objects

There are some times when decorators cannot be used and you need to explicitly set or get some thing on the cache.

Inside a view or a blueprint you can use current_app

from flask import current_app

def some_function():
    cached = current_app.cache.get('a_key')
    if cached:
        return cached
    result = do_some_stuff()
    current_app.cache.set('a_key', result, timeout=300)
    return result

Or if using a separete cache instance you can do this directly

from cache import cache

def function():
    cached = cache.get('a_key')
    if cached:
        return cached
    result = do_some_stuff()
    cache.set('a_key', result, timeout=300)
    return result

Clearing the cache

You can create a script to clear the cache, or a function to use it when needed

from flask.ext.cache import Cache    
from yourapp import app
cache = Cache()

def main():
    cache.init_app(app)

    with app.app_context():
        cache.clear()

if __name__ == '__main__':
    main()

WARNING: Some backend implementation do not support completely clearing the case. Also, if you’re not using key prefix, some implementation (e.g. Redis) will flush the whole database. Make sure you’re not storing any other data in your caching database.

There is a lot of examples and well documented API in flask-Cache website http://pythonhosted.org/Flask-Cache/ you can also create your own cache backend following the examples in the Flask-Cache docs.

April 18, 2014 11:05 PM


Peter Bengtsson

Grymt - because I didn't invent Grunt here

grymt is a python tool that takes a directory full of .html, .css and .js and prepares the html for optimial production use.

For a teaser:

  1. Look at the "input"

  2. Look at the "output" (Note! You have to right-click and view source)

So why did I write my own tool and not use Grunt?!

Glad you asked! The reason is simple: I couldn't get Grunt to work.

Grunt is a framework. It's a place where you say which "recipes" to execute and how. It's effectively a common config framework. Like make.
However, I tried to set up a bunch of recipes in my Gruntfile.js and most of them worked well individually but it was a hellish nightmare to get it all to work together just the way I want it.

For example, the grunt-contrib-uglify is fine for doing the minification but it doesn't work with concatenation and it doesn't deal with taking one input file and outputting to a different file.
Basically, I spent two evenings getting things to work but I could never get exactly what I wanted. So I wrote my own and because I'm quite familiar with this kind of stuff, I did it in Python. Not because it's better than Node but just because I had it near by and was able to quicker build something.

So what sweet features do you get out of grymt?

  1. You can easily make an output file have a hash in the filename. E.g. vendor-$hash.min.js becomes vendor-64f7425.min.js and thus the filename is always unique but doesn't change in between deployments unless you change the files.

  2. It automatically notices which files already have been minified. E.g. no need to minify somelib.min.js but do minify otherlib.js.

  3. You can put $git_revision anywhere in your HTML and this gets expanded automatically. For example, view the source of buggy.peterbe.com and look at the first 20 lines.

  4. Images inside CSS get rewritten to have unique names (based on files' modified time) so they can be far-future cached aggresively too.

  5. You never have to write down any lists of file names in soome Gruntfile.js equivalent file

  6. It copies ALL files from a source directory. This is important in case you have something like this inside your javascript code: $('<img>').attr('src', 'picture.jpg') for example.

  7. You can chose to inline all the minified and concatenated CSS or javascript. Inlining CSS is neat for single page apps where you have a majority of primed cache hits. Instead of one .html and one .css you get just one .html and the amount of bytes is the same. Not having to do another HTTP request can save a lot of time on web performance.

  8. The generated (aka. "dist" directory) contains everything you need. It does not refer back to the source directory in any way. This means you can set up your apache/nginx to point directly at the root of your "dist" directory.

So what's the catch?

  1. It's not Grunt. It's not a framework. It does only what it does and if you want it to do more you have to work on grymt itself.

  2. The files you want to analyze, process and output all have to be in a sub directory.
    Look at how I've laid out the files here in this project for example. ALL files that you need is all in one sub-directory called app. So, to run grymt I simply run: grymt app.

  3. The HTML files you throw into it have to be plain HTML files. No templates for server-side code.

How do you use it?

pip install grymt

Then you need a directory it can process, e.g ./client/ (assumed to contain a .html file(s)).

grymt ./client

For more options, check out

grymt --help

What's in the future of grymt?

If people like it and want to add features, I'm more than happy to accept pull requests. Some future potential feature work:

April 18, 2014 09:58 PM


Yasoob Khalid

I didn’t know Skype stores your data in a local database without a password!

Hi guys! How are you? I hope you are doing great. Recently I came to know that Skype (video conferencing software) stores a local database with almost all information of a user who has logged on to skype from that computer. You might be thinking “So what? A lot of apps do that, right?”. Yes you are right. This is mostly done to increase speed. It’s like caching the content so that whenever you log in again to your account you don’t have to wait to see your contacts. It is fine but only to this extent.

I came to know that one can take a look at the local database and extract data from it. Is that scary for you? No? Listen this. If you have some guests at your house and someone from them is a computer freak and asks you to let him use your computer. What will you do? Definitely you will say ok.

Now comes the scary part. That freak can use a simple program called SkypeFreak to connect to the local Skype database and get the info regarding your friends, the messages you have sent, the calls you have made and their duration etc, without knowing your password! He can even know about the secret messages which you send to your girlfriend. I guess now that seems scary. Right? Lets move on and see how this SkypeFreak works.

SkypeFreak is a simple Python program written by Osanda Malith for info-sec purposes. He is a security person, not a professional programmer. I recently stumbled on his program and ended up doing a complete rewrite of the source code to make it more readable, shorter and compatible with Python 3. This program contains some carefully crafted database queries which return the data from the database. Some example queries include:

SELECT fullname, skypename, city, country,\
datetime(profile_timestamp,'unixepoch') FROM Accounts
SELECT displayname, skypename, country, city, about,\
phone_mobile,homepage, birthday , datetime(lastonline\
_timestamp,'unixepoch') FROM Contacts;

The database can be connected with our Python script using sqlite3 and then we can execute these queries. The only gotcha is that the freak has got to know you Skype username but we all know that the auto complete option in Skype client can help us get that. Lets understand the main working of this program.

In all major OS’s Skype stores the database in a known location without any encryption or password (not even a simple one). For example on windows it is stored in

<$appdata>\Skype\<skype username>\main.db

Firstly you tell SkypeFreak about the skype username of the victim. After that SkypeFreak searches the local directories for a folder with that name and finally it lays its hands on the database. Furthermore after connecting to that database SkypeFreak gives you some options like get calls data, get messages data etc. When you utilize any of these commands SkypeFreak prompts you to save this info in a separate text file. That’s it! Now you are hacked! The freak can not do much with your Skype account. He only gets the data out of it, not your password which means that you do not have to change your password.

I was myself shocked when I got to know that it’s that simple to get Skype data. Microsoft should take some steps to ensure the privacy of user and prevent this type of data falling into wrong hands. They should at least password protect the database so that it is not this much simple to access it. The password can be hard-coded into the application or anything like that. I can no longer trust Microsoft with my sensitive data. If you have any questions, comments or suggestions feel free to comment below.

Last but not the least, follow my blog in order to stay up to date with my new articles. See you later!

Source: SkypeFreak


April 18, 2014 07:09 PM


Damián Avila

48 themes for your IPython notebook

OK, a short post to give you some material to play with over the weekend ;-).

Today, I woke up early and whereas I was drinking a mate (a native drink here in Argentina) for breakfast, I remember a tweet from Nikhil Sonnad where I was mentioned:

Read more…

April 18, 2014 03:10 PM