Category Archives: programming languages

The PyCon Summits

This year at PyCon there are two new events, a Dynamic Language Virtual Machine Summit, and a Python Language Summit. Each summit lasted one day, and occurred during the tutorial program of the main PyCon conference. I am really happy to see these events happen, to take advantage of people all being in one place.

Dynamic Language Summit

The organizers of this summit invited a broad variety of virtual machine implementors to meet and discuss their successes and challenges in implementing various dynamic languages.   

Sun has hosted a similar summit for people implementing languages on the JVM, and Microsoft’s Lang.NET, which is their corresponding event for the CLR, is in a few weeks. There’s been a nice cross pollination of people between those summits. The PyCon event is the corresponding event for people working on non-JVM/CLR virtual machines. I would love to see all of these communities cross pollinating each other eventually.

The VM’s represented were CPython, PyPy, Jython, JRuby, Parrot, MLVM/Da Vinci Machine, IronPython, DLR, Factor, TraceMonkey, Cog, unladen_swallow, Rubinius, and MagLev. Since this was the first time this group of people met together, we spent the morning doing brief introductions of the various projects. From there the room exploded into a bunch of small unstructured discussions. There were many of these, and they are hard to summarize, so I am going to pass on trying to summarize this any further. I really would have liked to see representatives from PHP, Erlang, io, Newspeak, SquirrelFish, and V8.   

Disclosure: I was involved in getting Sun to sponsor this summit (and PyCon)

Python Language Summit

The Python Summit was focused on bringing together developers of Python implementations and the Python standard library in order to discuss issues of importance to Python.   Major scheduled topics included the roadmap for Python 2.x/3.x (including release scheduling), cross-implementation issues, and package distribution and installation.

During the 2.x/3.x section, there was some question about whether or not “batteries included” was still the right approach for the standard library or whether improvements to packaging mechanisms might lead to a different approach. Since packaging still has some major issues, there wasn’t a lot of progress on this front. There was also some discussion on the merits of using C++ in the Python compiler. Again not much of a conclusion here, but the majority of the thought in the room was to use a few C++ features as possible. There was also discussion of including some improvements to unittest.
The largest discussion during that section, was of course, the roadmap for 2.x versus 3.x, and how to encourage people to move from 2.x to 3.x. It looks like there is (good) reluctance to keep the 2.x series moving forward, so there may be a long (possibly infinitely long) gap between 2.7 and 2.8. At the same time, it seems clear that 3.1, which is currently in alpha, will be the first truly usable release in the 3.x line. The goal is to get 3.1 out pretty quickly and deprecate 3.0 – an odd and one time practice for the Python community. There was also a discussion of what could be done to help library and framework developers jump to 3.x. One concrete outcome was agreement to start work on a 3to2 tool which would allow developers to develop on 3.x and then backport to 2.x, thus helpoing developers to flipp their effort into (in my opinion) the correct release stream

The cross implementation session was uncontroversial and very friendly. One thing that I love about the Python community is the openness to alternate implementations of the language. Implementors of the alternate implementations were encouraged to just go and do what was needed to accommodate their needs. One item that came up was the utility of splitting the standard lib, the unit tests, and (perhaps) the docs, so that all implementations could reuse them. Everyone agreed that this was a good idea, so look for this to happen soon. Also, developers of the alternate implementations were given commit bits to CPython and told to make the necessary changes. One phrase that I heard during the discussion was “putting CPython on an equal footing with the other implementations”. Based on the decisions and actions during the session, I’d say this is way more than lip service.

The last planned session covered the issue of packaging. Tarek Ziade presented the results of his packaging survey. After that there were introductions of a few of the isolation tools, virtualenv, and zc.buildout. I’d never taken a serious look at zc.buildout, but Jacob Kaplan-Moss convinced me to take a serious look by showing a sample buildout file. Apparently the documentation for zc.buildout is sorely lacking, so if any readers are zc.buildout experts or fans, please leave a comment with pointers. From there, discussion went on to a very good list that Tarek assembled. You can read his final summary of the discussion as well as what should happen next.

unladen_swallow

One unscheduled item that got a lot of buzz was the unladen_swallow project. This is a branch of CPython (2.6 initially, but eventually forward ported to 3.x) initiated by people at Google, with the goal of improving CPython’s performance by 5X. A 5X speedup would be pretty amazing – the details of how they plan to accomplish this are in the project plan. Apparently it is already 30% faster than CPython, and this version is being used to run some of the Python code on YouTube. They have yet to embark on the task of integrating LLVM as JIT for Python. I am also happy to see that they are going to tackle removal of Python’s global interpreter lock (GIL). The team at Google is planning to merge their code back into the main CPython implementation, and have been discussing this in detail with the existing core Python developers. The unladen_swallow is focusing on Linux, but they told me that they would gladly accept community help for testing on Macintosh, Windows, and other platforms.

Ever since the Javascript engine arms race started earlier this year, I’ve been concerned about the lack of a similar situation in the various dynamic language communities. It appears that things in the Python world are starting to heat up.

On the whole, I found these two summits to be highly useful. I hope that the PyCon organizers will do them again.

Refactoring in the Functional Programming world

I’m an Emacs guy. I was first exposed to Emacs back in 1984 on a VAX running BSD. This was prior to GNU Emacs, so the Emacs that I saw was James Gosling’s Emacs. At the time, I was working on a compiler for a functional programming language called SUPER, which was evaluated using combinator graph reduction.

For many years, and across many languages, including Scheme, C, C++, Perl, TCL, and Java, Emacs was my tool of choice. My hands had the muscle memory for the keystrokes, and over those years I accumulated a file full of Emacs-Lisp customizations for Emacs (by this time, mostly GNU Emacs). When Eclipse started to support refactoring I started using Eclipse as my primary tool for editing Java programs. Refactoring is an example of the kind of high leverage features that I want in my programming tool set.

A few days ago I found some gems buried in a thread on the Scala mailing list. Dave Griffith has been accumulating a list of refactorings for Scala. Here’s his complete list:

Curry Method (split a parameter list, and the arg lists of all callers).

Uncurry Method (merge split parameter list, including merging the arg lists of callers. If method is called with partial args, either complain or automatically create a helper method which represents the partial application, and replace partial calls with it.)

Extract Trait (including searching for other classes which can have the same trait extracted. Tricky with super calls, but not impossible)

Split Trait (splits trait into two traits (putting in self-types if needed), change all extending classes to extend both traits)

Extract Extractor (select a pattern, automatically create an extractor)

Extract Closure (similar to extract method, but creating a function object)

Introduce by-name parameter

Extract type definition (obvious)

Merge nested for-comprehensions into single for-comprehension (and converse)

Split guard from for-comprehension into nested if (and converse)

Convert for-comprehension into map/filter/flatmap chain (and converse)

Wrap parameter as Option (converting null checks, etc.)

Convert instanceOf/asInstance pair to match

Replace case clause with if body to guarded case clause(s)

I was particularly interested in those refactorings related to functional/higher-order programming and pattern matching. Between the surge of interest in Scala, F# and Haskell, it looks like there’s room for some more work in refactoring.

Py3k sprints

The call for sprints at this year’s PyCon 2009 is now up.

If you are involved with a Python project and you are going to be at PyCon this year, may I suggest that one topic for your sprint be porting your code to Python 3000? This is the perfect opportunity to work with other people who will likely be doing the same, while having access to the Python 3000 developers. Transitioning to Python 3000 is one of the opportunities/challenges for the Python community in the next two years. More about that at the conference.

Register for PyCon 2009

Registration is now open for PyCon US 2009. Like last year, PyCon is being held in Chicago, and the official program looks pretty good. As is true with all conferences, the unofficial program (the hallway track) is always great at PyCon. I am also going to make a concerted effort in the Open Spaces portion of the conference.

There will be a week of sprints held after the conference, so if you are looking to get involved with a Python project, those are a great opportunity to get a quick start while sitting side by side with some of the project developers.

I’ve gone to quite a number of conferences due to my job responsibilities over the last few years, and PyCon is definitely my favorite conference.

Northwest Python Day

Northwest Python Day

This weekend I attended the Northwest Python day. Actually, more properly, I attended the afternoon half of Northwest Python day. We’ve had some illnesses making the rounds in our house, and we had to do some creative schedule juggling on Saturday. In any case, the day was organized mostly by people from the Seattle Python User’s group, and run in a very low key fashion. The organizers managed to secure a decent size room at the University of Washington, and people worked together to bring some food and so forth. The program consisted of presentations that people volunteered to do – we had just enough to fill the day. A Python get together of any decent size just isn’t complete unless there are lightning talks, so there were two sets of lightning talks during the day.

My favorite talk of the day was William Stein’s talk about the Sage symbolic math system. Sage is a system in the flavor of Mathematica, Maple, or Matlab, and if you need to do any kind of symbolic math at all, this seems like something that is well worth looking at. One thing that I particularly liked was the use of a local web browser as a way of maintaining the worksheet/notebook workspace that is common to this class of applications. I was impressed to learn that Sage’s UI code actually contains a Javascript translation of the TeX layout engine for math, which is the gold standard for formatting math of any kind. Very cool stuff. I gave a talk on some of the Python projects that we have underway at Sun, and I tried to motivate my description of those projects with a bigger picture look at the state of the “market” for dynamic languages. The talk included some of the material that I used for my PyCon UK keynote and added some more Sun specific details.

This was my first conference of the year, and a good start.

dtrace -G on Mac OS 10.5 or dtrace for Python

Let’s suppose that you were trying to port dtrace probes from (Open)Solaris to Mac OS X, and the makefile for the probes on (Open)Solaris calls for the -G option, which isn’t recognized by dtrace in 10.5. You might want to check out this mail thread to find out what to do.   

The driver for this is an effort to port OpenSolaris’ dtrace probes for Python to run on OS X. One benefit of this will be that we’ll have access to John Levon’s ustack provider for Python on the Mac. If someone wants to tackle a port for FreeBSD 7.1, it would be great to get this support into all dtrace enabled platforms.

Python in NetBeans

Along with today’s launch of NetBeans 6.5, Sun, in cooperation with the NBPython community, are releasing an early access version of Python support for NetBeans. This is a result of the collaboration between Sun people and the NBPython project that I wrote about back in July. This release has been tested by folks in the NetBeans community and some folks from Sun’s NetBeans QA team, and it’s in pretty good shape for an early access release. We’re interested in getting people’s feedback. We would also love to see more people get involved with NBPython.

How to get it?

You can get NetBeans Python from the NetBeans download page.

What’s in it?

The basic feature set for the early access release consists of an editor for Python, the ability to execute Python programs (using CPython or Jython), and a debugger.

There’s a tutorial up on the NetBeans wiki.

Tor Norbye, who did most of the work on the editor, has written a series of blog posts detailing various features of the Python editor.

Who did it

Allan Davis – project and platform management, interactive console.

Jean-Yves Mengant – Jean-Yves is the author of the jpydbg debugger, which he’s merged into NBPython.

Amit Saha – documentation and help sets – Amit works for Sun, but he’s doing Python on his own time.

Tor Norbye (Sun) – editing.

Tomas Zezula (Sun) – project and platform management.

Ted Leung (me) (Sun) – various behind the scenes stuff.

Frank Wierzbicki (Sun) – NBPython is using Jython’s parser and Frank worked with Tor to add support for positions and better error reporting.

Peter Lam (Sun) – Sun QA

Tony Beckham (Sun) – Sun QA

The NetBeans CAT community as well as those folks who drove by and reported bugs.

How to get involved

NBPython has become a full fledged NetBeans project, so the main project page is now on NetBeans.org, as are the issue tracker and mailing lists:

nbpython-dev@netbeans.org
nbpython-issues@netbeans.org
nbpython-commits@netbeans.org
nbpython@netbeans.org

Book Review: Practical Django Projects

James Bennett, the release manager for Django has written a book called Practical Django Projects. At 237 pages, this book is even shorter than Learning Website Development with Django. Miraculously, Bennett manages to pack even more content into his book. He uses three different projects to illustrate the basic concepts of using Django. Early on in the book you are introduced to Django’s generic views mechanism, which makes it very easy to do the kinds of displays that are common in many web applications. Along with generic views, there’s a good treatment of how to keep functionality separated and reusable by good design of URLs. There’s also a much stronger treatment of the templating aspects of Django. He covers template inheritance, but also covers the creation of custom template tags.

Like “Learning Website Development with Django”, “Practical Django Projects” was completed before Django 1.0 shipped, so there are going to be some differences between what is described in the book and what you’ll encounter with Django 1.0. I like both books, and I’m probably biased by the order in which I read them. Either book would be fine for someone learning Django from scratch, but I think that “Practical Django Projects” is the book that I would turn to first when I couldn’t recall how to do something. In addition to the extra topics covered (and that’s not entirely fair because “Learning Website Development with Django” also has a few areas that it covers better), there’s more of an application building focus in Bennett’s book. That emphasis appeals to me, and Bennett has lots of application architecture hints sprinkled throughout the book.

Django is a great piece of work and deserves very serious consideration by people looking to build web applications. I’m glad to see that publishers are getting Django books out into the world.

Lua in Lightroom

One of the points that I discussed in my PyConUK keynote was the role (or lack thereof) of dynamic languages in desktop software development. I know that some people believe that desktop software is dead, but I am not one of them. I think that the nature of desktop software is changing, but that it is still relevant. In my discussion of dynamic languages and desktop software, I used the example of Adobe’s Lightroom, as the best example of commercial software that is written in a dynamic language. Lightroom is written in Lua, and according to this presentation by Troy Gaul, 63% of the code written by the Lightroom team is Lua (there is a bunch of C code for the low level image processing code, which is shared with Adobe Camera Raw). A number of people at PyConUK had no idea of the role of Lua in Lightroom, which is unsurprising. The only reason that I knew was that I was following Lightroom as a photography nut, and the information flew by on some boards. Troy’s presentation is pretty illuminating, so if you are interested in dynamic languages in desktop applications, you should watch the whole thing.