Category Archives: python

EuroPython 2008

Next week I’ll be at EuroPython 2008, in Vilnius, Lithuania, where I’ll be moderating a panel on Jython. If you have questions that you’d like to put to the Jython committers, leave them as comments on this post. I’ll make sure that I post answers for any of those questions.

After the conference ends, I’ll be spending a few days at Sun’s office in Prague, which is where much of the NetBeans team is located. I’ve never been to Vilnius or Prague, so suggestions for things to see, etc., would also be great comments.

DTrace on Linux?

I’ve been meaning to write a post about DTrace, and Tim Bray’s tweet finally got me moving. It looks like some people are trying to make DTrace a topic for this year’s Linux Kernel Summit. I hope they succeed. I also hope that those folks pushing for user level tracing have their voices heard. I was amused to read one of the messages which claimed that DTrace is:

DTrace is more a piece of sun marketing coolaid which they use to beat us up at every opportunity.

My experience at Sun thus far is that people generally don’t really appreciate the benefits of DTrace. It stems from a view that I also saw in the LKS threads, which is that DTrace (and tools like Systemtap) is a tool for system administrators, because it reports on activity on the kernel. That’s not how I look at it. DTrace is a tool for dealing with full system stack problems, which initially manifest themselves as operating system level problems. The fact that DTrace can trace user land code as well as kernel code is what makes it so important, especially to people building and running web applications. Because of all the moving parts in a complicated web application (think relational database, memcached or other caching layers, programming language runtime, etc), it can be hard to debug a web application that has gone awry in production. Worse, sometimes the problems only appear in production. Tools which cut across several layers of the system are very important, and DTrace provides this capability, if all the layers have probes installed. When a web application goes wrong in production, you see it at the operating system level – high usage of various system resources. That’s where you start looking, but you will probably end up somewhere else (unless you are ace at exercising kernel bugs). Perhaps a bad SQL query or perhaps a bad piece of code in part of the application. A tool that can help connect the dots between operating system level resource problems and application level code is a vital tool. That’s where the value is.

One of the cooler features of DTrace is that you can register a user level stack helper (a ustack helper), which can translate the stack in a provider specific manner. One cool example of this is the ustack helper that John Levon wrote for Python, which annotates the stack with source level information about the Python file(s) being traced. On an appropriately probed system, this would mean that you could trace the Python code of a Django application, memcached, and your relational database (PostgreSQL and soon MySQL). That would be very handy.

I’d love to see DTrace on Linux, because I have it on OS X and it’s in OpenSolaris and FreeBSD, but I’d also be happy to see SystemTap get to the point where it could do the same job.

OS X Scripting

John Gruber followed up on Daniel Jalkut’s suggestion that Apple replace AppleScript with Javascript:

I agree with this wholeheartedly. Or maybe even make a clean break and scrap OSA and introduce a new system.

I’ve been talking up the benefits of scripting apps on the Mac since the 1990’s. The sad fact of it is that Apple has never really supported scripting to the level that it deserves. It’s even more important now in the days of a UNIX based MacOS. I have a bunch of scripts that I rely on daily to help me get things done more efficiently. I’d write more of them, but two things hold me back. AppleScript is really a funky language. I’ve partially solved that by switching to using Python (via appscript) to do the scripting, but that’s only half the problem. The other half of the problem is that the API exposed via OSA is also pretty funky. If Apple cleaned all that up, in say, 10.6, I’d be happy to rework my existing body of scripts.

Even if that happened, the big problem is that developer’s don’t really support scripting that well, so a good scripting system overhaul needs to look at making it easy for developers to expose application functionality to scripts. Unless that part happens, improvements in the scripting language, and OSA’s API’s will not be enough to push scripting to the level where it belongs.

PyCon 2008

It’s been 2 years since I’ve been to PyCon, and things have definitely changed. The last time I went to PyCon (2006 in Dallas), it was still a relatively small conference (3-400), if I remember what I was told), with a familiar feel, especially if you had attended in previous years, or were a part of the Python community. This year, there were over 1000 people (double the 500 people that came in 2007, apparently). I spent a sizable portion of the conference days feeling like “I miss a year, and you guys go and get 1000 people”. It’s a great thing that so many people are interested in Python.

PyCon 2008: Day 1

The talks

I went to a reasonable number of talks – talk quality at PyCon has historically been pretty good, and I was a little out of date on the latest on things like Django and Turbogears. The best talk that I went to was Raymond Hettinger’s talk “Core Python Containers — Under the Hood”. This was a great talk for several reasons: Raymond was a good and entertaining speaker. There was significant technical meat – explanations of the implementation choices for all the core containers in Python, lists, sets, and dicts. We heard about doubling factors and amortized big-oh time. Most importantly, there was significant practical applications for Python programmers. Raymond’s talk gives a cost model for the core containers, and having an understanding of that model is important for folks who are writing Python programers. It’s also useful for developers of alternate Python implementations because it allows them to follow suit or to diverge and (hopefully) document the places where the cost model is different. My next favorite talk was Jim Baker’s “More Iterators in Action”. I missed the talk given last year, but I liked this one. Jim hit two of my favorite topics, language integrated query (LINQ) (albeit without the DSL), and concurrency.

Concurrency

There was a lot of interest in concurrency this year, which warms my heart, because I see high-level/dynamic languages and concurrency as the chocolate and peanut butter in the old Reese’s peanut butter cup commercials. There were 2 open space sessions and 1 lightning talk, and the topic entered many of the conversations that I had.

Sun, Jython, and JRuby

People were generally positive to learn about Sun’s interest in Python and Jython. A number of people stopped me to congratulate me on the new job, and we had a nice turnout at the open space session, where people were free with ideas, comments, and a few not so easy to answer questions. I hope that Sun can live up to the goodwill that people extended towards me and Frank.

If I was surprised about the jump in size of PyCon, I was even more surprised by the amount of energy around Jython. At most of the previous PyCon’s that I attended, people would mention Jython, and either be sorry that it was too out of date to consider, or be just plain dismissive of it. This year there was none of that. People were very interested in Jython. I was really surprised by how much interest there was, and by some of the people who were interested. It was certainly a nice feeling to sit in the sprint room and occasionally have people pop in to ask if such and such was running in Jython yet, or did Jython support X because package Y needed it.

This was the first time that I had met Frank Wierzbicki in person — I think he’s the happiest person at Sun right now. I was also able to spend some time hanging out with various folks from the Jython community. It seemed to me that the community was doing quite nicely. If you looked at some of the community metrics that we would use at the ASF to allow a project to graduate from incubation, almost all of those criteria have already been fulfilled. One of my goals for Sun’s Python efforts is for as many of them as possible to be highly community oriented, so it was nice to see that Jython is well on it’s way in that regard. The folks working on Jython are very sharp (including the aforementioned Jim Baker, who it turns out was a classmate of mine at the Brown CS dept – although neither of us can remember meeting the other), and have one of the those (in my mind) essential community ingredients, a community sense of humor.

PyCon 2008: Day 3

Jim snuck this bit of commentary on Jython’s lack of a global interpreter lock into his talk.

There were several Ruby related surprises at PyCon this year. David Heinemeier Hansson, create of Ruby on Rails, made an appearance for one day, and a number of the JRuby committers made a road trip down from Minnesota, to hang out, meet the Jython folks, and generally display their hacker prowess. Which they totally did. Charlie and Tom powered their way to JRuby 1.1RC3 during the conference. Meanwhile Nick Sieger demonstrated what a happens when you stick a bunch of hackers, an EVDO card, and an EVDO hub into a car. The Jython guys (if any of them lived in the same state) need to get some of that – The best thing since the Adobe AIR Bus Tour, and at a fraction of the cost.. The JRuby folks and Jython folks are already starting to talk and share experiences, and I am sure that this will only result in even better dynamic language stuff for the JVM.

Other Cool stuff

On one of the sprint days, I did a bit of wandering and stopped to talk to my friend Brian Dorsey, who is doing some cool stuff here in Seattle. Brian was working with Richard Jones on pyglet and Bruce. Pyglet is a set of Python libraries for writing games and doing other kinds of multimedia. There’s pygame, which I am aware of because of Armin Rigo’s infamous use of pygame to deliver talks about PyPy. Richard has created Bruce, a presentation tool based on pyglet. In addition to being able to do cool multimedia presentation effects, there are some really cool things that you can do. Perhaps the coolest is that you can have a slide which is essentially an embeddd Python interpreter, so no more switching out of your presentation to demo your Python code at work. Really slick.

On a different note, on several evenings, conference goers who stuck around to hang out in the hotel’s common area were treated to musical performances by a dynamic (as in constantly changing set of members) band of Pythonistas:

PyCon 2008: Day 2

PyCon 2008: Day 2

The Sprints

Perhaps the most amazing way in which the conference has changed is this picture.

PyCon 2008: Sprints

That is a picture of a part of the lunch crowd on the first day of the sprints following the conference. When I talked to David Goodger about it, he said that he had taken a count and there were over 250 people at the sprints. Visually, that sprint lunch room looked to be about the size of the room for the first PyCon that I attended (PyCon 2004). Simply amazing. The ASF has a hackathon before every ApacheCon, but I can’t remember one ever reaching this kind of size or scale. Another thing about the PyCon sprints is that they are aimed at growing the community — you don’t have to be a committer on any project in order to attend, and experienced project members will take time to sit and help new people get started. There were several people like that in the Jython sprint room. I was more impressed by what happened with the sprints than any other part of the conference. The only central organization here was that the conference planners obtained sprint space, and in a few cases got some sponsors to cough up money for lunch. Everything else was organized by the projects themselves (I heard that the Django folks closed 100 bugs in a single day). If you want to get a sense of what kinds of things got accomplished at the sprints, you can look at this page on the wiki — it’s not exhaustive, but it’s a start.

Travel

In the past, I’ve had some travel nightmares getting home from PyCon. This year I am happy to report that I didn’t have any problems at all, except for a fight that I had with the Sun internal travel system (and lost).

Conclusion

It was great to be back at PyCon. Interest in Python is growing (as measured by attendance), as is interest in Jython, and interested people are also rolling up their sleeves and pitching in (as measured by sprint attendance growth).

PyCon: Open Space with Sun folks

For those of you at PyCon, Frank Wierzbicki and I will be hosting an open space session on Saturday (tomorrow) at 2pm for people to come and tell us what they think Sun should do in the Python space. We are definitely interested in input and feedback from the larger Python community. If you aren’t at PyCon but have ideas, you can drop either of us e-mail. Our Sun e-mail addresses are <firstname>.<lastname>@sun.com.

If you think my job is all about Jython, you are confused.

Apparently people are confused about what I am working on at Sun, and with PyCon starting tomorrow, this is not a good thing. I am not going to be working on Jython directly, although I will certainly be poking my nose in to see what’s going on. The Python related part of my job (which will be the majority in the short term) is to figure out what Sun should be doing in the Python space, across all of the relevant platforms at Sun, including but not limited to: the JVM and JEE, Netbeans, and Solaris.

Sun isn’t done in the dynamic language space, and I will also be looking for opportunities with other dynamic languages and related technologies.

The Sun is going to shine on Python

Today is my first day as a Sun employee.

How?

Tim Bray was one of the first people to respond to my “looking for a job” blog post. I have not written about it much, but I’ve been very impressed with how Sun has handled the JRuby project. Tim told me that Sun was interested in ramping up their support for Python in a similar fashion, and asked if I would be interested in coming to Sun to lead such an effort.

Why?

After a bunch of talking and interviewing and so forth, it turns out that I was very interested. Long time readers know that I am a dynamic languages guy, going back to the original dynamic language, Lisp. I spent 2.75 of the last 4 years at OSAF working on a big desktop application written in Python (Contrary to some recent blog posts, Python was not a factor in the difficulties that we had with Chandler). The prospect of doing something that would help Python was very attractive. However, Sun has been slow to embrace dynamic languages (whether atop the JVM or not), and Sun’s history in open source has been somewhat checkered in my view. So there were some questions that I had to answer for myself before deciding to go to Sun (especially since I had 3 other very good options):

1. Can Sun actually work with an open source community?

It’s no secret that I have not been a fan of Sun’s handling of the open sourcing of Java, and it seems like OpenSolaris is having some governance problems of its own at the moment. However, if you look at the way that JRuby has been handled, you’ll see that there are parts of Sun that are learning how to work with a community, and doing a very good job of it. Sun hired two of the leading JRuby contributors and gave them license to keep doing what they had been doing. The JRuby guys have been well received by the “C” Ruby community and even the CLR/.NET Ruby community. In addition Sun has been investing in Ruby via support in NetBeans and via some collaborations with the University of Tokyo on the C VM for Ruby. Over the years, I’ve met many people at Sun who understand a collaborative development style. Many of those folks are committers on Apache projects.

2. How serious is Sun about dynamic languages and how deep does that support go?

Sun is (finally?) very serious about this. As part of Sun’s new direction, Sun wants to give developers the ability to use whatever tool sets they want. Ruby, Python, PHP, Java. On or off OpenSolaris. On or off the JVM. There is an official project, John Rose’s DaVinci Machine project, to modify the JVM to support dynamic languages. As far as Python goes, Frank Wierzbicki, the maintainer of Jython, started at Sun last Monday, so there will be at least two of us working on Python related stuff. That includes Jython, Python support for Netbeans, and some other stuff that we haven’t quite figured out yet. We definitely will be looking for things that we can do to support CPython and the Python language as a whole. This is not just about Python on on the JVM. Sun will try to make its platforms, OpenSolaris and the JVM, the best place to develop and deploy Python applications. But at the moment that’s a goal and not a reality, so there is lots to do.

What’s Next?

Frank and I will be at PyCon in Chicago in a week or so. One of my goals (besides hooking back up with people since I missed PyCon last year) will be to sit down and talk to anyone who has ideas about sensible things that Sun could do to help Python. In the mean time, my e-mail address will be <FirstName>.<LastName>@Sun.com

Oh, one more thing. My new job title is “Principal Engineer, Dynamic Languages and Tools”, so expect to see me dinking around with other dynamic language stuff as well.

My thanks to Tim Bray for helping to make this happen.

Update:
It looks like it’s going to take a little longer to get my e-mail address fully operational…
Update 2:
Ok, e-mail is set and ready to go.