Tag Archives: erlang

CouchCamp 2010

I spent a few days last week at CouchCamp, the first mass in-person gathering of the community around CouchDB. There were around 80 people from all over the world, which is pretty good turnout. The conference was largely in unconference format although there were some invited speakers, including myself.

I think it says a lot about the CouchDB community that they invited both Josh Berkus and Selena Deckelmann from Postgres to be speakers. The “NoSQL” space has become quite combative recently, so it is great to see that the CouchDB has connections to the Postgres community, and respect for the history and lessons that the Postgres folks have learned over the year. Josh’s talk on not reinventing the wheel was well received, and his discussion of Joins vs Mapreduce took me back to my days as a graduate student in databases. His talk made a great lead in for Selena’s talk on the nitty gritty details of MultiVersion Concurrency Control

There were lots of good discussions on issues related to security and CouchApps, but the discussion that got my attention the most was Max Ogden’s discussion on the work that he is doing to open up access to government data, particularly around the use of location information. He’s been using GeoCouch as the platform for this work. In the past I’ve written about the importance of a good platform for location apps, particularly in the context of GeoDjango. GeoCouch looks to be a very nice platform for location based applications. This is a very nice plus for the CouchDB community.

These days, it’s impossible to be at a conference that involves Javascript and not hear some buzz about Node.js. As expected, there was quite a bit of it, but it was interesting to talk to people about what they are doing with Node. Everything that I heard reinforces my gut feel that Node.js is going to be important.

I was one of the mentors for the CouchDB project when it came to the Apache Software Foundation, and I was asked to speak about community. The CouchDB community has accomplished a lot in the last few years, and is doing really well. I prepared a slide deck, but didn’t project it because my talk was the last talk of the conference, and we wanted to do it in the outside amphitheater. I also wanted to tune some of the sections of the talk to include things that I observed or was asked about during the conference. The biggest reason that I prepared slides was to show excerpts of Noah Slater’s CouchDB 1.0 retrospective e-mail. A lot of what I think about community is summarized well in Noah’s message, and the note summarizes the state of the community better than I could have done it myself. I hope that we’ll be hearing more testimonials like Noah’s in the years to come.

Erlang Factory 2009

I spent Thursday and Friday of last week at the Erlang Factory in San Francisco (although the event was actually in Palo Alto).

Why did I go?

I’ve written about Erlang in this space before. Erlang is having a major influence on other languages, such as Scala on the JVM side and Axum on the CLR side. In addition every language seems to have several implementation of Erlang style “actors” (despite the fact that this is historically incorrect). Erlang has been around for a long time, and has seen industrial usage in demanding telecom applications. As a dynamically typed functional language with good support for concurrency and distribution, it is (if nothing else) a source of interesting ideas. Earlier this year, my boss asked me to start doing some thinking about cloud computing in addition to the stuff that I was already doing around dynamic languages — another good match for Erlang. This was the first large scale gathering of Erlang people in the US (at least that I am aware of), so I wanted to drop in and see what was going on, what the community is like, and so on.

Talks

The program at the Erlang Factory was very strong. In many of the slot sessions, there were 3 excellent talks to choose from. Every single talk that I went to was of very high quality. It was so bad that I wasn’t able to explore all the areas that I wanted to. Fortunately, the sessions were videotaped and are supposed to be made available on the web. Also, there was a decent amount of twittering going on, so a Twitter search for #erlangfactory will turn up some useful information.

I attended a number of “experience” talks by companies / individuals. There were experience talks from Facebook, SAP, Orbitz, and Kreditor (the fastest growing company in Sweden). I made it to the Facebook talk and the Kreditor talk. Facebook’s usage/deployment is on the order of 100 machines, which provide the chat facility for Facebook. Erlang is doing all the heavy lifting, and PHP is doing the web UI part. There was a lot of this kind of architecture floating around the conference. It seemed like the most popular combination was Ruby/Erlang, but there was definitely Python and PHP as well. The Kreditor talk was interesting because their site has been running for 3 years with very small amounts of downtime. Unfortunately, their entire deployment is probably less than 10 machines, so that blunts the impressiveness of what they have done. Still it was interesting to hear how they accomplished this using features of Erlang. In addition to the talks, I spoke with many attendees who are using Erlang in their companies. One such person was eBay founder Pierre Omidyar, who is running Ginx, a web based Twitter client. Pierre is doing the coding and deployment of the site, and was well versed in the Erlang way of doing things. An interesting data point.

The Erlang community (like all communities) has it’s old guard. These are folks who have worked with Erlang for years, before its recent burst of interest. There were a pair of keynotes by Erlang long-timers Robert Virding (The Erlang Rationale) and Ulf Wiger (Mulitcore Programming in Erlang). Both of these talks shared a common trait — the speakers were pretty honest about what was good about Erlang, and where there were problems. Given how prone the computing business is to fashion, I found this to be refreshing. Virding talked about the reasons why Erlang is designed the way it is. He accepted the blame for inconstencies in the libraries, talked about the need to avoid the process dictionary, and agreed that “a char type is probably not wrong”. Wiger’s talk was about why parallelizing code is hard (even with Erlang). He used the example of parallelizing map to demonstrate this, and showed the use of the QuickCheck testing tool to aid in finding parallelism bugs. The Erlang version of QuickCheck was inspired by the Haskell version of QuickCheck, and it’s a very very useful tool. The adaptations for parallelism look very nice. It’s a shame that the Erlang version is commercial software. I don’t grudge the authors the right to charge money for their software, but I do think that this will hold back adoption of this important tool.

There were many talks on what I would describe as “cloud problems”. For example, Ezra Zygmutowicz’s “You got your Erlang in my Ruby” was really about how he built a self assembling cluster of Ruby daemon’s (Nanite), Dave Fayram and Abhay Kumar’s “Building Reliable Distributed Heterogenous Services with Katamari/Fuzed“, and Lennart Ohman’s “A service fail over and take-over system for Erlang/OTP”. Like PyCon, there was a lot of interested in eventually consistent databases/key-value stores/non-relational databases. Cliff Moon’s talk on dynomite (a clone of Amazon’s Dynamo system), was particularly encouraging because he was reaching out to other people in the audience (and there were a decent number of them) to try an consolidate all their efforts into a single project. From what I could tell, people seemed receptive to that idea.

CouchDB also fits into that last category of non-relational databases, but it gets it’s own paragraph. One reason is that I helped mentor the project through the Apache Incubator (and chauffeured those CouchDB committers who were present). Another is that CouchDB creator Damien Katz got a keynote. Third is that there was basically a CouchDB track on the second day of the conference. There was a lot of interest in CouchDB, and a lot of activity as well. I was told that some of the people who took the CouchDB training during the training days had actually submitted patches on the project already. Damien’s talk was not about the technical details of CouchDB, but about his personal journey to CouchDB, which included selling his house and living off his savings in order to see CouchDB come to life.

Activity has really picked up in the Erlang web framework space. In addition to Erlang Web, and Yariv Sadan’s Erlyweb, there is also Rusty Klophaus’ Nitrogen. Nitrogen focuses more on the UI side of the web framework, omitting any kind of data storage. It’s very easy to create an AJAX based user interface using Nitrogen, and there is nice support for Comet. As part of his presentation, Rusty showed his slides on a Nitrogen based webcast reflector. You specify the UI using Erlang terms, which then causes HTML/Javascript/etc to be generated, which caused a stir in part of the Twitter peanut gallery. I was mostly happy to see people focusing on solving the current generation of problems. My favorite web space talk was probably Justin Sheehy’s talk on Webmachine. I think that I prefer the description of WebMachine as a REST or HTTP toolkit. Webmachine gives you what you need to implement any HTTP method correctly, and then provided a set of callback functions that can be implemented to customize that processing to do actual work. One of the coolest things about Webmachine is it’s ability to visually show you that path taken in processing a particular HTTP request, and being able to inspect/dump data at various points in the diagram. It makes for a very nice demo.

There were not that many “language geek” talks. This contrasts with the early years of PyCon (at least for as long as I have attended) where there were quite a number. I missed Robert Virding’s talk on Lisp Flavored Erlang (but I saw some example usage in a CouchDB talk), because it overlapped the dynomite talk. I was able to attend Tony Arcieri’s talk on “Building Languages on Erlang (and an introduction to Reia)”. During the first part of his talk, Tony showed how to construct an Erlang module on the fly in the Erlang shell. He then discussed some tools which are useful to people trying to build languages on top of BEAM, the Erlang virtual machine:

  • Robert Virding has written leex, a lexical analyzer generator
  • yecc, a Yacc style parser generator is included in the Erlang distribution
  • the erl_syntax_lib library aids in constructing Erlang abstract syntax trees, which can then be compiled to Erlang bytecode.
  • Erlyweb contains the smerl (simple metaprogramming) library for creating and manipulating Erlang modules at runtime.

After that, he launched into a description of REiA. I’m not sure that I agree with some of the choices that he has made, but I am happy to see people experimenting with languages on top of BEAM, and in keeping with Erlang’s process model and the OTP infrastructure. One of the things that Tony mentioned was abandoning indentation based syntax. He wrote an entire postmortem on that experience in his blog. Python’s indentation based syntax has won me over and made me a fan, and I am sad to see that indentation syntax, blocks/closures, and expression orientation continue to be at odds.

Coda

It looks like Erlang is starting to find a home. Companies are using it in production. There are books starting to be written about it. Many (not all) of the things which make Erlang seem odd to “mainstream” programmers also appear in languages like Scala, Haskell, and F#. At the same time, Erlang has a long history of industrial deployment, albeit in a single (large) market segment. Many of the problems which we now face in large web systems (and the cloud): concurrency, distribution, high availability, and scalability are strengths for Erlang. Indeed, many of the people that I heard from or talked to basically said that they couldn’t solve their problem with any other technology, or that their solutions were dramatically simpler than the technologies that they already knew. Will that be enough to propel Erlang into the mainstream? I don’t know. I also don’t know if our current state of mainstreamness is going to remain. More and more I’m seeing an attitude of “let’s use the best tool for the job”, not only in languages, but in all parts of (web) applications.

There’s also the issue of the Erlang community itself.   Around 120 people showed up for the conference. As I mentioned previously, there are the folks who have been doing Erlang for years. Then there are the relative newcomers, who are web oriented/web savvy, and solving problems in very different domains than the original problem domain of Erlang and it’s inventors. Thus far, the two segments seem to be getting along fine. I hope that will continue — success or the potential for success has a tendency to bend relationships.

Notes on A History of Erlang

Joe Armstrong wrote a paper for last year’s HOPL-III conference on the history of Erlang. For some reason, I didn’t get a paper copy of those proceedings, and was too busy to notice their absence. Fortunately Lambda the Ultimate picked it up and supplied links to the paper and the accompanying presentation. Digging into the history of something like Erlang is always fascinating, and Armstrong has done a good job of explaining how Erlang came to be.

Here are a bunch of quotes on topics that I found interesting. I’ve grouped them into categories, but searching the PDF of the paper shouldn’t be hard if you want to know where they originated.

Sources of inspiration

Those familiar with Prolog will not find it at all surprising that Erlang has its roots in Prolog (mostly due to implementation reasons). What I did find interesting was the origin/history/viewpoint on the concurrency model

The explanations of what Erlang was have changed with time:

1. 1986 – Erlang is a declarative language with added concurrency.

2. 1995 – Erlang is a functional language with added concurrency.

3. 2005 – Erlang is a concurrent language consisting of communicating components where the components are written in a functional language.

Today we emphasize the concurrency.

Note that the word actor never appears in those descriptions. Indeed, the word actor does not appear in the paper at all. So for all the discussion about Erlang’s usage of the actor model, it appears that the Erlang folks independently duplicated many of the ideas for Hewitt’s Actors. I think that is kind of interesting.

Lisp and Smalltalk are cited as inspirations, but more for the implementation of the runtime than for any features in the language. I came away from the paper with the impression that Armstrong and his colleagues are not paradigm ideologues. They are trying to get the job done.

Reliability

There is a huge emphasis on reliability throughout the paper, supporting Steve Vinoski’s remarks about Erlang. I’l just include a series of quotes, which you can interpret as you see fit:

Erlang was designed for writing concurrent programs that “run forever”

At an early stage we rejected any ideas of sharing resources between processes because of the difficulty of error handling. In many circumstances, error recovery is impossible if part of the data needed to perform the error recovery is located on a remote machine and if that remote machine has crashed.

In order to make systems reliable, we have to accept the extra cost of copying data structures between processes and always make sure that processes have enough data to continue by themselves if other processes crash

The key observation here is to note that the error-handling mechanisms were designed for building fault-tolerant systems, and not merely for protecting from program exceptions. You cannot build a fault-tolerant system if you only have one computer. The minimal configuration for a fault-tolerant system has two computers. These must be configured so that both observe each other. If one of the computers crashes, then the other computer must take over whatever the first computer was doing.

This means that the model for error handling is based on the idea of two computers that observe each other. Error detection and recovery is performed on the remote computer and not on the local computer.

Links in Erlang are provided to control error propagation paths for errors between processes.

It was about this time that we realized very clearly that shared data structures in a distributed system have terrible properties in the presence of failures. If a data structure is shared by two physical nodes and if one node fails, then failure recovery is often im-possible. The reason why Erlang shares no data structures and uses pure copying message passing is to sidestep all the nasty problems of figuring out what to replicate and how to cope with failures in a distributed system.

In our world, we were worried by software failures where replication does not help.

Design criteria

Here are some quotes related the design criteria for Erlang.

Changing code on the fly was an initial key requirement

the notion that three properties of a programming language were central to the efficient operation of a concurrent language or operating system. These were:

1) the time to create a process

2) the time to perform a context switch between two different processes

3) the time to copy a message between two processes

The performance of any highly-concurrent system is dominated by these three times.

One of the earliest design decisions in Erlang was to use a form of buffering selective receive

Pipes were rejected in favor of messages

In the concurrent logic programming languages, concurrency is implicit and extremely fine-grained. By comparison Erlang has explicit concurrency (via processes) and the processes are coarse-grained.

The final strategy we adopted after experimenting with many different strategies was to use per-process stop-and-copy GC. The idea was that if we have many thousands of small processes then the time taken to garbage collect any individual process will be small.

Current systems run with tens to hundreds of thousands of processes and it seems that when you have such large numbers of processes, the effects of GC in an individual process are insignificant.

The BEAM compiler compiled Erlang programs to BEAM instructions.

On functionalness

This next series of quotes will probably make the pure functional language people shake their heads, but i think that it’s important to understand Erlang in contrast with pure functional languages.

Erlang is not a strict side-effect-free functional language but a concurrent language where what happens inside a process is described by a simple functional language.

Behaviors in Erlang can be thought of as parameterizable higher-order parallel processes.

… the status of Erlang as a fully fledged member of the functional family is dubious. Erlang programs are not referentially transparent and there is no system for static type analysis of Erlang programs. Nor is it relational language. Sequential Erlang has a pure functional subset, but nobody can force the programmer to use this subset; indeed, there are often good reasons for not using it.

An Erlang system can be thought of as a communicating network of black boxes.

In the Erlang case, the language inside the black box just happens to be a small and rather easy to use functional language, which is more or less a historical accident caused by the implementation techniques used.

History and Usage

One thing that I was looking for in the paper was more details on how long Erlang had been around (besides before Java), how big the largest programs/systems were, and so forth. Here is what I found.

This history spans a twenty-year period…

(The history starts in 1986)

The largest ever system built in Erlang was the AXD301. At the time of writing, this system has 2.6 millions lines of Erlang code.

The AXD301 is written using distributed Erlang. It runs on a cluster using pairs of processors and is scalable up to 16 pairs of processors.

In the analysis of the AXD reported in [7], the AXD used 20 supervision trees, 122 client-server models, 36 event loggers and10 finite-state machines. All of this was programmed by a team of 60 programmers.

As regards reliability, the AXD301 has an observed nine-nines reliability [7]—and a four-fold increase in productivity was observed for the development process [31].

The AXD 301 is circa 1998.

Perhaps the most exciting modern development is Erlang for multicore CPUs. In August 2006 the OTP group released Erlang for an SMP.

This corroborates something that David Pollak told me at the RedMonk unconference during CommunityOne, namely that SMP support in Erlang had not been there very long. Of course, Erlang was running on systems with 16 physical (pairs, no less) of processings in a distributed environment. So while the runtime might not be that mature on SMP, the overall runtime for concurrency is probably a bit more mature than that. Nonetheless, worthwhile to know the precise facts.

All in all, I found the paper to be a very worthwhile read – (and a nice change from my usual intake of blog posts and tweets). One of my pet peeves about the computer business is the lack of awareness of the history of the field. At least I’ve removed a bit of my own ignorance as relates to Erlang.

Erlang == CGI?

Jay Nelson, in the comments to Damien Katz’s Lisp as Blub:

The two relevant issues are system granularity and garbage collector behavior (if it is related to memory and garbage collection).

Erlang encourages an architecture of many small-granularity processes. To the extent that this approach is followed, failures are localized. It is possible to do this with other languages, but erlang does encourage the approach more so than other languages.

The other difference is that erlang uses a single-threaded garbage collector per process. This makes the garbage collection process simpler, more finely grained and distributed. Smaller processes mean less complicated memory structures, and thus the language encourages a simpler model with localized garbage collection failure. Determining the cause of overburdened memory usage (or any other resource because of the localized nature of small processes) becomes easier.

An erlang system can get wedged, but following the principle of many small processes makes it less likely to happen than in other languages which encourage large processes with shared memory structures.

It strikes me that this is a sort of CGI’ish view of the world (well except for the garbage collector). CGI scripts run, use (non-shared) resources, release them all and die. The entire post and comment thread is worth some pondering.

The Erlang community

Matt Croydon Didn’t agree with my commentary on the Erlang community, and he’s partially right. I shouldn’t have said “we need a community” because there is an Erlang community, and I knew that. I have never been a fan of Java, and I don’t want to be stuck using the moral equivalent of Java when the multicore/concurrency thing shakes out. So if I want to be able to use Erlang (and I’ve not totally made that decision), then it needs to have a bigger, more diverse, and easier to find community.

Scalability != concurrency

Sam Ruby is writing about Russell Beattie writing about Java and Erlang.

Russell thinks Java needs an overhaul. I think that Java has reached the point where technical, community, and business forces well exert pressure on the language to evolve in a uniformly bad manner.

Russell wrote:

The reason people are looking at Erlang is not because its beautiful syntax, great documentation, or up-to-date libraries. Trust me. It’s because the Erlang VM can run for long periods of time, scaling linearly across cores or processors filling the same niche that Java does right now on the server.

Actually, I am looking at Erlang as a solution for anywhere, (including the client) where concurrency will be an issue. By the way, it is not VM’s that scale linearly, but computational problems. And there are some problems which just can’t scale linearly, no matter what VM we put them on.

Sam goes on to make the point which is the title of this post.

Next, to dispel a few myths. Slashdot is written in Perl, seems to handle the load, and also seems to stay up. While there are a number of BitTorrent implementations, the original and (to the best of my knowledge) the most pervasive version is written in Python. Yahoo is a mix, but a good portion of it is written in PHP, with critical functions written in C. Twitter is written in Ruby, had early scalability issues, but seems to be past them. These are all examples of massively scalable applications.

Scalability is not the same thing as concurrency. It is certainly possible to scale a program written in any language – that’s a given. Especially when scaling = throwing more hardware at it. But there’s got to be a better way of doing it. Question is whether the better way is worth the price of admission.

But as far as Erlang vs Java, the real kicker is here:

Unlike the CLR which was designed to be multi-language, and unlike the JVM which is in the process of being repurposed to be multi-language also, Erlang’s VM is designed from the ground up assuming that objects typically are immutable and serializable.

Which is what makes the situation with Java so bad. Not only is the language bad, the VM is fatally flawed when it comes to actor style concurrency (which is why for all its niceties, Scala will suffer the same problems as Java). There’s a real problem here — ask yourself why there is a market for these things, if all that is needed is to throw even more boxes at the problem.

In the comments, Sam wrote;

The biggest problem I have with Erlang is clearly an addressable one: the documentation of the libraries, and the lack of good samples that can be quickly found by Google/MSN-Live/Yahoo!/Ask searches. And many of the libraries appear to be abandoned at 0.n versions.

This is actually 2 problems. There’s the issue with the libraries, and there’s the issue with the community that did/didn’t produce the libraries. We don’t just need a technology, we need a community. Hmm, Erlang lab, anyone?

Some simple thoughts on Erlang

Our reading group on Bainbridge Island has been working its way through Programming Erlang. Actually we’re technically not done yet, but since I spent a fair amount of time on the ferry recently, I went ahead and finished it off. There’s been quite a bit of writing about Erlang recently, and I wanted to at least have finished the book before jumping in. Looking at Joe Armstrong’s PhD thesis is probably soonish on my list too.

Basics
Erlang is a functional language which incorporates a concurrency model based on very lightweight processes communicating via messages. I’ll cover the concurrency model a bit more below. Since many people have not really been exposed to functional programming, there are things in Erlang which seem odd when compared to more mainstream languages. In addition, Erlang relies heavily on pattern matching as a flow of control construct, and it takes some time to get used to it. Some people liken the pattern matching aspects of Erlang to Prolog, but this is not entirely accurate because Prolog uses unification, which works in “both directions” and not pattern matching, which only works in “one direction”. I can’t say that I care for the syntax of Erlang, but after using Python, there are very few syntaxes that I really like. Erlang supports higher order functions, so closure based control flow structures are included. There is a fairly usual set of basic data types which are provided. Probably the biggest problem with the basics of Erlang is the way that strings are handled. In reality there are no strings in Erlang, and strings are just lists of integers. More on that below.

Concurrency
Much of the current interest in Erlang is due to its concurrent programming capabilities. The foundations of these capabilities are the availability of processes at the language level. Erlang allows a programmer to create and destroy processes quickly and cheaply (in terms of resources). Processes can only communicate with each other by sending each other unidirectional messages. Every process has a mailbox, which is where messages for it are delivered. The messages are queued there until the process explicitly “receives” them.
The code that implements a process typically consists of a tail recursive loop which explicitly “receives” messages and uses pattern matching to examine the messages and dispatch to the correct behavior. Replying to the sender of a message must be handled by the programmer, but it is easy to code up simple rpc style message passing. Two (or more) processes can be linked to each other so that when one process dies, the other is sent a signal. The preferred mode of handling errors in processes is to kill them and restart them. This signaling forms the basis of the supervision tree concept in OTP. The basic concurrency model of Erlang is a version of the Actor model developed by Carl Hewitt at MIT. I took Hewitt’s class while I was an undergraduate, so the concepts were familiar to me. Erlang is relatively blind to where a process might be running – in the same VM, in a different hardware thread on the same VM, or on a VM on different computer altogether. This makes it easy to write programs that can grow easily when you want to add hardware, whether that is processors or computers.

OTP/Mnesia/ETS/DETS
The folks at Ericsson have also provided a bunch of libraries to raise the level of abstraction for concurrent programming in Erlang. There are 3 major libraries. OTP (Open Telecom Platform) helps a programmer to write scalable, fault-tolerant code. It takes advantage of Erlang’s hot code update facilities to allow processes to be upgraded in place. The basic abstractions to do this are very simple to work with. OTP includes the notion of supervision trees, which is an abstraction for managing networks of processes.

Mnesia is a (potentially) distributed database written in Erlang. It provides an easy mechanism for storing Erlang terms. While it is not an RDBMS, it does provide a query mechanism based on list comprehensions. It also supports transactional behavior and has the ability to duplicate Erlang tables on other machines

Runtime
One thing that isn’t discussed enough are the features of the Erlang runtime/VM. The runtime is very efficient at managing processes, much more so that languages like Python, Ruby, or Java. Erlang programs have been deployed in telephone switching products for years, with extremely long uptimes – due in part to Erlang’s hot code swapping capabilities. Java’s hot code replacement or Python’s reload are substantially weaker than Erlang’s hot code swapping. So while libraries that provide an actor like model can help people learn a good programming model for concurrency, it’s less clear to me that the languages (and the implementations of those languages) hosting the libraries will be as good as Erlang when it comes to highly concurrent applications. Of course, if an application isn’t that concurrent it might not matter.

Conclusion
Semantically, there is a lot to like about Erlang – the actor based concurrency model, hot code swapping, higher order functions, and (once one gets used to it) pattern matching. The OTP libraries have been refined by many years of production usage in demanding, commercial applications.

At the same time, there are number of issues which I think are real barriers to Erlang adoption. The syntax will prove difficult for many people, which is a big issue. I’ve already mentioned the problems with string handling, and really that generalizes to a lack of libraries for performing 21st century / web computing tasks. The nice thing about telephone switches is that you don’t really have to talk to the world. But if Erlang is to be viable as a solution for mainstream programming as it moves to more concurrency, Erlang programs must be able talk to the environments around them.

I am aware of several projects where Erlang is being used to do the heavy server lifting and then data is being passed off to programs written in more familiar languages like Python, Ruby, or Java. Certainly this is one way that people could begin to exploit the benefits of Erlang without converting wholesale. It would also give the Erlang community some time to improve Erlang to the point where it could be adopted by a larger audience.