Ted Leung on the air
Ted Leung on the air: Open Source, Java, Python, and ...
Thu, 03 Feb 2005
The Wisdom of Crowds

Another book that I've recently finished is James Surowiecki's "The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations" Julie has posted her review. Get ready for some Siskel and Ebert...

This is the sort of book that sounds interesting to someone interested in decentralization in any form and commons-based peer-production in particular. There's even a story in the book about Linux.

The first five chapters of the book are the strongest, where Surowiecki tries to ferret out the principles and situations in which groups of people are wise/smart/effective. After that (over half the book), I felt that he kind of lost focus on the topic. While many of the anecdotes in the rest of the book are interesting, I felt a little shortchanged by the end.

I was able to find lots of interesting stuff in the good chapters, though:

(these page numbers are from the Large Print edtion, which is what the library held for me)


The market was smart that day because it satisfied the four conditions of wise crowds: diversity of opinion (each person should have some private information, even if it's just an eccentric interpretation of the known facts), independence (people's opinions are not determined by the opinions of those around them), decentralization (people are able to specialize and draw on local knowledge), and aggregation (some mechanism exists for turning private judgments into a collective decision)

These four conditions are supposed to be necessary for wise crowds. One problem of the book is that while Surowiecki describes what these conditions are, there isn't much practical advice on actually achieving these conditions.


Diversity helps because it actually adds perspectives that would otherwise be absent and because it takes away, or at least weakens, some of the destructive characteristics of group decision making. Fostering diversity is actually more important in small groups and in formal organizations than in the kinds of larger collectives -- like markets or electorates -- that we've already talked about for a simple reason: the sheer size of most markets, coupled with the fact that anyone with money can enter them (you don't need to be admitted or hired), means that a certain level of diversity is almost guaranteed.


The fact that cognitive diversity matters does not mean that if you assemble a group of diverse but thoroughly uninformed people, their collective wisdom will be smarter than an expert's. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, your better off entrusting it with major decisions rather than leaving them in the hands of one or two people, no matter how smart those people are.


Wharton professor J. Scott Armstrong wrote, "I could find no studies that showed an important advantage for expertise". Experts, in some cases, were a little better at forecasting than laypeople (although a number of studies have concluded that nonpsychologists, for instance, are actually better at predicting people's behavior than psychologists are), but above a low level, Armstrong concluded, "expertise and accuracy are unrelated".


The positive case for diversity, as we've seen, is that it expands a group's set of possible solutions and allows the group to conceptualize problems in novel ways. The negative case for diversity is that diversity makes it easier for a group to make decisions based on facts, rather than on influence, authority, or group allegiance. Homogeneous groups, particularly small ones, are often victims of what the psychologist Irving Janis called "groupthink"

Diversity is a strong part of what makes crowds wise. I particularly like the part about decisions made based on "facts, rather than on influence, authority or group allegiance". When I read the part about group think, I wondered, "are smug lisp weenies suffering from groupthink?"


Independence is important to intelligent decision making for two reasons. First it keeps the mistakes that people make from becoming correlated. Errors in individual judgment won't wreck the group's collective judgment as long as those errors aren't systematically pointing in the same direction. One of the quickest ways to make people's judgment's systematically biased is to make them dependent on each other for information. Second, independent individuals are more likely to have new information rather than the same old data everyone is already familiar with.

Rephrased, independence is essential because it reinforces diversity of opinion and information.

p. 98

They're assuming that John Maynard Keynes was right when he wrote, in The General Theory of Employment, Interest, and Money, "Worldly wisdom teaches that it is better for reputation to fail conventionally than to succeed unconventionally."

In the computer business, we say "Nobody ever got fired for buying IBM".

p. 138

In terms of decision making and problem solving, there are a couple of things about decentralization that really matter. It fosters, and in turn is fed by, specialization -- of labor, interested, attention, or what have you. Specialization, as we've known since Adam Smith, tends to make people more productive and efficient. And it increases the scope and the diversity of the opinions and information in the system (even if each individual person's interests become more narrow).

Decentralization is also crucial to what the economist Friedrich Hayek describe as tacit knowledge. Tacit knowledge is knowledge that can't be easily summarized or conveyed to others, because it is specific to a particular place or job or experience, but it is nonetheless tremendously valuable. (In fact, figuring out how to take advantage of individuals' tacit knowledge is a central challenge for any group or organization.) Connected with this is the assumption that is at the hear of decentralization, namely that the closer a person is to a problem, the more likely he or she is to have a good solution to it.

p. 139

Decentralization's great strength is that it encourages independence and specialization on the one hand while still allowing people to coordinate their activities and solve difficult problems on the other. Decentralization's great weakness is that there's no guarantee that valuable information that is uncovered in one part of the system will find its way through the rest of the system. (which is why blogs, etc are important)

The usual "decentralization is good" stuff.


What you'd like is a way for individuals to specialize and to acquire local knowledge -- which increases the total amount of information in the system -- while also being able to aggregate that local knowledge and private information into a collective whole, much as Google relies on the local knowledge of millions of Web-page operators to make Google searches ever-smarter and ever-quicker.

p. 146

Aggregation -- which could be seen as a curious form of centralization -- is therefore paradoxically important to the success of decentralization.

Aggregation is really the hard part. It's what Wiki technology does for the Wikipedia, you could argue that it's an important next step for blogging/microcontent. In open source projects this is accomplished at the code level by the version control system (see these articles about the impact of Bitkeeper), and at the human level by individuals or groups depending on the specific open source tribe.

p. 363

Similarly, people who have more extreme positions are more likely to have strong, coherent arguments in favor of their positions and are also more likely to voice them.

This matters because all the evidence suggests that the order in which people speak has a profound effect on the course of a discussion. Earlier comments are more influential, and they tend to provide a framework within which the discussion occurs. As in an information cascade, once that framework is in place, it's difficult for a dissenter to break it down.

p. 365

In a market or even a democracy, champions are far less important because of the sheer number of potential decision makers. But in a small group, having a strong advocate for an idea, no matter how good it is, is essential. And when advocates are chose, as it were, on the basis of status or talkativeness, rather than perceptiveness or keenness of insight, then the group's chance of making a smart decision shrinks.

p 365

Talkativeness may seem like a curious thing to worry about, but in fact talkativeness has a major impact on the kinds of decisions small groups reach. If you talk a lot in a group, people will tend to think of you as influential almost by default. Talkative people are not necessarily well liked by other members of the group, but they are listened to. And talkativeness feeds on itself. Studies of group dynamics almost always show that the more someone talks, the more he is talked to by others in the group. So people at the center of the group tend to become more important over the course of a discussion.

These last three quotes are more about group dynamics, which is one of the more "operational" discussions in the book. We all have been in situations where people speak more than they ought to. I tend to be the opposite -- I speak infrequently and usually towards the end of conversations, so it was interesting to see the impact of early and talkative people on the wisdom of a crowd. I'll certainly be paying more attention in the groups I'm a part of.

[23:31] | [books] | # | TB | F | G | 1 Comments | Other blogs commenting on this post
Programming Languages as cars

[via Phil Windley's Technometria | Java vs. C++ ]:

Java is preferable to C++ in exactly the same way that driving a 1994 Chevy Impala is preferable to driving a 1978 Ford LTD.

And of course neither is anything like a BMW, Acura, Porsche or Ferrari.

[22:22] | [computers/programming] | # | TB | F | G | 3 Comments | Other blogs commenting on this post
A G5 at last - a Shuttle G5 that is.

For work I got a new Intel based PC, a Shuttle G5 9500 XPC. This is an AMD64 nForce3 based machine. Mine has an AMD64 3500, 1Gig of RAM and 74G 10000RPM Raptor. It's not the top of the line, but it's pretty close. The form factor is great (goes right under one of the shelves of my desk), the heat is nice in the winter, and the noise is tolerable (but could be better)

I've spent the last few days wrestling with the machine. To do the job, the machine needs to run Windows XP, Fedora Core n (which is our supported Linux) and Ubuntu (because I'm willing to do the work to get things going on Ubuntu).

I started by grabbing the latest Hoary AMD64 Live CD, in order to make sure that Ubuntu could detect all the hardware. I had some concerns about the SATA disks and the Ethernet controller on the nForce. The LiveCD's found everything, put up a beautiful GNOME desktop, and seemed to run just fine. Next stop Hoary Install CD for AMD64. This installed like a champ, but wouldn't boot. I got dropped into the Intel PXE netboot. Something was messed up with the boot records. A frenzy of wiping, repartitioning, reinstalling ensued punctuated by reboot after reboot (I ended up blasting Windows too). I learned about Debian From Scratch, which is a handy way of using Debian as a rescue CD.

In the end I put Ubuntu on a primary partition (it had been on an extended paritition), installed GRUB's master boot record on that partition, and then copied that boot record to Windows, where I could put it into boot.ini and use the Windows boot loader to load GRUB which would then boot Ubuntu. I had never done anything with GRUB before Ubuntu, and I now know more about it that I wanted to. But at least I have a working dual boot system.

Next problem, building Chandler on AMD64. This turned out to be too big of a hurdle for a one night project. Installing the ia32-libs package helped some, but not enough. The big remaining problem seems to be weirdness with Hoary's current packageing of gcj-3.4, which is needed for PyLucene. I was tempted to go do the chrooted 32bit library thing, but I think I need to learn a little more about 64 vs 32 bit environments before I do that. I ended up grabbing a 32bit i386 Hoary CD and installing that into the partition intended for Fedora Core 3. After I did that I was able to successfully get the prebuilt Chandler to run (I have no doubt I'll be able to get it to compile if I need to). The last snag is that Chandler/wxWidgets runs great on the Xorg X xserver in Ubuntu, but when I try to run it using XQuartz on the Mac, Chandler crashes. So, for native Linux, we're AOK, for the case which would be more convenient for my working, there's a bit more to go.

On the whole, I'm happy with how the Ubuntu install went -- much better than my previous Debian installs (and most of my Debian knowledge carried right over), and without many cuts. In fact, when I just made one huge Ubuntu partition, things went really easily. The problem came in with me wanting a triple (perhaps now I'll need quadruple) boot setup, being inexperienced with GRUB, and trying to build a 64 bit Chandler, which I'll eventually figure out how to do.

[02:44] | [computers/operating_systems/linux/ubuntu] | # | TB | F | G | 10 Comments | Other blogs commenting on this post

twl JPG


Ted Leung FOAF Explorer

I work at the Open Source Applications Foundation (OSAF).
The opinions expressed here are entirely my own, not those of my employer.

Creative Commons License
This work is licensed under a Creative Commons License.

Now available!
Professional XML Development with Apache Tools : Xerces, Xalan, FOP, Cocoon, Axis, Xindice
Technorati Profile
PGP Key Fingerprint
My del.icio.us Bookmarks
My Flickr Photos

RSS 2.0 xml GIF
Comments (RSS 2.0) xml GIF
Atom 0.3 feed
Feedburner'ed RSS feed

< February 2005 >
   1 2 3 4 5
6 7 8 9101112


Macintosh Tips and Tricks

Blogs nearby
geourl PNG

/ (1567)
  books/ (33)
  computers/ (62)
    hardware/ (15)
    internet/ (58)
      mail/ (11)
      microcontent/ (58)
      weblogs/ (174)
        pyblosxom/ (36)
      www/ (25)
    open_source/ (145)
      asf/ (53)
      osaf/ (32)
        chandler/ (35)
        cosmo/ (1)
    operating_systems/ (16)
      linux/ (9)
        debian/ (15)
        ubuntu/ (2)
      macosx/ (101)
        tips/ (25)
      windows_xp/ (4)
    programming/ (156)
      clr/ (1)
      dotnet/ (13)
      java/ (71)
        eclipse/ (22)
      lisp/ (34)
      python/ (86)
      smalltalk/ (4)
      xml/ (18)
    research/ (1)
    security/ (4)
    wireless/ (1)
  culture/ (10)
    film/ (8)
    music/ (6)
  education/ (13)
  family/ (17)
  gadgets/ (24)
  misc/ (47)
  people/ (18)
  photography/ (25)
    pictures/ (12)
  places/ (3)
    us/ (0)
      wa/ (2)
        bainbridge_island/ (17)
        seattle/ (13)
  skating/ (6)
  society/ (20)

[Valid RSS]

del.icio.us linkblog



Listed on BlogShares

Locations of visitors to this page
Where are visitors to this page?

pyblosxom GIF