Wisdom of the Crowds on House?

This past week, I was watching one of my favorite TV shows House, and I was surprised when they mentioned The Wisdom of the Crowds (WOC). For the most part, the show got the main points of the theory correctly; i.e., the aggregation of knowledge from many diverse people can perform better than a single expert opinion. In the scene, the patient does not trust his doctors and posts a $25,000 reward to the person who first presents the correct diagnosis. After receiving many different opinions, he aggregates them, takes the diagnosis most often given, and demands treatment. Of course, this fails and everyone is given the impression that the theory is faulty. One problem - this is not a true example of the theory.

The problem is the style of the monetary reward - it encourages quackery opinions to stand-out more than normal. A fundamental principle that is often ignored when discussing WOC is the incentives for offering opinions are often small and are rarely monetary or form transitional wealth. An example of transitional wealth is money: I can give someone a dollar and they can now use that dollar to buy something. Non-transitional wealth only possesses value to the person who created it. An example is pride: I can create pride in myself and I may be able to find pride in the actions of others, but I cannot be proud and give that pride to someone else. Another good example is Last.fm. By labeling songs, I can recover them quicker, but someone else may find my choice in labels confusing, wrong, or meaningless. The wealth is entirely within me.

The reason it is important to monetize opinions and effort in this fashion is that it acts as a noise filter; i.e., The Wisdom of the Crowds does not mean every nut delivers an opinion. For example, if I poll the average person on the street (at least in America), I will find some very wrong answers. Most people do not know what a molecule really is. This might explain why many people have a fear of things with "chemicals," which is a rather ridiculous statement that I have previously ridiculed. Further, only 1/3 of people can state what the purpose of DNA is. In other words, the average opinion is, well, average.

We can actually think of WOC from a statistical sampling perspective. While it's true that on average the group opinion will be better than an individual, it does mean that this is occurs with probability of one. The purpose of the WOC is to remove bias, but not expertise. Or from a Bayesian perspective:

P(correct answer AND crowd) = P(answer|crowd)P(crowd)

If I have a bunch of creationists, I am not going to get the correct answer (evolution). Same for medical diagnosis. If I tell people that they'll get $25,000 if they guess the correct answer, then their monetary gain is high. Unless their monetary cost is high as well, they'll offer any crap-based solution they could get from a little bit of time on Wikipedia. Worse, in this particular example, it's the cranks and quacks who are more likely to respond. Who do you think will respond quicker to that reward, a guy going around selling reflexology out of his Winnibego or the Head of Infectious Diseases at Johns Hopkins?

However, I must admit that my ears perked when I heard the phenomena cited on a primetime TV show.