The Wisdom of Crowds

The phrase The Wisdom of Crowds was popularised by James Surowiecki in his excellent book book of the same name. So, exactly what is it?

Th Wisdom of Crowds is an idea that at first sight seems counter-intuitive: a group of people can have a collective intelligence far greater than the average of its members. In particular a large group of "ordinary" people can be smarter than a small group of specialists. Of course, there may well be individuals who could get better results than the crowd. The problem is that we usually can't identify these individuals in advance, especially since their identity might change from issue to issue.

Information Theory

How can this be? Essentially it's because we all have flaws, preconceptions and blind spots. Put enough people together and statistically these errors are likely to cancel out - the noise is eliminated to leave only the signal.

A well-known example of this crowd wisdom is "Ask the audience" slots in certain quiz shows. The audience often has a surprisingly high success rate even with questions that seem rather obscure. There are problems with this from a statistical viewpoint - for instance, the player is likely to ask the audience quesitions they can be expected to answer - however it does seem to supply good anecdotal evidence.

Of course it's not quite as simple as that. The members of the crowd need at least some familiartity with the relevant universe of discourse. A crowd of random people trying to design a new aircraft would be a disaster!

It's also necessary to ensure that the members of the crowd are reaching truly individual decisions. The "herding instinct" is a big problem as people are liable to make decisions in order to go along with their peer group or to please authority. To be wise a crowd must be diverse, independent and decentralised with their individual opinions aggregated.

The wisdom of crowds theory is sometimes called upon to support unfettered market economics, the marketplace considered as the ultimate crowd. Personally I consider that a deeply flawed argument. It might be true in a theoretically perfect market, but such a thing cannot exist - as a simple example, much branding, marketing and advertising sets out to reduce diverse, independent opinion and encourage herding.

Creativity

So, how does the Wisdom of Crowds relate to creativity? In a way it can be seen as an extension of the Law of Requisite Variety: one way to solve a problem is to be open to input from as many different sources as possible.

On an individual basis we do this without thinking about it. If we have a difficult decision to make we will talk it over with friends and get a feel for their consensus view. If this discussion takes place in the pub then it very often also results in ways to end famine, cure cancer and usher in a new era of global peace! However even the wisdom of crowds has its limits.

In a business setting the wisdom of crowds shows us the potential for opening up decision making processes more widely than the traditional group of experts. For example, perhaps engineering and marketing really do have something more to offer each other than problems. Getting together a large group of employees with adequate yet diverse background experience could provide insights and ideas that the more experienced experts might miss.

Of course this can be hampered by corporate culture. If the "crowd" feels compelled to follow "conventional wisdom" or support the boss's pet projeect then it will quickly degenerate from wise to foolish.