Why doesn’t brainstorming work?

From an interview with Jonah Lehrer, author of Imagine: How Creativity Works.

Why doesn’t brainstorming work? What should we do instead?

I think the failure of brainstorming is inseparable from its allure, which is that it makes us feel good about ourselves. A group of people are put together in a room and told to free-associate, with no criticism allowed. (The assumption is that the imagination is meek and shy — if it’s worried about being criticized, it will clam up.) Before long, the whiteboard is filled with ideas. Everybody has contributed; nobody has been criticized. Alas, the evidence suggests that the overwhelming majority of these free-associations are superficial and that most brainstorming sessions actually inhibit the productivity of the group. We become less than the sum of our parts.

However, in recent years, scientists have shown that group collaborations benefit from debate and dissent; it is the human friction that makes the sparks. (There’s a reason why Steve Jobs always insisted that new ideas required “brutal honesty.”) In fact, some studies suggest that encouraging debate and dissent can lead to a 40% increase in useful new ideas from the group.

You talk a lot about the benefits of cultural mixing. What legislative changes would encourage more of this?

More immigrants! The numbers speak for themselves. According to the latest figures from the U.S. Patent Office, immigrants invent patents at double the rate of non-immigrants, which is why a 1% increase in immigrants with college degrees leads to a 15% rise in patent production. (In recent years, immigrant inventors have contributed to more than a quarter of all U.S. global patent applications.) These new citizens also start companies at an accelerated pace, co-founding 52% of Silicon Valley firms since 1995.

Many of the anecdotes in Imagine have a disconcerting common theme of drugs or mental illness. Are creative people all doomed to be addicts or mad men?

I don’t think so. (Yo Yo Ma, for instance, is a very nice guy.) But I do think the prevalence of such stories reminds us that creativity is damn difficult, which is why those in the creativity business are always looking for every possible edge. That’s why many great writers experimented with amphetamines and why performers have always searched for compounds that let them get out of their head, silencing that voice that kills their spontaneity. In the end, of course, these chemical shortcuts rarely work out — there’s nothing creative about addiction. And that’s why I remained convinced that the best creativity booster is self-knowledge. Once we know how the imagination works, we can make it work better.

More Q&A at Mashable

 

Thx to Jesse Newhart

Twitter's edge is Big Data

Today Twitter announced that it purchased Backtype, a tiny company doing Big Data. The website says the company does “social media analytics” which is pretty much saying “Oprah sits on a couch.”

It’s the buzzy-est of buzz words, but if you dig into this you find that the company doing things that everybody wants. Like analyzing hits per link.

When a content creator shares their work the Backtype-created Storm product steps in. Giving you a realtime conversational graph, ability to search comments, and an influence score. Add to that comparison shopping through analysis of top sites and trending links.

But you may be asking, can’t I already do this?

Yes, you can, if you want it 24 hours later.

Stats on the web fall into two categories, instant low tech stats and delayed high tech stats. The market is saturated with the latter (delayed high tech stats) because the core innovations already exist. The reason is complicated but it boils down to the fact that today’s top hardware was built for the pre-Twitter/Facebook world.

To build for realtime processing requires a whole new set of operators where speed, size, and queries go supernova. To give you an example of this, see Backtype’s stats:

  • 100-200 machines (operating as one)
  • 100 million messages
  • 300 queries/second

This is the world that Twitter lives in, millions of messages/second. If you remember the early days of Twitter with all the downtime and Fail Whale messages, that was due to the technological limitations of the time.

They proved that quick, short messages are beloved by us humans to the tune of billions. Since then they have been massively scaling, customizing, and driving the industry. Not only do they need a way to process billions of messages without Fail Whal-ing, but they need to offer (paid) services on top of it.

This is where Backtype comes in. The team built a fascinating service on top of Twitter that does stream processing, continuous computation, and distributed RPC (remote queries of 100s of machines).

The simple translation of this is “live analytics”. The complicated version, pulled from Twitter’s:

“Imagine you have a cluster of 100 computers. Hadoop’s distributed file system makes it so you can put data…in…and pretend that all the hard drives on your machines have coalesced into one gigantic drive….it breaks each file you give it into 64- or 128-MB chunks called blocks and sends them to different machines in the cluster, replicating each block three times along the way.

“…the second main component of Hadoop is its map-reduce framework, which provides a simple way to break analyses over large sets of data into small chunks which can be done in parallel across your 100 machines.”

By buying this technology Twitter is pushing it’s edge with Big Data.

An advantage they started building years ago to make sure the product stopped failing all the time. It has taken them years, millions, and transformed the company into a professionally respectable “Big Data operation”, that is world class and in many ways unique.

Now they have some freedom to play around and Backtype provides the playground. Links stats, emerging trends, and viral memes are just the beginning.

We are about to see how realtime we can get…

Twitter’s edge is Big Data

Today Twitter announced that it purchased Backtype, a tiny company doing Big Data. The website says the company does “social media analytics” which is pretty much saying “Oprah sits on a couch.”

It’s the buzzy-est of buzz words, but if you dig into this you find that the company doing things that everybody wants. Like analyzing hits per link.

When a content creator shares their work the Backtype-created Storm product steps in. Giving you a realtime conversational graph, ability to search comments, and an influence score. Add to that comparison shopping through analysis of top sites and trending links.

But you may be asking, can’t I already do this?

Yes, you can, if you want it 24 hours later.

Stats on the web fall into two categories, instant low tech stats and delayed high tech stats. The market is saturated with the latter (delayed high tech stats) because the core innovations already exist. The reason is complicated but it boils down to the fact that today’s top hardware was built for the pre-Twitter/Facebook world.

To build for realtime processing requires a whole new set of operators where speed, size, and queries go supernova. To give you an example of this, see Backtype’s stats:

  • 100-200 machines (operating as one)
  • 100 million messages
  • 300 queries/second

This is the world that Twitter lives in, millions of messages/second. If you remember the early days of Twitter with all the downtime and Fail Whale messages, that was due to the technological limitations of the time.

They proved that quick, short messages are beloved by us humans to the tune of billions. Since then they have been massively scaling, customizing, and driving the industry. Not only do they need a way to process billions of messages without Fail Whal-ing, but they need to offer (paid) services on top of it.

This is where Backtype comes in. The team built a fascinating service on top of Twitter that does stream processing, continuous computation, and distributed RPC (remote queries of 100s of machines).

The simple translation of this is “live analytics”. The complicated version, pulled from Twitter’s:

“Imagine you have a cluster of 100 computers. Hadoop’s distributed file system makes it so you can put data…in…and pretend that all the hard drives on your machines have coalesced into one gigantic drive….it breaks each file you give it into 64- or 128-MB chunks called blocks and sends them to different machines in the cluster, replicating each block three times along the way.

“…the second main component of Hadoop is its map-reduce framework, which provides a simple way to break analyses over large sets of data into small chunks which can be done in parallel across your 100 machines.”

By buying this technology Twitter is pushing it’s edge with Big Data.

An advantage they started building years ago to make sure the product stopped failing all the time. It has taken them years, millions, and transformed the company into a professionally respectable “Big Data operation”, that is world class and in many ways unique.

Now they have some freedom to play around and Backtype provides the playground. Links stats, emerging trends, and viral memes are just the beginning.

We are about to see how realtime we can get…