Twitter's edge is Big Data

Today Twitter announced that it purchased Backtype, a tiny company doing Big Data. The website says the company does “social media analytics” which is pretty much saying “Oprah sits on a couch.”

It’s the buzzy-est of buzz words, but if you dig into this you find that the company doing things that everybody wants. Like analyzing hits per link.

When a content creator shares their work the Backtype-created Storm product steps in. Giving you a realtime conversational graph, ability to search comments, and an influence score. Add to that comparison shopping through analysis of top sites and trending links.

But you may be asking, can’t I already do this?

Yes, you can, if you want it 24 hours later.

Stats on the web fall into two categories, instant low tech stats and delayed high tech stats. The market is saturated with the latter (delayed high tech stats) because the core innovations already exist. The reason is complicated but it boils down to the fact that today’s top hardware was built for the pre-Twitter/Facebook world.

To build for realtime processing requires a whole new set of operators where speed, size, and queries go supernova. To give you an example of this, see Backtype’s stats:

  • 100-200 machines (operating as one)
  • 100 million messages
  • 300 queries/second

This is the world that Twitter lives in, millions of messages/second. If you remember the early days of Twitter with all the downtime and Fail Whale messages, that was due to the technological limitations of the time.

They proved that quick, short messages are beloved by us humans to the tune of billions. Since then they have been massively scaling, customizing, and driving the industry. Not only do they need a way to process billions of messages without Fail Whal-ing, but they need to offer (paid) services on top of it.

This is where Backtype comes in. The team built a fascinating service on top of Twitter that does stream processing, continuous computation, and distributed RPC (remote queries of 100s of machines).

The simple translation of this is “live analytics”. The complicated version, pulled from Twitter’s:

“Imagine you have a cluster of 100 computers. Hadoop’s distributed file system makes it so you can put data…in…and pretend that all the hard drives on your machines have coalesced into one gigantic drive….it breaks each file you give it into 64- or 128-MB chunks called blocks and sends them to different machines in the cluster, replicating each block three times along the way.

“…the second main component of Hadoop is its map-reduce framework, which provides a simple way to break analyses over large sets of data into small chunks which can be done in parallel across your 100 machines.”

By buying this technology Twitter is pushing it’s edge with Big Data.

An advantage they started building years ago to make sure the product stopped failing all the time. It has taken them years, millions, and transformed the company into a professionally respectable “Big Data operation”, that is world class and in many ways unique.

Now they have some freedom to play around and Backtype provides the playground. Links stats, emerging trends, and viral memes are just the beginning.

We are about to see how realtime we can get…

The surfing pop up

I’m back at the beach. A barefoot drop-in to the doughnut shop for a ‘buttermilk chocolate’. Standing on a fence to spot the best break. Squeezing into a wetsuit like a cannonball through a garden hose. Finally, down to the water..

There’s only one problem, I don’t know how to stand-up.

Everyone thinks I’m a surfer and I sure tell them I am. But, it’s one thing to love the beach and own a surfboard. A whole different type of fish to stand up and ride a wave diagonal instead of straight towards the shore.

I practice when I’m in the water but my inner dork tugs at me. I must hit the internet and learn the mechanics of surfing. Is it a full body gesticulation or a precise pressure point? Are my legs and abdomen juicing the push or is it just my hands?

With the help of Google, I’ve narrowed it down to these precise movements:

  • Place palms on top of the board, do not grip the side.
  • Push your shoulders off the board but keep your thighs on the board, creating an arch in your back (in Yoga it’s upward dog)
  • Swing your knees to your chest.

That’s it.

Now, for some deeper tips.

  • When you stand on land you push your chest back to stand. This shifts your weight backwards and on water will cause you to tip over.  Instead keep your weight steady by pulling your feet underneath you.
  • Never use your knees. All the sites say this is the first and worst habit that every new surfer learns.
  • It is tempting to position your toes for better kung fu grip but resist. The motion needs to be a push up to a pendulum feet swing underneath.
  • You can let your back foot drag into place because the key is to get your front foot underneath you.

Cool, now I have my mechanics down and it’s time to make it happen.

Oh, and the term for this is called the “pop up”.

Sources: how to surf, surfing handbook, I just surf

We finally made it to California

I feel compelled to write this after Amy’s posts on being bicoastal and her dream of living the ideal life. I am part of this too…well, half of it.

I also want to live in both Washington, DC and Southern California. Each one is home to a lifelong love of surfing and politics.

In both places I have learned who I am as a man, what I love,  what I want to do, and how I want to do it. Now it’s just a matter of achieving it.

So here I sit, day two, of making it all happen. Amy and I are in Southern California after packing up our belongings into a Washington, DC storage unit. We have no rent due since we’re staying at my parents.

Which isn’t so bad since are close to the beach and surfing, they have plenty of room and wifi and it let’s us reconnect while looking for a place to live in LA. All in all, it makes the transition super easy.

More importantly, it frees us up to focus on exactly what we need to:

  • Working from home
  • Establishing solid and productive work habits
  • Bringing together the final parts of our social campaign business
  • Enjoying the whole process