Ho John Lee's Weblog

Crossroads of the World at the Beach Bar, Waikiki

As some of you know, I have been exploring a variety of paths forward for SocialQuant, my real time social search and analytics project. My family, friends, and colleagues have given me much support, patience, and advice during this process, which has reached a crossroads, and as Yogi Berra says, “When you come to a fork in the road, take it!”

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online. One interesting aspect of real time social media is that it’s not just algorithmic, it’s based on human connections and emotions. So a message  that “feels right” from people you trust can be more relevant than one that is “correct” at times.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as well funded teams make progress on their platforms (+Google).  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 looking leaner, Azure becoming non-vaporous, more web APIs getting published, core online applications starting to turn up, and a cool Office 2010 video. Even Mini-Microsoft is getting positive recently. And Google is starting to have “bigness” issues.

I look forward to working with Sean Suchter and the Microsoft Bing search team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

You can follow along on Twitter (@hjl). As always, any and all opinions here are solely mine and do not reflect the position of any past, present, or future employer, partner, or business associate.

Here are my current informal policies for using Facebook, LinkedIn, Twitter, and Friendfeed.  Short version – Facebook and LinkedIn I use for people I know personally, Twitter and Friendfeed any interesting input is welcome.

Facebook: This has been rapidly going mainstream lately. I had a mostly unused account for a long time, which has become more interesting/active as people I know sign up.  I presently only link to people I know in real life. Facebook is interesting because there are people I haven’t interacted with for years (high school friends etc) as well as people that live next door (literally) and colleagues from past work projects all mixed together, and they all get to eavesdrop and engage in casual/passive interaction. I currently have my Twitter feed linked to update my Facebook status, which means my messages are probably cryptic to about half the readers at any given time.

LinkedIn: I originally only linked to people I worked with and knew very well. I have broadened out the criteria over the years, and at this point I will link to people that I haven’t worked with but have at least actually met and had a conversation with. I basically don’t link to people I don’t know and haven’t met, though. I’d to at least be able to recognize people I’m linked to, and have a clue about what they’re like. So no “LinkedIn Open Networking” for me.

Twitter: I look for interesting (to me) streams, whether or not I know the author. Most of my twitter feed is people I haven’t met in person.  I follow people I know in real life, and also discover people who have commented on something that turned up in a conversation or a search. I don’t auto follow, although I do try to take a look at who’s on my follower list periodically to see if there is someone I should add.  Twitter has also been the most interesting for making new connections with people in real life, as you can get a sense of topic people are thinking about and what they’re more generally like.  I use Twitter for scanning a range of topics, so I’m a little less interested in people with huge follower counts and more interested in people kicking out uncorrelated but interesting ideas and data.  I’m working on tools for scanning and filtering status and sentiment streams, so in theory a bigger source network is better, if you can make effective use of it.

Friendfeed: Sometimes I feel like Friendfeed is the Robert Scoble/Louis Gray channel, but I have seeded it with my Twitter feed and have gradually added people as they are exposed through the “friend of” feature.  I always have the feeling that I’m not making the best use of Friendfeed. I like the conversations that pop up on posted items, but wish for the range of input that comes from the huge user bases on Twitter and Facebook. Then again, maybe not Facebook inputs here, I also enjoy the relative skew towards content from early adopters that persists for now on Friendfeed.

If I know you in real life, feel free to send me a Facebook or LinkedIn request, there have been a lot of people signing up lately and I’ve been enjoying reconnecting with people I haven’t heard from in a while.  If I don’t know you (yet), you’re welcome to follow on Twitter (@hjl) or Friendfeed (hjl).

The automatic nightly link posts from del.icio.us stopped working properly sometime last year. The links would get posted, but had extra “\n” inserted at every line break. Here’s an example. An unexpected side effect of having “ugly” link posts is that I mostly stopped posting links to del.icio.us for a while.

As part of the recent blog platform update, I’ve switched from the del.icio.us  “experimental” nightly blog posting to Postalicious, which seems to be working nicely, you can see the new link post style (and the old ones too, unless I get around to cleaning them up) here.

This evening I’m rolling out a long overdue update to the blogging platform. It’s been a little complicated, because I ‘ve been running a heavily customized WordPress 1.5.2 for a long time, and there have been a lot of changes since then to WordPress, various plugins, and the underlying database (the current release is 2.7.1).

The new version is based on Atahualpa, which has many customizable options. The Recent Posts, Tag Cloud, Recent Links, Twitter status, and permalinks are all working as before.  The new template doesn’t have a place for the randomly selected banner thumbnail images from my Flickr account, but does incorporate a larger random image at the top, which currently selects from a few photos I picked out of my snapshot collection. I may figure out some other way of sharing some photos here. I’ve also added a random quote widget. You have to provide your own collection of quotes, so there aren’t many in there yet.

It might be a little slower than the old platform for a while until I get the caching set up, all those customizable options use a lot of database queries.

Let me know what you think, and if you are have any suggestions or are having problems viewing things. I’ve mostly been looking at this with Firefox 3, so people with other browsers may have a different experience.

I haven’t been posting here in a while, but think I will try picking up the keyboard here a little more frequently. I added a twitter box on the sidebar a while back, as I have been experimenting with that more, along with friendfeed, facebook, etc. I like the brevity and immediacy of twitter, but not everything fits in 140 characters. You can find me on twitter and friendfeed as “hjl”, also on Facebook.