Warning: Parameter 1 to ace_where() expected to be a reference, value given in /home/hjlee/hjl/hojohnlee.com/weblog/wp-includes/plugin.php on line 227

Warning: Parameter 1 to ace_join() expected to be a reference, value given in /home/hjlee/hjl/hojohnlee.com/weblog/wp-includes/plugin.php on line 227
Ho John Lee’s Weblog

When you come to a fork in the road, take it!

Crossroads of the World at the Beach Bar, Waikiki

Crossroads of the World at the Beach Bar, Waikiki

As some of you know, I have been exploring a variety of paths forward for SocialQuant, my real time social search and analytics project.

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have far more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as teams make progress on their platforms.  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 looking leaner, Azure becoming non-vaporous, more web APIs getting published, and core online applications starting to turn up. Even Mini-Microsoft is getting positive recently. I look forward to working with Sean Suchter and the Bing team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

I also want to acknowledge my family, friends, and colleagues for all their support, patience, and advice during this process. I met a lot of new people during this process and got to know old friends better.

sq-central

When you come to a fork in the road, take it!

Crossroads of the World at the Beach Bar, Waikiki

Crossroads of the World at the Beach Bar, Waikiki

As some of you know, I have been exploring a variety of paths forward for SocialQuant, my real time social search and analytics project.

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have far more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as teams make progress on their platforms.  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 looking leaner, Azure becoming non-vaporous, more web APIs getting published, and core online applications starting to turn up. Even Mini-Microsoft is getting positive recently. I look forward to working with Sean Suchter and the Bing team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

I also want to acknowledge my family, friends, and colleagues for all their support, patience, and advice during this process. I met a lot of new people during this process and got to know old friends better.

sq-central

When you come to a fork in the road, take it!

As some of you know, I have been exploring a variety of paths forward for SocialQuant, my real time social search and analytics project.

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have far more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as teams make progress on their platforms.  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 looking leaner, Azure becoming non-vaporous, more web APIs getting published, and core online applications starting to turn up. Even Mini-Microsoft is getting positive recently. I look forward to working with Sean Suchter and the Bing team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

I also want to acknowledge my family, friends, and colleagues for all their support, patience, and advice during this process. I met a lot of new people during this process and got to know old friends better.

sq-central

crossroads-IMG_6123

Crossroads of the World at the Beach Bar, Waikiki

When you come to a fork in the road…

As some of you know, I have been exploring a variety of paths forward for SocialQuant, my real time social search and analytics project.

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have far more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as competing organizations make progress on their platforms.  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 getting leaner, and more online applications starting to turn up. Even Mini-Microsoft is getting positive recently. I look forward to working with Sean Suchter and the Bing team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

I also want to acknowledge my family, friends, and colleagues for all their support, patience, and advice during this process. I met a lot of new people during this process and got to know old friends better.

sq-central

When you come to a fork in the road…

As some of you know, I have been exploring paths forward for SocialQuant, my real time social search and analytics project.

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have far more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as competing organizations make progress on their platforms.  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 getting leaner, and more online applications starting to turn up. Even Mini-Microsoft is getting positive recently. I look forward to working with Sean Suchter and the Bing team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

I also want to acknowledge my family, friends, and colleagues for all their support, patience, and advice during this process. I met a lot of new people during this process and got to know old friends better.

sq-central

When you come to a fork in the road…

As some of you know, I have been exploring paths forward for SocialQuant, my real time social search and analytics project.

The rise of Twitter, Facebook, and other social media, combined with web-based applications, smartphones, and cloud computing have all set the stage for new applications and use models based on social discovery, collaboration, and communications, in addition to traditional search. What we’re all calling “real time search” lately isn’t exactly real time, nor is it exactly search, in which you find a definitive/authoritative answer. Much of the opportunity revolves around discovering people, discussions, and events that are relevant to you and bringing it to your attention in a timely, actionable fashion. Information streams from social media are transient, unreliable, and noisy. At the same time, the sheer volume of data can help provide the basis for building better filters. As an added bonus, you can ask questions to people in the social graph itself, and there are numerous examples of communities of interest forming around current events such as Barack Obama’s inauguration, the Iran elections, or even Michael Jackson’s funeral, all of which help surface information content, opinion, and sentiment that were previously inaccessible online.

The challenge then is in filtering and ranking the massive flow of information in a way that helps direct the user’s limited (and non-expanding) time and attention in a way that’s most valuable to them. With today’s information technology, amazing things are possible with limited resources. I personally have far more computing and storage resources than the facility we launched HP’s original photo site with (for millions of dollars), at a fraction of the cost, routinely pushing around datasets of millions of rows on the local development servers. Unfortunately, that’s just the ante to get started on the problem. Running ranking, clustering, and semantic analysis for filtering the ever-growing stream of social media eventually requires web scale computing, even with careful problem selection and data pruning. The bar is also going up every day as the social media user base grows, and as competing organizations make progress on their platforms.  So very shortly, to be competitive in real time, social search and discovery is going to require access to lots of data and either getting a datacenter or working with someone who has one.

In my case, I have recently chosen the latter path, and will be joining the Microsoft Bing search team, focusing on real time and social search. Microsoft itself has been showing signs of a renaissance, with search relaunching, Windows 7 getting leaner, and more online applications starting to turn up. Even Mini-Microsoft is getting positive recently. I look forward to working with Sean Suchter and the Bing team (and likely expanding their carbon footprint) in pursuit of new applications and services as the social media and online application space evolves.

I also want to acknowledge my family, friends, and colleagues for all their support, patience, and advice during this process. I met a lot of new people during this process and got to know old friends better.

sq-central

sq-central

SocialQuant Central

SocialQuant Central

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for someone interested in, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705There are two different issues here: 1) finding a set of interesting / relevant people for new users to follow, and 2) limiting the impact of spam and affiliate marketers, who appear to be scanning the follower lists of the Suggested Users to identify new accounts to spam.

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for someone interested in, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705There are two different issues here: 1) finding a set of interesting / relevant people for new users to follow, and 2) limiting the impact of spam and affiliate marketers, who appear to be scanning the follower lists of the Suggested Users to identify new accounts to spam.

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for someone interested in, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705There are two different issues here: 1) finding a set of interesting / relevant people for new users to follow, and 2) limiting the impact of spam and affiliate marketers, who appear to be scanning the follower lists of the Suggested Users to identify new accounts to spam.

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for someone interested in, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705There are two different issues here: 1) finding a set of interesting / relevant people for new users to follow, and 2) limiting the impact of spam and affiliate marketers, who appear to be scanning the follower lists of the Suggested Users to identify new accounts to spam.

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for  someone interested, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for  someone interested, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705

Follow suggested users, attract instant spamcloud

Despite Twitter’s amazing growth rate, there is general agreement that the Suggested Users List and the new user experience has shortcomings. As an experiment, I created a new Twitter account. I wanted to see what the experience might look like for  someone interested, but otherwise completely unfamiliar with the service. During the signup process, it automatically picks some suggested users (apparently random), which I selected all of, about a dozen or so. Then it asked for my email credentials to check for other people I know on Twitter, which I declined, since I generally don’t give web applications access to my email services. Then I went back to “Suggested Users” under the “Find People” section, and selected all of them. In total, the Suggested Users list got me up to 237 friends in my incoming stream.

Within a few minutes of completing this process, I already had 13 spam followers offering affiliate links for cameras, porn, and twitter followers. A day later I was up to 41 spam followers, plus 4 follow-backs from accounts I followed in addition to the Suggested Users List.

twitter-newuser-spam-090705

twitter-newuser-spam-090705

twitter-newuser-spam-090705

Twitter’s user growth per day

Twitter estimated new users per day through May 2009

Twitter estimated new users per day through May 2009

Here is a companion to the Twitter user population growth chart from last week. This chart shows an estimate of the number of new users per day. The dashed blue bar is the 2009 US inauguration of Barack Obama, and the extreme spike is the Oprah Winfrey show featuring Twitter.

The data used for this chart isn’t as complete for the last week or so at the right hand edge, i.e. the rate of new user signups hasn’t gone to zero, and in fact remains quite high, not 100k users per day, but well above the “pre-mainstream adoption” user signup rates, in the range of 30-50K users/day. As of mid June, Twitter has more than 8M user accounts that have been created.

Twitter’s user growth per day

twitter-userbase-growthrate-may09-annotated

Twitter estimated new users per day through May 2009

Twitter’s amazing user growth

Twitter estimated userbase through May 2009

Twitter estimated userbase through May 2009

The graph above shows an estimate of Twitter’s user population from its launch in March 2006 through May 2009, based on a sample of around 6 million observed user profiles. The dashed blue line is around the 2009 US inauguration of Barack Obama and where the transition from early adopter to early mass audience seems to have taken off.

The entire user population of Twitter appears to have reached 1 million sometime in January but today there are several accounts that have over 1M followers each.

Stated another way, if you signed up before February 2009, you can consider yourself something of an early adopter on Twitter, and among the earliest 15% or so of the entire user population.

The numbers in this survey are inexact but representative, taken from research I’ve been doing for SocialQuant and FailWatch.  There is some survivor bias built in, since I’m pruning spam and suspended accounts. Only Twitter knows the true state of the user base and the social graph, of course.

The initial Twitter users tend to know each other more in real  life, since much of the social network grew from friends of founders, SWSX attendees, and the San Francisco / Silicon Valley tech community. The more recent (post-Obama)  arrivals tend not to have connections to those networks, and often don’t know anyone else to follow. They arrive via mass media and celebrity campaigns, and end up following mass media and celebrities, either from the suggested users list or because those are the only people they know of.

If you look carefully, you can see the rate of increase slows down toward the end of the graph. There was a huge ramp in  new user signups around the time of the Oprah show, which has receded somewhat. This has led to blog posts about Twitter’s impending demise, but looking back, there have been previous surges in the user base (typically around SXSW etc) which led to a peak, then a drop in new user signups to an off-peak but higher-than-before average. So far the current surge is the largest, but seems to be following the pattern. In the absence of any  new driver, user growth should continue at an off-peak but higher level, until the next big jump, or something better comes along.

Page 6 of 78« First...45678...203040...Last »