Ho John Lee's Weblog

These are my links for May 29th from 05:17 to 12:45:

• Some stats from Twitter conference compared to… – Robert Scoble – FriendFeed – Anecdotal data from 140tc this week. 200 tweets/second at peak. Didn't see an estimate of current user account population though, I keep seeing site unique visitor estimates, which aren't useful.
• Microsoft Silverlight vs Google Wave: Why Karma Matters | Zoho Blogs – "The real interesting contrast to us, as independent software developers, is the way developers responded to Silverlight as opposed to the reaction yesterday to Google Wave. Both Silverlight and Wave are aimed at taking the internet experience to the next level. To be perfectly honest, Silverlight is a great piece of technology. Google Wave, as yet, is not much more than a concept and an announcement. It is easy to dismiss all this with "Oh, the press just loves to hype everything Google, and loves to hate Microsoft," but that cannot explain why even competitors like us are willing to embrace Google's innovations, but stay away from perfectly good innovations from Microsoft, such as Silverlight? It comes down to one word: karma."
• makerfaire.com: Maker Faire – This weekend at San Mateo Expo Center
• Google Wave Federation Protocol‎ –
• Google Wave API Overview – Google Wave API – Google Code – APIs for Google Wave email / bbs / wiki / chat / collaboration / communications mashup platform introduced yesterday.
• What Emacs Commands Do You Use Most and Find Most Useful? : programming – Reddit thread discussing favorite emacs commands

These are my links for May 5th through May 6th:

• Coding Horror: I Just Logged In As You: How It Happened – On good password management, why forums should mostly not be storing user passwords in general, and how re-use of passwords on multiple sites can lead to vulnerability on other sites.
• Arc Forum | Arc – Arc is a version of Lisp. Among other things it is used to implement Hacker News.
• John Graham-Cumming: Can you trust Paul Graham with your password? – On best practices for storing password hashes to avoid attacks on compromised password files and the use of rainbow files, in a look at Hacker News implementation of passwords
• Deliberate Ambiguity: How *not* to rate a search engine – Search engines have very simple user interfaces, but are used in many different contexts, most of which don't resemble the way people often try out a new search engine.
• The Slow Erosion of Google Search – Bokardo – On changes in internet user behaviors over time, more social media (ask your Twitter friends) vs directed search (send a keyword query) etc.
• Brynn Marie Evans » Why social search won’t topple Google (anytime soon) – On differences between searching through social media such as Twitter, Facebook etc, vs Google etc.
• The Financial Services Club’s Blog: Stock picking with real-time news – Looking at real time social media trends for trading ideas.
• Lisp’s reputation is so bad that many people don’t even take a look at Lisp | International Lisp Conference 2009 – I haven't touched Lisp in years, except maybe for configuring emacs. A list of possible reasons why Lisp is not more widely used, e.g. "Lisp is old and moldy. It must be primitive by today's standards.", "The exciting languages to learn now are Python, Ruby, Groovy, etc."
• Peering into North Korea – The Big Picture – Boston.com – A collection of recent photos of scenes from North Korea.

These are my links for April 30th from 05:57 to 07:10:

• SIGUSR2 > The Power That is GNU Emacs – "If you've never been convinced before that Emacs is the text editor in which dreams are made from, or that inside Emacs there are unicorns manipulating your text, don't expect me to convince you."
• How To Be A Successful Evil Overlord – 100 remedies for the fatal flaws exhibited by famous evil overlords of the past. Also some business executives, I think.
• Google Could Have Caught Swine Flu Early | Wired Science – Google’s search data may have been able to provide an early warning of the swine flu outbreak — if the company had been looking in the right place. Last week, at the request of the Centers for Disease Control, Google took a retroactive look at its search data from Mexico. And there the team found a pre-media bump in telltale flu-related search terms (you know, “influenza + phlegm + coughing”) that was inconsistent with standard, seasonal flu trends.
• What Twitter Looks Like For Twitter Employees (SCREENSHOTS) – Some screen shots of current admin tools at Twitter for managing user accounts, blocks, whitelisting, suspensions, and user stats such as # follow attempts, # updates, #directs, etc
• Twitter Aggregator Sawhorse Media Raises Seed Round, Launches Pets, Celeb Sites | paidContent.org – "Channelized" feeds from curated lists of twitter sources.

These are my links for April 20th through April 23rd:

• What I’ve Learned from Hacker News – Paul Graham on social dynamics and managing Hacker News, user submitted comments and ranking (voting up/down) , editorial intervention and moderators, project goals.
• SEOmoz | Reddit, Stumbleupon, Del.icio.us and Hacker News Algorithms Exposed! – Looking at variations on algorithms for ranking items on social news aggregators
• NGINX + PHP-FPM + APC = Awesome – Walkthrough on setting up cached PHP web server on nginx with apc.
• Particletree » PHP Quick Profiler – Lightweight tool for profiling PHP code.
• MySQL’s Full-Text Formulas – Database Journal –
• http://www.acapela-group.com/text-to-speech-interactive-demo.html – Online text-to-speech demo, with various male and female speakers, plus a few translations.
• Dealing with Duplicate Person Data – Proud to Use Perl – Classifying likely duplicate entries in name/address contact data using Levenshtein distance and tables of nickname synonym and assigned distance weights.
• Web Security Horror Stories: The Director’s Cut at <head> – Presentation slides from a talk by Simon Willison on cross site scripting, SQL injection, referer forgery, and clickjacking attacks on web applications.

These are my links for February 16th through February 17th:

• Top 100 Network Security Tools – Many many security testing and hacking tools.
• FRONTLINE: inside the meltdown: watch the full program – "On Thursday, Sept. 18, 2008, the astonished leadership of the U.S. Congress was told in a private session by the chairman of the Federal Reserve that the American economy was in grave danger of a complete meltdown within a matter of days. "There was literally a pause in that room where the oxygen left," says Sen. Christopher Dodd"
• The Dark Matter of a Startup – "Every successful startup that I have seen has someone within their ranks that just kinda “does stuff.” No one really knows specifically what they do, but its vital to the success of the startup."
• Why I Hate Frameworks – "A hammer?" he asks. "Nobody really buys hammers anymore. They're kind of old fashioned…we started selling schematic diagrams for hammer factories, enabling our clients to build their own hammer factories, custom engineered to manufacture only the kinds of hammers that they would actually need."
• Mining The Thought Stream – Lots of comments around what is Twitter good for and how will it make money, revolving around real/near-time search, analytics, marketing, etc.
• Understanding Web Operations Culture – the Graph & Data Obsession … – Comparison of traffic at Flickr, Google, Twitter, last.fm during the Obama inauguration. "One of the most interesting parts of running a large website is watching the effects of unrelated events affecting user traffic in aggregate."

I recently went for my two-year followup to see how my eyes are doing after wavefront LASIK. At the initial exam and each followup visit, they measure the point spread function of your eye. Here’s a before-and-after.

The scale of the two graphs are different, so the improvement is even better than it appears at first glance. The upper plot corresponds to roughly 20/80 vision. The lower plot, two years later, is at 20/15.

I tripped over a reference to an interesting paper on PageRank hacking while looking at some unrelated rumors at Ian McAllister’s blog. The undated paper is titled “Faults of PageRank / Something is Wrong with Google’s Mathematical Model”, by Hillel Tal-Ezer, a professor at the Academic College of Tel-Aviv Yaffo.

It points out a fault in Google’s PageRank algorithm that causes ’sink’ pages that are not strongly connected to the main web graph to have an unrealistic importance. The author then goes on to explain a new algorithm with the same complexity of the original PageRank algorithm that solves this problem.

After a quick read through this, it appears to describe one of the techniques that had been popular among some search engine optimizers a while back, in which link farms would be constructed pointing at a single page with no outbound links, in an effort to artificially raise the target page’s search ranking.

This technique is less effective now than in the past, because Google has continued to update its indexing and ranking algorithms in response to the success of link spam and other ranking manipulation. Analysis of link patterns (SpamRank, link mass) and site reputation (Hilltop) can substantially reduce the effect described here. Nonetheless, it’s nice to see a quantitative description of the problem.

See also: A reading list on PageRank and Search Algorithms

via Seomoz:

This week’s Newsweek (December 12, 2005) features an article on white hat vs black hat search engine optimization. Among other things, it’s interesting that the topic has made it into the mainstream media.

A “black hat” anecdote:

Using an illicit software program he downloaded from the Net, he forcibly injected a link to his own private-detectives referral site onto the site of Long Island’s Stony Brook University. Most search engines give a higher value to a link on a reputable university site.

The site in question appears to be “www.private-detectives.org”, still currently #1 at MSN and #4 at Yahoo for searches on “private detectives”. It appears to have been sandboxed on Google.

Another interesting post at Seomoz features comments from “randfish” and “EarlGrey”, the two SEO consultants interviewed by Newsweek on the merits of “White Hat” vs “Black Hat” search engine optimization, and gives further perspective on the motivation and outlook of the two approaches.

In some ways one can think of the difference between search engine optimization approaches as a “trading” approach vs a “building” approach to investment. The “Black Hat” approach articulated in the Seomoz article tends to focus purely on a tactical present cash return to the operator, while the “White Hat” approach presumes that the operator will realize ongoing future value by developing a useful information asset and making it visible to the search engines. This makes an implicit assumption that the site itself offers some unique and valuable information content, which can’t usually be the case in the long run.

From an information retrieval point of view, I’m obviously in the latter camp of thinking that identifying the most relevant results for the search user is a good thing. However, the black hat approach makes perfect sense if you consider it in terms of optimizing the short term value return to the publisher (cash as information), while possibly still presenting a useable information return to the search user. This is especially the case for commodity information or products, in which the actual information or goods are identical, such as affiliate sales.

I’m a little curious about the link from Stony Brook University. I took a quick look but wasn’t able to turn up a backlink. One of the problems with simply relying on trusted link sources is that they can be gamed, corrupted, or hacked.

See also: A reading list on PageRank and search algorithms

Update 12-12-2005 00:30 PST: Lots of comments on Matt Cutt’s post, plus Slashdot

From SANS: Multiple Linksys WRT54G Vunerabilities, published: 2005-09-14

iDefense has released five vulnerabilities against the Linksys WRT54G wireless access point/switch/router. Some of these vulnerabilities are very serious. Users of these products are highly recommended to patch their devices. Patches for the latest versions are available at http://www.linksys.com.

This is one of the most popular and widely modified wireless routers out there. If you have one that’s exposed to the public, time to patch it.

Here’s the capsule descriptions, these look like fun:

• Remote exploitation of a design error in the upgrade.cgi component of
  Cisco Systems Inc.’s Linksys WRT54G wireless router may allow
  unauthenticated modification of the router firmware.
• Remote exploitation of a design error in multiple versions of the
  firmware for Cisco Systems Inc.’s Linksys WRT54G wireless router may
  allow unauthenticated modification of the router configuration.
• Remote exploitation of an input validation error within the web
  management httpd component of Cisco Systems Inc.’s Linksys WRT54G
  wireless router may allow unauthenticated users to cause a denial of
  service (DoS).
• Remote exploitation of a buffer overflow vulnerability in multiple
  versions of the firmware for Cisco Systems Inc.’s Linksys WRT54G
  wireless router may allow unauthenticated execution of arbitrary
  commands as the root user.
• Remote exploitation of a design error in the ‘restore.cgi’ component of
  Cisco Systems Inc.’s Linksys WRT54G wireless router may allow
  unauthenticated modification of the router configuration.

Alex Muse takes apart his Apple iPod Nano and lays out the pieces so the curious among us won’t have to.

Inside Apple’s New Nano (wonder what’s inside)

For some reason, I get a 1024×768 photo viewing this in my RSS reader, but only 425×321 viewing his web site directly. Obviously, the larger photo has better details.

Wonder if he can get it back together (and working)? Those connectors look pretty fussy.

Update 09-22-2005 13:41 PDT: The Inquirer has some details on the components from a report by iSuppli:

The firm offers a “teardown analysis” which it said showed the device uses a Portalplayer 5021C system on a chip and a Cypress CY8C21434 for the circuitry behind the “click wheel” interface.
…
It said that these, along with other ICs (integrated circuits), account for 77 per cent of the $90.18 total bill of materials (BOM) cost of the Nano.

The NAND flash memory in the device is made by Samsung, and iSuppli estimates it got a big discount from the Korean giant, making the twin NAND flash parts cost only $54.

Isuppli said the iPod Nano costs $400 when you buy it.

Update 09-23-2005 10:41 PDT: Longer analysis of the iPod Nano bill of materials and margins at Business Week.

Here’s some sample data, and an example of how to read the heart rate data from the binary dump file eeprom.bin, continuing from part 1 of my notes on the Timex Bodylink.

This is what we don’t want to see when using the Timex Trainer software to download from the data recorder.

In my case, this happened in the middle of a long (4+ hour HRM and GPS) data download. I suspect, but can’t confirm, that this was related to leaving the “speed smoothing” function on and tripping over an unhandled exception related to a missing point or something along those lines. I didn’t seem to have problems after disabling the smoothing option.

When retrying the download, the Timex software would complain:

Timex Trainer shut down unexpectedly during the last data transfer. This may indicate a problem with the way the data was stored in the memory of the Data Recorder. Check all the connections to the Data Recorder and make sure Recorder’s battery is notl exhausted, then try the transfer again. If the transfer is still unsuccessful, you may have to erase the data in the Data Recorder (refer to Help for the procedure). Unfortunately, this means that any data currently stored in the Data Recorder will behat the ue to a loose cable or low battery. If you continue to have problems after the Data Recorder memory has been erased, contact Timex Customer Service.

In my case, I had a fresh set of batteries in the GPS pod, the HRM strap, and in the data recorder, so that wasn’t the problem. The Timex software provides a raw data dump utility, which I used to save a copy of the bits with. This takes the binary data directly from the data recorder memory without attempting any processing, and always seems to work uneventfully, even when the software otherwise complains.

I spent an evening searching online for a fix with no luck, and also spoke with Timex service on the phone, also with no luck, so I’m left with the data and some curiosity.

Here’s are some bits for anyone who wants to play along at home:

Data dump from the 2004 Big Sur Marathon: eeprom-timex-040425.bin (46KB) and eeprom-timex-040425.txt 143KB)
This is around 5 hours of HRM plus GPS data, with the time stamp probably incorrect because I don’t recall resetting it after changing the data recorder battery.

Here’s another data dump of a 75 minute HRM + GPS run: eeprom-timex-040516.bin (12KB) and eeprom-timex-040516.txt (37KB)

This run was around 75 minutes, 7.5 miles, and came after giving up on a useful response from Timex Customer Support, and resetting the data recorder using the recessed silver button on the back of the unit. This time I remembered to set the time after resetting the unit.

Here’s a data dump of a HRM-only run: eeprom-timex-040517.bin (3KB), eeprom-timex-040517.txt (9KB)

This run was a 70 minute treadmill hill interval session. There’s only one session in the recorder, as the previous session was cleared.

Now let’s have a look at the data. Here’s the top of the HRM-only session:

80 01 00 94 0B 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 11 03 0C 10 04 04 63 65 65 65 68 6A 6A 68 69
69 68 6B 6C 6B 6A 68 67 66 6B 68 69 69 6B 6C 6E
6F 6E 6E 6E 6E 6F 70 70 6F 70 6F 6F 6F 6C 6D 6D
6C 6D 6D 6C 6B 68 66 66 67 68 68 68 68 67 68 67
66 67 68 68 68 68 6A 6B 68 68 68 68 67 68 68 68
69 68 67 67 65 66 66 67 67 64 64 64 65 64 64 64
66 6B 6B 6B 68 68 68 67 67 64 64 64 65 65 65 66
66 63 63 63 67 68 65 67 67 65 65 65 64 64 64 65
64 62 62 62 63 62 62 63 63 62 63 62 63 63 63 66
65 64 63 63 63 63 63 62 64 64 6B 6C 6D 6E 70 72
72 72 72 72 72 72 72 73 74 75 75 76 76 76 76 77
77 77 79 79 7A 79 77 77 77 78 79 79 79 79 79 77
77 79 79 79 7A 7A 7A 7C 7C 7C 7C 7C 7D 7D 7D 7E
7D 7E 7E 7E 7F 7F 7E 7E 7E 80 7F 7F 7F 7F 80 7E
7E 7D 7E 7D 7D 7D 7E 7F 7F 7F 7F 80 7F 7F 7F 7E
7F 7D 7D 7B 7C 7D 7E 7E 80 81 81 83 83 83 83 82

Here’s how to read the heart rate data dump:

The first 240 bytes contains directory header info for all sessions in the recorder memory.

The first 2 bytes contain the magic number

80 01

Each subsequent 3 bytes contain the offset in bytes to the end of the Nth data session.
In this case we have

00 94 0b

After byteswapping to

00 0b 94

this works out to 2964, which is the length of the binary dump. If there were a second data session logged, it would start at offset 2964 from the top of memory.

Now let’s look at the actual session data. The first 240 bytes appear to be reserved for storing the session offsets. The beginning of interesting data looks like this:

00 11 03 0C 10 04 04 63 65 65 65 68 6A 6A 68 69 

The first few heart rate entries that turn up in Timex Trainer for this session are

99,101,101,101,104,106,106,104,105,105,104

In hex, this would be

63,65,65,65,68,6A,6A,68,69,69,68

So it looks like there’s a 7 byte session header. This is probably just a serialized internal data structure from the Timex Trainer software, which appears to be built with Visual Basic for the GUI, and a DLL to handle device functions, probably written in C.

From looking at various headers, here’s how to read the session header:

00 11 03 0C 10 04 04

00 = heart rate data only
11 hex = 17 decimal = seconds
03 hex = 03 decimal = minutes
0C hex = 12 decimal = hours
10 hex = 16 decimal = day
04 hex = 04 decimal = month (0 = January)
04 hex = 04 decimal = year (0 = 2000)

So this session starts at 12:03:17 on May 16, 2004.

After the heart rate data starts, there’s not much else going on here, since there’s no GPS records mixed in. Occasionally, we see something like this in the middle of the data:

59 59 59 59 60 60 60 61 00 00 00 00 00 00 54 00
00 00 00 00 00 04 66 65 66 69 68 68 67 68 00 6B 

I think the recorder just fills in zeros when it doesn’t have a valid input.

At the end of the heart rate data session, we again see a 7 byte trailing record.

00 31 1C 0D 10 04 04

Not so sure about this part, but the

04 04

seems to turn up at the end of other data sessions.

At this point, you should now be able to reliably extract heart rate data from the raw data dump provided by the Timex Bodylink data recorder.

The next post in this series will look at the GPS-based speed and distance data.

See also: Hacking the Timex Bodylink – Part 1

I have used various heart rate monitors and GPS-based distance measuring systems as part of my running in the past. A couple of years ago, I wore a Timex Bodylink HRM and GPS system, with the data recorder, during the Big Sur Marathon. Since I’m a bit of a data junkie, I wanted to compare the race data with my previously recorded training data.

Unfortunately, the Timex Trainer software choked while downloading the 4+ hours of data. It did let me download the raw data from the recorder, though. I was hoping that someone at Timex might be able to either parse the data or provide a specification so I could process the bits myself. Other than signing up as an OEM developer, there wasn’t much in the way of software support, and no useful response from Timex with regard to either recovering the data from the binary dump (eeprom.txt) or getting a specification.

I never got around to fully decoding the raw data dump format, but thought I would share my notes for anyone who is interested in picking this up. At present, I can read the session directory, and the heart rate data. The GPS data encoding is more complicated, and may use the native encoding from the GPS chipset used by the (Garmin-manufactured) Timex GPS pod (might be SIRF).

From viewing the Timex Trainer database (Microsoft Access format):

• we can see that all HR data times are integer multiples of 2 seconds
• no HR data times are recorded on odd seconds
• HR records store prkey, sessionkey, cum_duration(int seconds), and HR
• HR only session is session_type 0, GPS only is session_type 1, combined is session_type 20
• all GPS data times are exact multiples of 3.57 seconds
• GPS records store prkey, sessionkey, cum_duration(float seconds), speed, and cum_distance
• minimum recorded distance is .001 miles
• before that cum_distance is 0>
• minimum recorded speed is 0.075
• we see cases where speed is non zero and cum_distance is 0
• we see cases where speed is zero and cum_distance is non-0

Dump header format (eeprom.bin / eeprom.txt)

• 2 bytes = 80 01 magic number?
• 3 byte offset to end of session N
• repeated for each session.
• 1st session starts at offset +240 from top of file.

Session header and trailer:

• 1 byte = session type (0 = HRM, 0×22 = GPS, 0xFF = GPS+HRM)
• 1 byte = seconds
• 1 byte = minutes
• 1 byte = hours (24 hour format)
• 1 byte = day-1 (1st = 0)
• 1 byte = month-1 (Jan = 0)
• 1 byte = year-2000 (2004 = 4)

See also: Hacking the Timex Bodylink – Part 2

Google Hacking for Penetration Testers (Review at Slashdot)

author Johnny Long pages 448 publisher Syngress reviewer Corey Nachreiner ISBN 1931836361
summary Google’s dark and dork sides exposed; despite the title, useful for everyone who’d like to get the most out of google

Most Web surfers don’t realize the sheer amount of extremely sensitive information available for the harvesting on the Internet. In that sense, Google Hacking is eye-popping. Do you want to find misconfigured Web servers that publicly list their directory contents? A quick Google search does the trick. Or, suppose you found some new exploit code that only works against a particular version of IIS 5.0. Submit a quick Google query for a helpful list of possible targets. Do you want to harvest user logins, passwords (for example, mySQL passwords in a connect.inc file), credit card numbers, social security numbers or any other potentially damaging tidbit that Web users and administrators accidentally leak onto the Internet? Google Hacking shows you how, with highly refined searches gleaned from the community contributing to the Google Hacking database (GHDB) found on Long’s Web site.

Haven’t read this particular one, but it sounds fun. There are a bunch of these books and articles on interesting uses for Google and other web services coming out lately. I’m still trying to stay mostly in book-reduction mode, having donated 90+ boxes of books to the library last year to clear out some space.

Advertised in EETimes, December 20/27, 2004
www.rabbitethernetize.com

Includes development board with RCM3720, 512K flash, 256K SRAM, 1MB Serial Flash, 33 digital I/O, full version of development software, TCP/IP stack, sample programs, AC adapter

Promotional price $99, normally $199.

Could be handy for building ethernet devices of various sorts.