Bookmarks for December 31st through January 17th

These are my links for December 31st through January 17th:

  • Khan Academy – The Khan Academy is a not-for-profit organization with the mission of providing a high quality education to anyone, anywhere.

    We have 1000+ videos on YouTube covering everything from basic arithmetic and algebra to differential equations, physics, chemistry, biology and finance which have been recorded by Salman Khan.

  • StarCraft AI Competition | Expressive Intelligence Studio – AI bot warfare competition using a hacked API to run StarCraft, will be held at AIIDE2010 in October 2010.
    The competition will use StarCraft Brood War 1.16.1. Bots for StarCraft can be developed using the Broodwar API, which provides hooks into StarCraft and enables the development of custom AI for StarCraft. A C++ interface enables developers to query the current state of the game and issue orders to units. An introduction to the Broodwar API is available here. Instructions for building a bot that communicates with a remote process are available here. There is also a Forum. We encourage submission of bots that make use of advanced AI techniques. Some ideas are:
    * Planning
    * Data Mining
    * Machine Learning
    * Case-Based Reasoning
  • Measuring Measures: Learning About Statistical Learning – A "quick start guide" for statistical and machine learning systems, good collection of references.
  • Berkowitz et al : The use of formal methods to map, analyze and interpret hawala and terrorist-related alternative remittance systems (2006) – Berkowitz, Steven D., Woodward, Lloyd H., & Woodward, Caitlin. (2006). Use of formal methods to map, analyze and interpret hawala and terrorist-related alternative remittance systems. Originally intended for publication in updating the 1988 volume, eds., Wellman and Berkowitz, Social Structures: A Network Approach (Cambridge University Press). Steve died in November, 2003. See Barry Wellman’s “Steve Berkowitz: A Network Pioneer has passed away,” in Connections 25(2), 2003. It has not been possible to add the updating of references or of the quality of graphics that might have been possible if Berkowitz were alive. An early version of the article appeared in the Proceedings of the Session on Combating Terrorist Networks: Current Research in Social Network Analysis for the New War Fighting Environment. 8th International Command and Control Research and Technology Symposium. National Defense University, Washington, D.C June 17-19, 2003
  • SSH Tunneling through web filters | s-anand.net – Step by step tutorial on using Putty and an EC2 instance to set up a private web proxy on demand.
  • PyDroid GUI automation toolkit – GitHub – What is Pydroid?

    Pydroid is a simple toolkit for automating and scripting repetitive tasks, especially those involving a GUI, with Python. It includes functions for controlling the mouse and keyboard, finding colors and bitmaps on-screen, as well as displaying cross-platform alerts.
    Why use Pydroid?

    * Testing a GUI application for bugs and edge cases
    o You might think your app is stable, but what happens if you press that button 5000 times?
    * Automating games
    o Writing a script to beat that crappy flash game can be so much more gratifying than spending hours playing it yourself.
    * Freaking out friends and family
    o Well maybe this isn't really a practical use, but…

  • Time Series Data Library – More data sets – "This is a collection of about 800 time series drawn from many different fields.Agriculture Chemistry Crime Demography Ecology Finance Health Hydrology Industry Labour Market Macro-Economics Meteorology Micro-Economics Miscellaneous Physics Production Sales Simulated series Sport Transport & Tourism Tree-rings Utilities"
  • How informative is Twitter? » SemanticHacker Blog – "We undertook a small study to characterize the different types of messages that can be found on Twitter. We downloaded a sample of tweets over a two-week period using the Twitter streaming API. This resulted in a corpus of 8.9 million messages (”tweets”) posted by 2.6 million unique users. About 2.7 million of these tweets, or 31%, were replies to a tweet posted by another user, while half a million (6%) were retweets. Almost 2 million (22%) of the messages contained a URL."
  • Gremlin – a Turing-complete, graph-based programming language – GitHub – Gremlin is a Turing-complete, graph-based programming language developed in Java 1.6+ for key/value-pair multi-relational graphs known as property graphs. Gremlin makes extensive use of the XPath 1.0 language to support complex graph traversals. This language has applications in the areas of graph query, analysis, and manipulation. Connectors exist for the following data management systems:

    * TinkerGraph in-memory graph
    * Neo4j graph database
    * Sesame 2.0 compliant RDF stores
    * MongoDB document database

    The documentation for Gremlin can be found at this location. Finally, please visit TinkerPop for other software products.

  • The C Programming Language: 4.10 – by Kernighan & Ritchie & Lovecraft – void Rlyeh
    (int mene[], int wgah, int nagl) {
    int Ia, fhtagn;
    if (wgah>=nagl) return;
    swap (mene,wgah,(wgah+nagl)/2);
    fhtagn = wgah;
    for (Ia=wgah+1; Ia<=nagl; Ia++)
    if (mene[Ia]<mene[wgah])
    swap (mene,++fhtagn,Ia);
    swap (mene,wgah,fhtagn);
    Rlyeh (mene,wgah,fhtagn-1);
    Rlyeh (mene,fhtagn+1,nagl);

    } // PH'NGLUI MGLW'NAFH CTHULHU!

  • How to convert email addresses into name, age, ethnicity, sexual orientation – This is so Meta – "Save your email list as a CSV file (just comma separate those email addresses). Upload this file to your facebook account as if you wanted to add them as friends. Voila, facebook will give you all the profiles of all those users (in my test, about 80% of my email lists have facebook profiles). Now, click through each profile, and because of the new default facebook settings, which makes all information public, about 95% of the user info is available for you to harvest."
  • Microsoft Security Development Lifecycle (SDL): Tools Repository – A collection of previously internal-only security tools from Microsoft, including anti-xss, fuzz test, fxcop, threat modeling, binscope, now available for free download.
  • Analytics X Prize – Home – Forecast the murder rate in Philadelphia – The Analytics X Prize is an ongoing contest to apply analytics, modeling, and statistics to solve the social problems that affect our cities. It combines the fields of statistics, mathematics, and social science to understand the root causes of dysfunction in our neighborhoods. Understanding these relationships and discovering the most highly correlated variables allows us to deploy our limited resources more effectively and target the variables that will have the greatest positive impact on improvement.
  • PeteSearch: How to find user information from an email address – FindByEmail code released as open-source. You pass it an email address, and it queries 11 different public APIs to discover what information those services have on the user with that email address.
  • Measuring Measures: Beyond PageRank: Learning with Content and Networks – Conclusion: learning based on content and network data is the current state of the art There is a great paper and talk about personalization in Google News they use content for this purpose, and then user click streams to provide personalization, i.e. recommend specific articles within each topical cluster. The issue is content filtering is typically (as we say in research) "way harder." Suppose you have a social graph, a bunch of documents, and you know that some users in the social graph like some documents, and you want to recommend other documents that you think they will like. Using approaches based on Networks, you might consider clustering users based on co-visitaion (they have co-liked some of the documents). This scales great, and it internationalizes great. If you start extracting features from the documents themselves, then what you build for English may not work as well for the Chinese market. In addition, there is far more data in the text than there is in the social graph
  • mikemaccana’s python-docx at master – GitHub – MIT-licensed Python library to read/write Microsoft Word docx format files. "The docx module reads and writes Microsoft Office Word 2007 docx files. These are referred to as 'WordML', 'Office Open XML' and 'Open XML' by Microsoft. They can be opened in Microsoft Office 2007, Microsoft Mac Office 2008, OpenOffice.org 2.2, and Apple iWork 08. The module was created when I was looking for a Python support for MS Word .doc files, but could only find various hacks involving COM automation, calling .net or Java, or automating OpenOffice or MS Office."

Bookmarks for June 13th through January 16th

These are my links for June 13th through January 16th:

  • StarCraft AI Competition | Expressive Intelligence Studio – AI bot warfare competition using a hacked API to run StarCraft, will be held at AIIDE2010 in October 2010.
    The competition will use StarCraft Brood War 1.16.1. Bots for StarCraft can be developed using the Broodwar API, which provides hooks into StarCraft and enables the development of custom AI for StarCraft. A C++ interface enables developers to query the current state of the game and issue orders to units. An introduction to the Broodwar API is available here. Instructions for building a bot that communicates with a remote process are available here. There is also a Forum. We encourage submission of bots that make use of advanced AI techniques. Some ideas are:
    * Planning
    * Data Mining
    * Machine Learning
    * Case-Based Reasoning
  • Measuring Measures: Learning About Statistical Learning – A "quick start guide" for statistical and machine learning systems, good collection of references.
  • Berkowitz et al : The use of formal methods to map, analyze and interpret hawala and terrorist-related alternative remittance systems (2006) – Berkowitz, Steven D., Woodward, Lloyd H., & Woodward, Caitlin. (2006). Use of formal methods to map, analyze and interpret hawala and terrorist-related alternative remittance systems. Originally intended for publication in updating the 1988 volume, eds., Wellman and Berkowitz, Social Structures: A Network Approach (Cambridge University Press). Steve died in November, 2003. See Barry Wellman’s “Steve Berkowitz: A Network Pioneer has passed away,” in Connections 25(2), 2003. It has not been possible to add the updating of references or of the quality of graphics that might have been possible if Berkowitz were alive. An early version of the article appeared in the Proceedings of the Session on Combating Terrorist Networks: Current Research in Social Network Analysis for the New War Fighting Environment. 8th International Command and Control Research and Technology Symposium. National Defense University, Washington, D.C June 17-19, 2003
  • SSH Tunneling through web filters | s-anand.net – Step by step tutorial on using Putty and an EC2 instance to set up a private web proxy on demand.
  • PyDroid GUI automation toolkit – GitHub – What is Pydroid?

    Pydroid is a simple toolkit for automating and scripting repetitive tasks, especially those involving a GUI, with Python. It includes functions for controlling the mouse and keyboard, finding colors and bitmaps on-screen, as well as displaying cross-platform alerts.
    Why use Pydroid?

    * Testing a GUI application for bugs and edge cases
    o You might think your app is stable, but what happens if you press that button 5000 times?
    * Automating games
    o Writing a script to beat that crappy flash game can be so much more gratifying than spending hours playing it yourself.
    * Freaking out friends and family
    o Well maybe this isn't really a practical use, but…

  • Time Series Data Library – More data sets – "This is a collection of about 800 time series drawn from many different fields.Agriculture Chemistry Crime Demography Ecology Finance Health Hydrology Industry Labour Market Macro-Economics Meteorology Micro-Economics Miscellaneous Physics Production Sales Simulated series Sport Transport & Tourism Tree-rings Utilities"
  • How informative is Twitter? » SemanticHacker Blog – "We undertook a small study to characterize the different types of messages that can be found on Twitter. We downloaded a sample of tweets over a two-week period using the Twitter streaming API. This resulted in a corpus of 8.9 million messages (”tweets”) posted by 2.6 million unique users. About 2.7 million of these tweets, or 31%, were replies to a tweet posted by another user, while half a million (6%) were retweets. Almost 2 million (22%) of the messages contained a URL."
  • Gremlin – a Turing-complete, graph-based programming language – GitHub – Gremlin is a Turing-complete, graph-based programming language developed in Java 1.6+ for key/value-pair multi-relational graphs known as property graphs. Gremlin makes extensive use of the XPath 1.0 language to support complex graph traversals. This language has applications in the areas of graph query, analysis, and manipulation. Connectors exist for the following data management systems:

    * TinkerGraph in-memory graph
    * Neo4j graph database
    * Sesame 2.0 compliant RDF stores
    * MongoDB document database

    The documentation for Gremlin can be found at this location. Finally, please visit TinkerPop for other software products.

  • The C Programming Language: 4.10 – by Kernighan & Ritchie & Lovecraft – void Rlyeh
    (int mene[], int wgah, int nagl) {
    int Ia, fhtagn;
    if (wgah>=nagl) return;
    swap (mene,wgah,(wgah+nagl)/2);
    fhtagn = wgah;
    for (Ia=wgah+1; Ia<=nagl; Ia++)
    if (mene[Ia]<mene[wgah])
    swap (mene,++fhtagn,Ia);
    swap (mene,wgah,fhtagn);
    Rlyeh (mene,wgah,fhtagn-1);
    Rlyeh (mene,fhtagn+1,nagl);

    } // PH'NGLUI MGLW'NAFH CTHULHU!

  • How to convert email addresses into name, age, ethnicity, sexual orientation – This is so Meta – "Save your email list as a CSV file (just comma separate those email addresses). Upload this file to your facebook account as if you wanted to add them as friends. Voila, facebook will give you all the profiles of all those users (in my test, about 80% of my email lists have facebook profiles). Now, click through each profile, and because of the new default facebook settings, which makes all information public, about 95% of the user info is available for you to harvest."
  • Microsoft Security Development Lifecycle (SDL): Tools Repository – A collection of previously internal-only security tools from Microsoft, including anti-xss, fuzz test, fxcop, threat modeling, binscope, now available for free download.
  • Analytics X Prize – Home – Forecast the murder rate in Philadelphia – The Analytics X Prize is an ongoing contest to apply analytics, modeling, and statistics to solve the social problems that affect our cities. It combines the fields of statistics, mathematics, and social science to understand the root causes of dysfunction in our neighborhoods. Understanding these relationships and discovering the most highly correlated variables allows us to deploy our limited resources more effectively and target the variables that will have the greatest positive impact on improvement.
  • PeteSearch: How to find user information from an email address – FindByEmail code released as open-source. You pass it an email address, and it queries 11 different public APIs to discover what information those services have on the user with that email address.
  • Measuring Measures: Beyond PageRank: Learning with Content and Networks – Conclusion: learning based on content and network data is the current state of the art There is a great paper and talk about personalization in Google News they use content for this purpose, and then user click streams to provide personalization, i.e. recommend specific articles within each topical cluster. The issue is content filtering is typically (as we say in research) "way harder." Suppose you have a social graph, a bunch of documents, and you know that some users in the social graph like some documents, and you want to recommend other documents that you think they will like. Using approaches based on Networks, you might consider clustering users based on co-visitaion (they have co-liked some of the documents). This scales great, and it internationalizes great. If you start extracting features from the documents themselves, then what you build for English may not work as well for the Chinese market. In addition, there is far more data in the text than there is in the social graph
  • mikemaccana’s python-docx at master – GitHub – MIT-licensed Python library to read/write Microsoft Word docx format files. "The docx module reads and writes Microsoft Office Word 2007 docx files. These are referred to as 'WordML', 'Office Open XML' and 'Open XML' by Microsoft. They can be opened in Microsoft Office 2007, Microsoft Mac Office 2008, OpenOffice.org 2.2, and Apple iWork 08. The module was created when I was looking for a Python support for MS Word .doc files, but could only find various hacks involving COM automation, calling .net or Java, or automating OpenOffice or MS Office."
  • Handy one-liners for SED – Sed expressions are powerful, but somewhat obscure and easy to screw up. A handy cheat sheet for common tasks.

Bookmarks for June 9th through June 10th

These are my links for June 9th through June 10th:

Bookmarks for June 3rd through June 4th

These are my links for June 3rd through June 4th:

Bookmarks for May 30th through May 31st

These are my links for May 30th through May 31st:

Bookmarks for May 22nd from 06:31 to 07:14

These are my links for May 22nd from 06:31 to 07:14:

  • Javascript Malware Analysis: A Case Study – "This particular beast was found in the wild in May 2009 on a site phishing for Facebook user credentials, and is a particularly-nasty bugger. Note the number of strangely-named variables created up front, many of which are not even referenced in the code blocks that follow. Additionally notice the odd ternary statements which have no impact on the operation of the code, and presumably must exist to trip up scanners (unless there is a fancy form of string replacement on the body of some functions, in which case the functions could be mutated before execution – and that would be scary. A cipher based on the body of the function has also been seen.)"
  • MySQL: Forked beyond repair? | Developer World – InfoWorld – Now that MySQL is part of Oracle, will the forks take over? "if MySQL's approval ratings are slumping, all the more reason for Oracle to move decisively. Oracle must work to regain the trust and support of the MySQL community or risk losing mindshare to a fork, such as Drizzle or MariaDB. To do that, it has to avoid making the mistakes that Sun made when it acquired MySQL. In a sense, to succeed with MySQL, Oracle will have to stop acting like Oracle."
  • Scott Hanselman’s Computer Zen – Less Virtual, More Machine – Windows 7 and the magic of Boot to VHD – Notes on using Windows virtual hard drives to manage instances of multiple version of Windows in parallel, e.g. Windows 7 beta, WinXP, etc.
  • How Opera’s business model works – Communication Breakdown – David Meyer’s Blog at ZDNet.co.uk Community – Around 40M users, "Most of our revenue — 75-80 percent — comes from mobile devices, fom a free browser. We provide the browser for free, like Opera desktop and Mini, and then we generate revenue through our content partners. We provide the search in the right corner and things like that, and that generates revenues in the free distributions. Then you get paid by OEMs [original equipment manufacturers] for distribution — companies like Nokia and Motorola. Most of the mobile OEMs and a fair amount of the other OEMs. We signed up Ford recently and we're now in Ford trucks."
  • Digicorp » Blog Archive » Prevention of Sql Injection with PHP – Notes on good coding hygiene for avoiding SQL injection attacks while processing web form input such as passwords and other text fields.

Bookmarks for May 12th from 10:52 to 21:56

These are my links for May 12th from 10:52 to 21:56:

Bookmarks for April 30th through May 2nd

These are my links for April 30th through May 2nd:

  • FusionCharts Free – Animated Flash Charts and Graphs for ASP, PHP, ASP.NET, JSP, RoR and other web applications – Flash charting component that can be used to render data-driven & animated charts for your web applications and presentations. It is a cross-browser and cross-platform solution that can be used with PHP, Python, Ruby on Rails, ASP, ASP.NET, JSP, ColdFusion, simple HTML pages or even PowerPoint Presentations to deliver interactive and powerful flash charts. You do NOT need to know anything about Flash to use FusionCharts. All you need to know is the language you're programming in.
  • Raphaël—JavaScript Library – Raphaël is a small JavaScript library that should simplify your work with vector graphics on the web. If you want to create your own specific chart or image crop and rotate widget, for example, you can achieve it simply and easily with this library. Raphaël uses the SVG W3C Recommendation and VML as a base for creating graphics. This means every graphical object you create is also a DOM object, so you can attach JavaScript event handlers or modify them later. Raphaël’s goal is to provide an adapter that will make drawing vector art compatible cross-browser and easy.
  • A Really Gentle Introduction to Data Mining | Regular Geek – List of data mining blogs and related resources.
  • BlackBerry SSH Tutorial: Connect to Unix Server using MidpSSH for Mobile Devices – Notes on using MidpSSH on Blackberry for remote access to servers. Seems to work, although big network lag on my BlackBerry Bold / AT&T.
  • Country Reports on Terrorism 2008 – U.S. law requires the Secretary of State to provide Congress, by April 30 of each year, a full and complete report on terrorism with regard to those countries and groups meeting criteria set forth in the legislation. This annual report is entitled Country Reports on Terrorism. Beginning with the report for 2004, it replaced the previously published Patterns of Global Terrorism.
  • DIY: How To Find Authoritative Twitter Users Plus 100 To Get You Started | Ignite Social Media – Some comments on recommendation metrics for Twitter, trying to use "favorites" mark as an indicator.
  • SIGUSR2 > The Power That is GNU Emacs – "If you've never been convinced before that Emacs is the text editor in which dreams are made from, or that inside Emacs there are unicorns manipulating your text, don't expect me to convince you."

Bookmarks for April 3rd through April 7th

These are my links for April 3rd through April 7th:

  • Agile Testing: Experiences deploying a large-scale infrastructure in Amazon EC2 – Practical guidance on using cloud computing at EC2. Expect failures, automate deployment, more.
  • joshua’s blog: on url shorteners – Joshua Schachter (founder of del.icio.us) summary on the state of URL shorteners (tinyurl, bit.ly, etc), and issues with 3rd party redirects, link sharing through twitter, etc.
  • Control Yourself » status.net coming soon – On status.net, plans for hosting laconi.ca sites, and federating microblogging status networks
  • There must be some way out of here (Scripting News) – Comments on the rise of celebrity accounts on Twitter, increasing spam/noise, and alternative models for laconi.ca and status.net
  • Stochastic Models of User-Contributory Web Sites – Tad Hogg, Kristina Lerman 31 Mar 2009 Abstract: We describe a general stochastic processes-based approach to modeling user-contributory web sites, where users create, rate and share content. These models describe aggregate measures of activity and how they arise from simple models of individual users. This approach provides a tractable method to understand user activity on the web site and how this activity depends on web site design choices, especially the choice of what information about other users' behaviors is shown to each user. We illustrate this modeling approach in the context of user-created content on the news rating site Digg.

Bookmarks for February 28th through March 1st

These are my links for February 28th through March 1st:

  • Community Data – Swivel – User contributed datasets, for visualization and graphs with Swivel
  • Obamameter – Map visualization of economic stimulus outlays. "Keep tabs on the the US economy, the global economy and the stimulus through our dashboard for the economy."
  • recovery.gov.pdf – Slide presentation on data sources and construction of initial Recover.gov site in Jan 2009, from talk at Transparency Camp.
  • Virtual Hoff : DoxPara Research – Slides from Dan Kaminsky's talk at CloudCamp Seattle on network and application security issues in cloud and virtualized computing environments.
  • Can You Buy a Silicon Valley? Maybe. – from Paul Graham – "If you could get startups to stick to your town for a million apiece, then for a billion dollars you could bring in a thousand startups. That probably wouldn't push you past Silicon Valley itself, but it might get you second place. For the price of a football stadium, any town that was decent to live in could make itself one of the biggest startup hubs in the world."
  • Berkshire Hathaway 2008 shareholders letter (PDF) – Warren Buffet reviews the state of the financial markets, his worst year ever, and the outlook for 2009.
  • White House 2: Where YOU set the nation’s priorities – Not the actual White House, but an interesting experiment in collaborative input for setting government agenda.
  • Python for Lisp Programmers – Peter Norvig examines Python. "(Although it wasn't my intent, Python programers have told me this page has helped them learn Lisp.) Basically, Python can be seen as a dialect of Lisp with "traditional" syntax (what Lisp people call "infix" or "m-lisp" syntax). One message on comp.lang.python said "I never understood why LISP was a good idea until I started playing with python." Python supports all of Lisp's essential features except macros, and you don't miss macros all that much because it does have eval, and operator overloading, and regular expression parsing, so you can create custom languages that way. "

Bookmarks for February 27th through February 28th

These are my links for February 27th through February 28th:

Bookmarks for February 26th from 10:39 to 20:05

These are my links for February 26th from 10:39 to 20:05:

Bookmarks for February 23rd through February 24th

These are my links for February 23rd through February 24th:

Bookmarks for February 16th through February 17th

These are my links for February 16th through February 17th:

  • Top 100 Network Security Tools – Many many security testing and hacking tools.
  • FRONTLINE: inside the meltdown: watch the full program – "On Thursday, Sept. 18, 2008, the astonished leadership of the U.S. Congress was told in a private session by the chairman of the Federal Reserve that the American economy was in grave danger of a complete meltdown within a matter of days. "There was literally a pause in that room where the oxygen left," says Sen. Christopher Dodd"
  • The Dark Matter of a Startup – "Every successful startup that I have seen has someone within their ranks that just kinda “does stuff.” No one really knows specifically what they do, but its vital to the success of the startup."
  • Why I Hate Frameworks – "A hammer?" he asks. "Nobody really buys hammers anymore. They're kind of old fashioned…we started selling schematic diagrams for hammer factories, enabling our clients to build their own hammer factories, custom engineered to manufacture only the kinds of hammers that they would actually need."
  • Mining The Thought Stream – Lots of comments around what is Twitter good for and how will it make money, revolving around real/near-time search, analytics, marketing, etc.
  • Understanding Web Operations Culture – the Graph & Data Obsession … – Comparison of traffic at Flickr, Google, Twitter, last.fm during the Obama inauguration. "One of the most interesting parts of running a large website is watching the effects of unrelated events affecting user traffic in aggregate."

The inside of my Thinkpad T42p

The inside of my Thinkpad T42p 

This morning the IBM service tech came to replace the failed fan assembly in my Thiinkpad T42p. The Thinkpad has been fairly indestructable, having gone around the world several times without any problems. So I was surprised when I started getting “Fan Error” messages just after the BIOS splash screen while setting up on Sunday evening. Fortunately, I also got the 24-hour onsite support contract back when I got the system. It ended up taking more like 36 hours to get someone out here, but I did call in the middle of the night.

That reddish assembly at the middle left is the heatsink and fan. The system board runs a test to make sure the fan will spin up before proceeding with the system boot process; the original fan will spin manually, but the motor seems to have failed. I’m glad to have the technician replace the fan instead of doing it myself. Getting the heatsink off the graphics chip required some significant prodding with a sharp knife to unbond the heat compound sticking them together.

The past day and a half I’ve been using other computers around the office, which has been kind of strange. Even though they’re all part of my normal setup, nothing is in the right place, since I keep reaching or looking in a different direction than usual. It’s been like working in someone else’s office. This evening I’ve gotten everything synced back up, but probably need to start thinking about migrating off the Thinkpad at some point as it continues to age.

Hacked by keymachine.de

I just noticed that my WordPress installation got hacked by a search engine spam injection attack sometime in the past few weeks. This particular one inserts invisible text with lots of keywords in footer.php. The changes to the file were made using the built-in theme editor, originating from ns.km20725.keymachine.de, which is currently at 84.19.188.144. The spam campaign automatically updates the spam payload every day or so. The links point to a variety of servers that have also been hacked to host the spam content. Here is a sample: http://www.nanosolar.com/feb3/talk.php?28/82138131762.html
I’ve sent an e-mail to Nanosolar, so they’ll probably have that content cleaned up before long. But the automated SEO spam campaign updates the keyword and link payload regularly, so any affected WordPress sites will be updated to point at the new hosting victims.

From a quick check on Google, it looks like keymachine.de is a regular offender

Slow load times ahead

Dreamhost has been limping badly today, and will apparently be slow for a few more days. Even their status page has temporarily been replaced with a plain HTML version. So you may be experiencing very slow load times here for a while.

We have been experiencing Network Problems Today, these are the same problems that have actually been happening since we first reported problems with our network. Unfortunately these problems have gotten worse today and are causing a majority of the downtime and slowness issues you are reporting today. These problems, and our attempts at fixing them, have been an ongoing effort. The maintenance Monday Night is a big step in resolving these problems. We are working on the network and all servers having problems, but the real solution will be the Monday Night maintenance. Sorry about the downtime, we hope to have this all resolved soon.
- Sep. 8, 2006 5:45 p.m.

Random Dreamhost issues

In case you were wondering where the site went, the past 24 hours or so has been a day of random issues with Dreamhost.

Yesterday afternoon they were having connectivity problems, which took all their customers offline for a few hours.

This morning, I discovered that this site was running, but all Dreamhost sites were unreachable via SBC/PacBell here in the Bay Area. From the logs it looks like Comcast and a variety of overseas networks were still able to connect. The Google proxy hack mentioned this morning on O’Reilly provided another quick path for looking at the web site from a different network to verify that connectivity was still working, at least from the Google data center.

A couple of hours ago I got what I thought was a response to my e-mail regarding the network connectivity problem, but which turned out to be one of the CPU utilization warning letters that have been going out lately:

[your] CPU minute usage for today is 56.15. The daily limit is 60 CPU minutes. You will continue to receive these notifications as long as your resource consumption is over 50 CPU minutes.

A little mysterious, since traffic to the site was off because of the network outage, and spam traffic hasn’t spiked either.

There aren’t any resource utilization logs posted yet. I wonder if the flaky networking over the past day contributed to the high CPU use by leaving a lot of processes around waiting for I/O that was coming in slowly or never.

Anyway, the site seems to be running normally as of this afternoon (or at least, I can get to it now).

See also: Dreamhost load average = 1004.16?

Mod_rewrite for moving web content to a new domain

I just wasted 10 minutes getting this to work correctly, so I thought I’d write it down…

Here’s what you need to use mod_rewrite to implement a permanent 301 Moved HTTP response when you move a web site from a subdirectory on one domain to a new top level domain.

(Assuming you’re on a hosted service, and can use .htaccess):

RewriteEngine on
RewriteBase /
RewriteRule ^olddir/?(.*)$ http://new-domain.com/$1  [R=permanent,L]

where the old content was originally in a subdirectory called “olddir” and is getting moved to a new directory on a different server.

This allows you to move the content to a new, separate domain and/or server without breaking your existing links.

link: more on .htaccess and mod_rewrite in the Apache documentation

Blocking Referrer Spam

This afternoon, I’ve noticed there’s a steady stream of HTTP referrer (aka referer) spam originating from a few IP addresses, so I’m finally getting around to making some updates to reduce the volume of spam traffic. In the past I’ve been getting a few spam referrers here and there, but today there are thousands in just a few hours, and these changes are a bit overdue.

Here are the IP addresses sending me spam today:

64.193.62.232
70.84.211.130
69.28.242.87

All of the HTTP requests are HEAD only, not GET. Here’s a typical one:

64.193.62.232 - - [02/Oct/2005:14:34:34 -0700]
    "HEAD / HTTP/1.1" 403 - "http://cheap-vicodin.none.pl"
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Notice the 403 Forbidden status code. That’s because I’ve added a section to .htaccess to block referrers with spammy keywords, and also to manually block IP addresses. Here’s an abbreviated version:

deny from 64.193.62.232
deny from 70.84.211.130
deny from 69.28.242.87

RewriteEngine on
RewriteCond %{HTTP_REFERER} ^(http://)?(www\.)?.*(-|.)vicodin(-|.).*$ [NC,OR]
< ...lots of other rules go here...>
RewriteRule .* - [F,L]

One convenient aspect of having non-stop incoming spam today is being able to make changes and immediately observe the effect. It’s modestly gratifying to see all the “200 OK” turn into “403 Forbidden” status.

The current block list I’m using for .htaccess is mostly from a list maintained by Aaron Logan.

I also looked through suggestions for .htaccess changes and block lists for referrer spam by Joe Maller, Dave Child, and Mike Healan.

Unfortunately, all of these approaches, especially the IP blocking, are manual processes. I’ve been meaning to get Bad Behavior implemented here, but this was a quick fix for today.

Update 10-06-2005 08:25 PDT: Still getting lots of incoming spam traffic, plus many new IP addresses showing up now. Here’s the revised block list, all of these addresses are actively sending spam.

deny from 64.193.62.232
deny from 70.84.211.130
deny from 69.28.242.87
deny from 66.246.218.114
deny from 71.57.133.162
deny from 67.186.112.106
deny from 84.139.88.151
deny from 172.202.144.111
deny from 172.206.206.111
deny from 210.213.132.240
deny from 195.252.85.29
deny from 200.116.118.149
deny from 83.109.41.39
deny from 68.228.171.28
deny from 71.57.17.237
deny from 211.30.20.3
deny from 65.1.135.21
deny from 200.116.118.149
deny from 85.140.26.144
deny from 60.228.205.13
deny from 172.195.205.18
deny from 218.111.180.243
deny from 194.158.220.138
deny from 24.239.174.55
deny from 84.110.62.170
deny from 84.58.193.189
deny from 221.97.4.165
deny from 85.140.26.144
deny from 220.137.197.52
deny from 201.8.242.11
deny from 202.81.183.165
deny from 201.240.21.13
deny from 211.223.170.139
deny from 82.229.255.13