Statistical Data Mining Tutorials – A nice collection of presentations reviewing topics in data mining and machine learning. e.g. "HillClimbing, Simulated Annealing and Genetic Algorithms. Some very useful algorithms, to be used only in case of emergency." These include classification algorithms such as decision trees, neural nets, Bayesian classifiers, Support Vector Machines and cased-based (aka non-parametric) learning. They include regression algorithms such as multivariate polynomial regression, MARS, Locally Weighted Regression, GMDH and neural nets. And they include other data mining operations such as clustering (mixture models, k-means and hierarchical), Bayesian networks and Reinforcement Learning.
Ho John Lee | November 14th, 2007 | Comments are closed
Another day, another subprime-related fiasco. Today GE Asset Management announced that one of its not-quite-money-market short bond funds, the Enhanced Cash Trust, took a loss from subprime holdings, and is offering customer redemptions at 96 cents on the dollar. Normally these funds are considered to be a higher-yielding version of a money market fund. This would make you pretty unhappy if you were looking for 5%-ish stable returns while waiting for the stock market to settle down.
Along these lines, here are British comedians John Fortune & John Bird chatting about the state of the banking system, Northern Rock, and subprime in another interview of “George Parr, investment banker” from last month.
This afternoon I was in the audience at the CNBC Fast Money show, which was taping on the road at the Computer History Museum. Many of the attendees were customers at Charles Schwab, which is holding a local workshop for active trading clients this weekend. People seemed to be in good spirits despite today’s down market.
In addition to the cast of Fast Money, we had Paul Otellini from Intel, who brought along a UMPC and a silicon wafer. Also making an appearance was the Trading Goddess, although I may have been the only one who recognized her handle during the Q&A session.
The Computer History Museum has an interesting collection of preserved techno-rubble. In the past, I’ve worked with a remarkably large number of the items displayed in the exhibits. I also enjoyed the Cray-1 sitting out near the entrance.
Both CNBC and TheStreet.com have separately reported problems with their respective stock trading contests.
The CNBC contest site apparently had a coding flaw that allowed traders to open an order entry window before market close, then wait until after hours to actually enter the order, which would be executed the closing price.
The more interesting comment from CNBC is this:
In addition, there have been allegations that one or more contestants may have engaged in illegal market manipulation to affect actual prices of stocks represented in their contest portfolios.
With a million dollars at stake, this seemed like a plausible scenario even before the contest started. It’s unclear to me whether “illegal market manipulation” here actually means “illegal under contest rules” or “illegal under SEC regulations”, but it seems quite feasible.
Along similar lines at TheStreet.com, which offered $100,000 in their contest:
The final results of the game indicate that players employed trading strategies to achieve returns that could not be duplicated in the real world, thereby depriving other contestants of an equal chance to win.
I don’t know why these large contests don’t just partner with a retail brokerage that already has a paper trading (“demo”) mode instead of building their own. Wouldn’t solve the market manipulation problem though.
I’m currently in the top 4% at 14746 in the CNBC contest. Oddly, they don’t publish the portfolio values on their website, but did show the current portfolio values and standings on TV last Friday. The top positions are clustered around $2M, or a total return of around 100%. It would be interesting to know what the distribution of 1-week returns looked like. That’s probably the best opportunity for placing at this point.
In either case, I haven’t hit any 25% to 50% movers, which looks like a prequisite for placing in the standings so far.
It’s hard to find high volatility stocks that are likely to move up in this market. I made a little progress on the CNBC stock contest, but this is going to be a lot lower tomorrow when they update the results (seeing how the markets had another abysmal day). It would be a lot more interesting if you could take short positions. Everyone would probably be in NEW and LEND.
As an aside, the rules for the trading contest apparently don’t prohibit multiple accounts owned by the same person. Consequently, someone named “Nancy Beaumont” has entered hundreds of portfolios. By the rules, they can only win once, but having multiple high-volatility portfolios in the running significantly improves your chance of ending up with one or two successful outliers, sort of like buying multiple lottery tickets, but with much better odds. At the moment they’re occupying 8 of the top 20 rankings.
CNBC is running a stock trading contest starting today, with a prize of one million dollars for the best performance by May 25th. Signing up is free. Each participant gets a notional $1,000,000 to trade. There are a few non-obvious rules:
All trades are priced at end-of-day – no intraday trades, and no GTC limit orders.
No ETFs – this means no indexes, countries, sectors, or commodities, and no inverse trades
No mutual funds – again, no indexes, country, sector, commodity, or inverse vehicles
No options, futures, or other derivatives
Minimum market cap of $500 million as of the starting date – this is probably to keep people from winning by manipulating a thinly traded microcap stock.
In order to win this type of contest, you pretty much have to treat it like a free lottery ticket and make your selections accordingly. There is no risk-adjusted return to consider, and no downside to taking extreme losses. So on a short timeframe, and in a long-only US stock portfolio, you’re looking for something highly speculative that’s going to move a lot.
Before I read the rules, I was thinking that with a million real dollars at stake, someone would game the system by thrashing a microcap issue back and forth, but the $500MM market cap requirement makes it harder. Not out of the question, but probably not worthwhile for anyone in a position to control the stock price.
Since we can’t trade the easily manipulated microcap stocks, I’d look for low float, heavily shorted small cap stocks that have some positive opportunity for event or news driven movement. Some general areas to look:
Biotech development companies that have regulatory approval or conferences coming up
Despised companies that are potential buyout candidates
Disaster / event driven companies, such as avian flu, anti-terrorism, or whatever you can think of
Fad driven trades. Last year it was ethanol and energy drinks. It might be ethanol and alt-energy again this year
These are all exactly the opposite of good investment or trading approaches, but the contest has a binary outcome – either you win or you don’t. By taking on extreme levels of portfolio risk, you’re trying to increase the variance of portfolio returns during the contest period, in hope of coming up with the highest outlier by the end.
A contrarian approach might be to stay in cash or very conservative stocks, and hope that everyone else makes speculative bets that crash and burn.
Hmm. I just looked at the rules again, and they also have $10,000 prizes for each week’s best 1-week return. This actually increases the incentive for finding a more-or-less binary, news-driven trade each week (such as earnings, court ruling, regulatory approvals). If it works out, your portfolio goes up. If it goes down, it doesn’t affect your 1-week return for the following week.
I was going to run a scan for this over the weekend, but perhaps I should try to dig something up for this week in case something works out.
Needless to say, this isn’t how one should invest or trade with real money.
Here’s a look at today’s market heat map after the close. There’s a lot of green, but this was pretty unenthusiastic.
I wasn’t too impressed with John Thain on CNBC today explaining yesterday’s weirdness with the NYSE trading and reporting systems. The official story seems to be that the systems that compute the DJIA got backlogged with transactions, and separately an internal messaging system for floor traders also got backlogged, so they’re upgrading their servers. Today they had to delay closing a number of stocks to allow transaction queues to clear at the end of the day. Thain mentioned a volume of over 20,000 msgs/second. This is all plausible, but not reassuring. It sounds like something that would happen to a growing e-commerce site, not one of the world’s largest stock exchanges. I wonder how much reserve throughput they can actually deploy. If we get a “real” market crash, they’re going to have to handle a lot more than the 2.4 billion shares they traded yesterday.
It’s interesting to observe that diversification across asset classes and markets didn’t help you today. All 30 Dow stocks closed down. 99 of 100 Nasdaq-100 stocks closed down. Nearly all of the S&P 500 closed down. Oil, gold, and other commodities closed down. Emerging markets closed down. Basically, equities and commodities got sold, and the proceeds went to cash and bonds. (Update – here’s the summary from today’s Worden Report: “Zero Industry groups advanced while 239 declined. There was one winner in the Nasdaq-100, two in the SP-500 and zero in the Dow. HalfPoint+ Movers were seven against 2174. The Leadership Index was 34 versus 2228.”)
One nominal trigger for today’s selling was a 9% drop on the Shanghai exchange, but there have been any number of reasons to be concerned and raise a little cash for a while.
In general, diversifying an investment portfolio across asset classes and markets reduces overall risk for an equivalent level of returns. This works because the price behavior for different markets is supposed to be relatively uncorrelated over time. Lately, disparate markets have been more correlated than in the past, mostly going up. Today the risk was clearly to the down side, making it likely that your investment portfolio closed lower today, unless you were in cash or bonds. (I’m pleased to have reduced my trading positions in India and China over the past couple of weeks.)
Another unexpected systemic risk exposed today was in the odd behavior of the NYSE around 3pm. The new hybrid (electronic and open outcry) trading system was apparently getting backed up due to heavy order flow this afternoon. The DJIA appeared to gap down by 180 points when the backlog was cleared. Anyone trading intraday off a NYSE data feed probably had some problems. (Update – here’s TraderMike with more detail)
I think the price action today is overdone, but I’m also happy to have exited many of my positions in India, China, and other emerging markets over the past few weeks. This is a good time to think about where to invest after the dust settles, or focus on short term and day trading. (Some of you may be interested in using the Ultra and Ultrashort ETFs.)
In the past ten years (1996-2006), the NAHB homebuilders index tends to lead the performance of the S&P 500 by 12 months. The index goes from is based on a survey of homebuilding companies views on current sales, the outlook for the next 6 months, and the current level of prospective buyer traffic. This month was the 7th monthly drop in a row, and is a 15-year low.
In the period from 1985 to 1996, there is no correlation between the housing and stock market, so this could optimistically be viewed as a temporary coincidence. On the other hand, asset class correlations have been going up for a while. Draw your own conclusions, but real estate prices, home builders, and mortgage lenders are clearly having a difficult time recently.
John Mauldin has pulled together some observations on the housing market in his newsletter this week, leading with the graph above from David Rosenberg at Merrill Lynch. (Free e-mail registration required, but worthwhile reading.)
Here in the Bay Area, real estate prices are chronically high, ranging from “insanely high” to just “overpriced”. Here’s the pessimistic view. At least you can live in your overpriced house if you have enough cash to support it. I know of at least one dot-com zillionaire who lucked out by overpaying for his house in cash before the crash. The house went from something like $6M to $4M, but his stock went from something like $100M to $2M.
Silicon Valley is built on optimism and entrepreneurship, but lately, most tech companies can do no good in the eyes of public market investors, who are presently in a mood to sell on no news, bad news, or even good news.
At the same time, private market sentiment toward investing in Web 2.0, online services, and consumer media and publishing appears to be positive.
A lot of the problem with public tech company share prices is due to uncertainty about future prospects – a slowing economy, growing competition, increasing costs, and a general cloud of unknowable liability from options accounting issues. The actual businesses are often doing OK or even great, but investor sentiment has shifted, bringing down the share price. See BRCM, RACK, or NVDA for a few examples of what happens when you report a decent quarter without boosting forward guidance. Fewer people are willing to pay a 30 multiple for growth that may never come, or that may never have existed in the first place.
In contrast, there is still a lot of investor love for Web 2.0 startups and other “new” online services. Part of this reflects supply and demand — there are a lot of investable funds around, and it’s hard for a fund to invest a lot of money in small chunks.
There’s still a lot of excitement about the future of Web 2.0 et al, but it’s been feeling overdone for the past few months (“Digg is worth $60M“), without necessarily being over. On the other hand, I still get the impression that people here in Silicon Valley are still somewhere between denial (“It will go back up”) and fear (“What if it doesn’t”) with respect to the medium term prospects for tech stock share prices.
A couple of weeks ago, Apple Computer announced a better-than-expected quarter, and also mentioned that they didn’t expect any “material adjustments” to result from their stock options accounting investigation. Since then, the share price has been rocketing from the low 50s up to touch 70 today.
After the close today, they mentioned that they will restate earnings back as far as 2002 and will delay filing their 10-Q. Oops.
Apple Computer Inc. (NASDAQ:AAPL – News) said on Thursday it would likely need to restate earnings and will delay filing its quarterly report because of additional irregularities it found in its accounting of stock options and its shares fell 6.6 percent.
Apple also said in a SEC regulatory filing Thursday that all financial communications issued since September 29, 2002, should not be relied upon. The irregularities are related to the issuance of stock option grants made between 1997 and 2001.
As previously announced, an internal investigation discovered irregularities related to the issuance of certain stock option grants made between 1997 and 2001. A special committee of Apple’s outside directors has hired independent counsel to perform an investigation and the Company has informed the SEC. At this time, based upon the irregularities identified to date, management does not anticipate any material adjustment to the financial results included in this earnings release. However, if additional irregularities are identified by the independent investigation, a material adjustment to the financial information could be required.
It can be difficult to have a lot of confidence in financial statements published by publicly traded companies. Companies built on intellectual property or financial instruments are especially prone to varying interpretation of their finances, but accounting issues seem to plague companies from every industry these days.
Re: how can we ever trust any financial statements at all? An old joke an accountant once told me…
A business man was interviewing applicants for the position of divisional manager. He devised a simple test to select the most suitable person for the job. He asked each applicant the question, “What is two and two?”
The first interviewee was a journalist. His answer was “Twenty-two.”
The second was a social worker. She said, “I don’t know the answer but I’m glad we had time to discuss this important question.”
The third applicant was an engineer. He pulled out a slide rule and showed the answer to be between 3.999 and 4.001.
The next person was a lawyer. He stated that in the case of Jenkins v Commr of Stamp Duties (Qld), two and two was proven to be four.
The last applicant was an accountant. The business man asked him, “How much is two and two?”
The accountant got up from his chair, went over to the door and closed it then came back and sat down. He leaned across the desk and said in a low voice, “How much do you want it to be?”
If you’re an stock investor, you might be considering the possibility that the markets will go down (more) sometime in the near future.
The recently launched ProShares ETFs use derivatives to implement inverse and levered versions of several popular market tracking ETFs, including the Nasdaq-100 (QQQQ), S&P 500 (SPY), DJ-30 (DIA), and S&P Midcap (MDY).
The Short ProShares provide inverse performance to the underlying index, while the Ultra ProShares provide 2x the performance of the underlying index.
The performance doesn’t track perfectly. For example, today the Q’s are down by 0.88%, and the PSQ (inverse Q’s) are up 1.39% while the QLD (2x Q’s) are down by 1.66%. At the moment they also don’t appear to track intraday price movement very tightly, which might be a function of the relatively low volume. You can see the modest tracking divergence in the 5-min intraday chart above.
These aren’t the sort of instrument you’d want to hold for years, but could be handy for investors trying to hedge market risk (especially in self-directed IRAs, which don’t allow short selling). I also think a lot of people simply aren’t comfortable with short selling or derivatives, and this makes it much simpler for people to get short or lever up. If your account allows it and you’re already comfortable with short selling, it’s simpler and more efficient to just short the Q’s or other indices directly, though.
Prior to the ProShare ETFs, the Rydex mutual funds have been around for a long time, implementing inverse and levered index and sector funds. The main advantage of using ETFs here is that they trade continuously rather than once per day, making them more flexible as a hedging or trading vehicle.
Caution: If you have no idea what I’m talking about but are thinking about buying some because you think the market is going down, please don’t unless you check it out first and know what you’re getting. While the short and levered ETFs might be handy investment tools, they can just as easily provide you with a faster, easier way to lose your money if you’re not paying attention.
Disclaimer: Not investment advice. Trade at your own risk. Not affiliated with ProFunds or Rydex. You buy it, you own it.
Update Wed 07-12-2006 13:50PDT – Took a look at these today during market hours. The thin volume results in relatively wide spreads, 0.10 or more. I haven’t tried them yet, but Barry Ritholtz has written a note about his experiment with the PSQ (inverse Q’s), with mixed impressions.
Update Thu 07-13-2006 16:42PDT – Proshares introduces 2x inverse ETFs for the Q’s, S&P, DJIA, and S&P400 starting today – QID, SDS, DXD, MZZ. They picked a good day to launch.
Changing your mind about an investment – intellectual, emotional, or financial – is difficult.
Munjal Shah, founder and CEO of photo search company Riya, has been writing a series of posts narrating his experience over the past few months getting his startup off the ground. Highly recommended reading for entrepreneurs and others with an interest in startups, online web services and software applications.
In startups and other forms of investing, it’s common to have a vision or thesis guiding your decisions. Technology companies, and especially technology startups, tend to be big on vision, which can be helpful in achieving internal alignment and for building external awareness among potential customers and partners. It can give everyone a common sense of purpose or mission, and can sometimes take on a religious or political tone (i.e. Apple fanatics, the first Internet Boom, George Gilder’s Telecosm, or lately “Web 2.0″).
An important skill is recognizing when the facts on the ground are diverging from the vision and it’s time to reconsider. Some relevant investing adages:
It’s cheaper to lose your opinion than to lose your money
It can be very expensive to convince the market that you’re right
The market can stay irrational longer than you can stay solvent
In today’s installment of his startup journal, Munjal recounts their discovery that their users weren’t behaving the way they had expected. Riya was conceived as a platform for users to upload their own images for tagging and search, but more and more people come to the site to search for images without uploading anything. The Riya team has been puzzling over what’s going on with their beta users for weeks when they see:
The answer to the question we had been seeking. It didn’t come from our brains but from the silent majority of people who were using our site each and every day. For most people Riya was not a replacement to their Flickr accounts. Riya was not a replacement to Y! Photos. Riya was becoming a replacement for… you guessed it: public web image search!!
Munjal’s comment on entrepreneurs’ drive to change the world with the power of their vision, and the corresponding difficulty in assessing incongruent facts, is right on:
Entrepreneurs are an odd bunch. As an entrepreneur you create a vision of what can be and then work really hard to make that happen. It is your imprint on the world. It is your legacy. Maybe 2000 years ago if you wanted to leave a mark you would be Julius Cesear or Genghis Khan. Today you start a technology or Internet company. I believe almost all entrepreneurs seek immortality through their products. This is one of the reasons we all seek to build products that are used by and benefit the lives of as many people as possible. We want to do good, but we also want to be remembered. Some admit this and some don’t, but it is true. The greatest crusades in the world are always for the intangible. There is no other explanation for why founder’s of companies work so hard and sacrifice so much. Money can only account for so much of this. You have to believe that you are on this planet to somehow change it.
However, this is the achilles heel of the entrepreneur when changing strategies.
The desire to imprint can leave you deaf to the input from your customers. Like Alexander your men can want to go home and you want to press further into India. So you ignore the facts and continue trying to imprint your first vision on the world. When the data doesn’t support you, you say words like:
“We were ahead of our time.” – Full Denial
“I just have to hold out longer and people will see the value.” – Almost full denial
“If we only had this feature people will use it” – Medium denial
As a startup, Riya is in the midst of defining (and redefining) itself, and is taking rapid steps to revise their vision, rather than ignoring the mismatch between their original vision and the behavior of their growing user base.
Some would argue that they should stick with their original vision, the one that’s gotten them this far. But when a startup business is faced with flat to slow growth in it’s initial proposition versus strong demand growth in an unheralded aspect of their offering, it’s a good time to consider whether a new vision is order.
Long ago, I once had a company that spent a lot of time and effort building an advanced signal processing system, only to discover that most people wanted to do very simple things with it. It took us a while to realize that our “vision” helped customers find us, but most of them actually wanted something different than what we had in mind. (It didn’t help that were really proud of our product.) We eventually came out with a product that actually did what customers were asking for, which was predictably, far more successful than the original.
One of the things I’ve learned (and am still learning) over time is the value of recognizing and reevaluating a broken thesis (or vision), the sooner the better. One of the biggest inhibitors to changing one’s mind is the fear of looking silly, that other people will think badly of your decision. This can be an expensive fear, when it causes you to stubbornly follow an investment thesis or startup vision which is showing clear signs of being flawed.
It looks like Munjal and the Riya team have elected to change the vision rather than trying to convince the market that they were “right”. I’m looking forward to seeing how the new vision shapes up.
Two of the most important skills for long term investment success are capital allocation and risk management.
Capital allocation tends to get most of the attention in the press, which publishes headlines like “Where To Put Your Money Now”, or “10 Hot Stocks To Watch”, or perhaps a list of recently funded startups. This is especially true when markets are good and everything is going up.
Risk management isn’t nearly as interesting to the average person. It’s much more interesting to hear about the 800% return in titanium stocks over the past year than the possibility of losing money.
The past couple of weeks have reacquainted investors with market risk. For new investors who haven’t seen a down market in action, this is especially painful. The Indian stock market has been popular with emerging market investors in the US, as well as in India. It had been up around 50% since the beginning of the year but dropped 10% in a single session, and around 30% in a few days last week, triggering margin calls and placing police on suicide watch. The decline in US markets has been tame by comparison.
This afternoon the Enron trial came to an end, with Jeff Skilling and Ken Lay found guilty of fraud, conspiracy, and other charges in the collapse of what was once a $68 billion dollar company.
It wouldn’t have prevented you from losing money, but it would have kept you solvent. The buy-and-hold employee and retiree shareholders probably weren’t so lucky.
I recently became aware of a risk I hadn’t thought of in a while – platform risk. Yesterday I had a profitable short term trade locked in with a protective stop which failed to trigger when the share price turned and passed through the stop level. I asked Schwab to investigate, and it turns out they may be having intermittent problems with certain trailing stops entered through the trading window rather than the custom alerts window in Streetsmart Pro, their client application. They adjusted my trade to reflect the correct stop value, which was logged on their server but not executed.
Trading with unreliable stops in the current market is like driving a car with brakes that don’t always work. I appreciate Schwab making good on the trades, but I hadn’t considered the possibility of correctly entered stops not being executed when hit before yesterday. Kind of like people hadn’t considered the possibility of widespread fraud at Enron before it blew up.
Asset allocation is important, but risk management can keep you in the game when the unexpected inevitably happens.
I sometimes have CNBC on in my office, sort of as background noise. This afternoon I heard Maria Bartiromo describing her weekend conversation with Federal Reserve chairman Ben Bernanke.
The media and the markets basically got it wrong last week in speculating that the Fed is done raising interest rates.
I’m thinking to myself, “Hmm. That’s going to make some people uncomfortable with their positions.” Especially since this was pretty much the opposite of what people were thinking after last week’s Fed testimony.
Last Thursday, Bernanke told a congressional panel the central bank could pause — but not necessarily stop — its string of rate hikes while it keeps a close watch on the economy’s health. However, according to CNBC, Bernanke said future increases will depend mostly on economic data; that stand was troubling to an interest rate-sensitive market.
A spokeswoman for the Federal Reserve, Michelle Smith, declined to comment on the CNBC report
Well, that was fun. Some quick thoughts:
If this was intended as clarification of Fed policy, why not say so directly, instead of indirectly? And why not officially comment when given the opportunity?
Why did CNBC sit on this all day, instead of reporting pre-open, when they already had the story in hand?
Who profited from this? This would have been handy information to have in one’s pocket this morning…
These clearly demonstrate that B-school students have too much spare time on their hands.
Postscript: I see from this post that it’s actually a close look-alike (student), not actually Glenn Hubbard.
Update 04-28-2006 9:49 PDT – This clip has been making the rounds pretty quickly. They just showed it on CNBC, where they’ve also got the Columbia Business School students who made the video. 15 minutes of fame…
News events plotted on the stock chart timeline. I wish they’d add this to Yahoo Finance.
Ajax UI for scrolling the stock chart around and changing the time window
Recent blog search results on the right sidebar (although they seem to be a few hours behind)
I wish for:
More charting features. There basically aren’t any right now.
Better integration of the “More Resources” features. Things like SEC filings, institutional holders, and earning estimates are all provided by 3rd parties via outbound links, making it hard to flip through.
Technical charting and research reports are provided via Yahoo Finance, although the discussions are hosted at Google Groups.
The feature I’d really like to see is an intelligently filtered view of the Yahoo Finance discussion boards. There is some interesting and useful information there, but a far larger quantity of rants, spam, and trolling in between.