Recently, in my TraderFeed blog, I suggested that traders faced a greater emotional hurdle than either fear or greed: overconfidence. Overconfidence is what leads us to take on too much risk for too little reward. It is what allows us to wager our hard-earned money on untested and unproven market signals. Indeed, almost by definition, beginning traders start their trading careers in an overconfident state. After all, in what other performance field--sports, music, or chess--would a newcomer enter a competition with experienced professionals and truly hope to compete?

Research reviewed by Scott Plous in his book "The Psychology of Judgment and Decision Making" suggests that overconfidence is greatest in situations where individuals have no better than chance odds of being correct in their judgments. One of the reasons for this is called the Gambler’s Fallacy. A person who guesses market direction once a day and has a 50/50 chance of being correct will encounter, on average, about six occasions per year in which he or she is correct five times in a row. Some of these random traders will, by sheer good fortune, hit this streak early on in their career. According to researcher Ellen Langer, early (but random) experiences of success lead individuals to be highly confident in their ability--even when the task is guessing the outcome of coin tosses! This is because they are more likely to attribute success to internal factors--skill--than to situational reasons or chance. The gambler who experiences a (random) streak of wins becomes convinced that he has a hot hand and raises his bets accordingly. The result is predictably disastrous.

Do traders behave differently from gamblers? Research suggests not. Terence Odean found that traders who were most confident in their decision-making traded the most frequently--and lost more money than other traders because of the increased transaction costs. A provocative study from the London Business School presented traders with price data and asked the traders to make trading decisions based on the data. Traders were not informed that the data were generated randomly. The traders who expressed the greatest confidence in their decisions, not surprisingly, were also the ones who, on average, lost the most money. The “illusions of control” demonstrated by these traders can reach extremes bordering on the absurd. In Langer’s studies in which coin tosses were presented as tests of “social cues”, for instance, 40% of all subjects insisted that their ability to guess the outcome of the coin tosses could be improved with practice--and 15% believed that enhanced concentration and an absence of distractions would improve their results.

A different kind of overconfidence can be seen in surveys of traders and investors, asking them for their expectations for the market. Traders feel better about their ability to call market direction than is warranted. For example, in such surveys as those conducted by Investors Intelligence, over 70% of respondents pronounce themselves either bulls or bears--despite the fact that the majority of time the market is range bound. Research cited by Hersh Shefrin, in his review of behavioral finance studies entitled Beyond Greed and Fear, finds that traders are most bullish after extended rises--with inexperienced traders most bullish of all. That is paradoxical, because market returns historically have been greatest following years of decline, not years of strength. Similarly, it is not uncommon to see put-call ratios elevated after a five-day period of decline, despite the fact that returns, on average, are superior following five days of weakness than after five strong days. Quite simply, traders extrapolate from the past to the future--and confidently act upon these (false) expectations.

Is it possible to immunize oneself from overconfidence? My personal therapy for treating overconfidence has been to test out my trading ideas and calculate: a) precisely how often the pattern would have been successful if used in the past; and b) how much of a P/L edge was present over that time. Those statistics, which I report on my blog, ground me in the inherent uncertainty of markets and prepare me for the very real possibility, with any trade idea, that I will be wrong. This, in turn, has helped me greatly with risk management, as I am unlikely to wager a large proportion of my trading stake on any uncertain proposition--even when the odds are in my favor. The past is hardly a guarantor of the future, but by assuming that the future won’t be better than the past, we can soberly assess the downside and avoid overconfidence.

The best trades, I find, have enough of a historical edge to make me feel confident about the idea, but also enough potential downside to prevent me from feeling overconfident. Planning for each trade being a potential loser may seem counterintuitive, but it keeps risk management sharp and overconfidence at bay. And that makes a world of difference to the bottom line.

Brett N. Steenbarger, Ph.D. is Associate Clinical Professor of Psychiatry and Behavioral Sciences at SUNY Upstate Medical University in Syracuse NY and author of The Psychology of Trading (Wiley, 2003). As Director of Trader Development for Kingstree Trading, LLC in Chicago, he has mentored numerous professional traders and coordinated a training program for traders. An active trader of the stock indexes, Brett utilizes statistically-based pattern recognition for intraday trading. Brett does not offer commercial services to traders, but maintains an archive of articles and a trading blog at www.brettsteenbarger.com and a blog of market analytics at www.traderfeed.blogspot.com. His book, Enhancing Trader Development, is due for publication this fall (Wiley).