I remember dreaming about profits when I received my first charting package. However, I soon realized that there was no Holy Grail of trading. Things that appeared profitable on the screen and in test, didn’t work so well in real markets. The good news is, I eventually discovered that as long as you avoided the common pitfalls of system design, market “edges” could be found.

Below we will look at some of the more common pitfalls I have encountered in developing systems over the years. Although the article is slanted towards the pitfalls associated with researching a “mechanical” edge, these concepts can (and should) also be applied to more discretionary methods.

Know Thy Data

Make sure you have the cleanest and best data available. If you are testing futures contracts, be aware of the anomalies when contracts are strung together. This “back adjusting” can often cause a skew to the market. Also, make sure that the system also works on actual contracts and does not just work on the back-adjusted continuous contracts. For instance, I have discovered that this adjustment tends to throw off volatility readings — generating false signals.

Know Thy Market

Many markets have a longer-term bias. For instance, the S&P futures tend to have a longer-term upside bias. Therefore, results on the long side in this contract tend to be skewed. A long-only system here might only continue to work if the longer-term bull market remains intact.

Know Thy Market Efficiencies

Thanks to the widespread availability of computers, many methods simply no longer work. For example, In Cybernetic Trading Strategies, Murray Ruggiero, Jr. showed that currency systems based on simple moving average crossovers worked years ago, but hasn’t worked in more recent times.

Know Thy Current Market

Futures Truth once attributed a system’s success to dancing to your favorite song. When the system is in sync to the market, it will tend to work really well. However, when the music changes, it will more than likely under-perform. For example, a reversal system will work much better than a trend-following system in a choppy market. However, it will more-than-likely under-perform once the market begins to trend again.

Know Thy Concept

Make sure the concept you are testing is conceptually correct. I’m sure if you looked long enough, you could find relationships between many non-correlated events. For instance, I once heard of someone who observed that the price of soybeans fluctuated based on where his cows were grazing. By the way, this is not why I have cows grazing on my land.

Know Thy Self

You have to look deep within yourself when analyzing a system. If it’s something that trades very frequently, are you willing to take each and every trade? On the other hand, if it's something that trades infrequently, are you willing to sit around while waiting for signals? If it has high losses on individual trades before recovering, can you stomach those losses? By the same token, if it has very large drawdowns or long periods of no profitability (i.e., flat time), are you willing to sit through those periods?

One last point, if the method is more discretionary, make sure you have a good idea of how you will implement it.

Know Thy Observations

Make sure you have enough observations to make a valid hypothesis. Said another way, systems that trade more frequently and in a variety of conditions are likely more legitimate than those systems that trade infrequently and/or in rare conditions.

Know Thy Indicators

I once was working with a client who thought he had the Holy Grail. He would predict the next day’s closing prices, up or down, with uncanny accuracy. While helping him test his method, I noticed that the indicator used displayed a value that incorporated the next day’s closing price. So, obviously, the indicator would point up when the next day's close was higher and would point lower when the next day’s close was lower.

Know Thy Software

Make sure the software does what you intend it to do. For instance, once, while helping someone test a previously believed-to-be-profitable system, I found over 400 missed signals. After these signals were added in, the system was no longer profitable. One way to help prevent such errors is to “hand test” the computer-generated results.

Know Thy Eye

Just as I mentioned “hand testing” above to make sure the software is correct, it’s probably not a bad idea to also computer test methods. Remember, the computer will impartially find each and every signal (or pattern), whereas you may, by being optimistic, tend to focus on the winners and not notice the would-be losing trades.

Know Thy Money Management

Make sure the system, on average, makes much more than it loses. If you risk $3 per trade, it’s highly likely that you could easily develop a highly accurate system that would make $1 per trade. However, you would have to be right at least three times as much as you are wrong. Mark Boucher refers to systems that risk much more than they make on average, as “anthill” strategies. Ants can build a significant mound, bit by little bit, but all it takes is one footprint to knock it down.

Know Thy Outliers

Make sure your pattern or system is not based on statistical outliers — one or a few large winners which make the system profitable overall. Ask yourself, would the system still be profitable if any one trade or small group of trades were removed? For example, seasonally based systems in the energy markets may work, but won’t work nearly as well when you take out the energy spike associated with the Gulf War. Therefore, carefully analyze any system whose profits are largely based on any one (or several) move(s).

Know Thy Perfect Hindsight

When analyzing systems, realize that the system is based on 100% hindsight. Make sure that fundamentally, nothing has changed in today’s markets. For example, Mark Boucher (in a lesson) pointed out that trading off of short interest no longer worked after the introduction of put options. Also, remember there is no way a system can factor in unforeseen events. I once read where statistically, market outliers should only occur once every 100 years. However, they tend to occur on a much more frequent basis in the real world. The crash of 1987, the blow up of Long Term Capital Management, the Asian Crisis and the World Trade Center destruction are a few of the many events over the last 10-15 years that have caused large market moves.

Summary

Unfortunately, there is no Holy Grail when it comes to the markets. That’s the bad news. The good news is, by taking a realistic approach, studying your data, studying the markets and looking within yourself, profitable methods can be discovered.