“We all want to be rich by having a computer just generate piles of money for us,” writes long-time Slashdot reader TekBoy. “Here’s one man’s attempt at using AI to predict the market.
From the article (by tinkerer/writer/network guy Jason Bowling):
Models that did great during their initial training and validation runs might do ok during runs on later data, but could also fail spectacularly and burn all the seed money. Half the time the simulation would make money, and half of the time it would go broke. Sometimes it would be just a few percentage points better than a coin toss, and other times it would be far worse. What had happened? It had looked so promising. It finally dawned on me what I had done.
The results cycling around 50% was exactly what you’d expect if the stock price was a random walk. By letting my program hunt through hundreds of stocks to find ones it did well on, it did stumble across some stocks that it happened to predict well for the validation time frame. However, just a few weeks or months later, during a different slice of the random walk, it failed. There was no subtle underlying pattern. The model had simply gotten lucky a few times by sheer chance, and I had cherry picked those instances. It was not repeatable.
Thus, it was driven home — machine learning is not magic. It can’t predict a random sequence, and you have to be very careful of your own biases when training models. Careful validation is critical.
I am sure I will not be the last to fall victim to the call of the old treasure map in the attic, but exercise caution. There are far less random time series to play with if you are looking to learn. Simulate, validate carefully, and be aware of your own biases.
of this story at Slashdot.
Source:: Slashdot