How are our predictions faring so far?
The Trial Run has now been going on for slightly over five weeks which allows us to begin making initial conclusions about the quality of our predictions. The period of time is still too short to make overall observations with any kind of statistical significance, but I think taking an early peak will help in at least lining out the direction in which we are heading, so let’s dive in.
For this exercise, I have chosen to measure the accuracy of our 15-day indicator. This indicator shows where our current average predicts the market to close 15 calendar days after today and you can see it at the top of every chart alongside 1-day and 30-day indicators.
So, the question we are asking if how well has this indicator predicted the market over the past several weeks (well, over as many as we have the data for)?
Let us take our most popular chart, S&P500, and see how we did.
The grey line on the chart above is where the market itself has closed over the roughly three weeks that we have the data for. Remember, although the trial run is alive for over five weeks, our first 15-day prediction was given on the first day of the trial run and so the results can only be verified from day 15 onward.
The blue line is what the 15-day indicator showed our prediction to be 15 days before. In other words the value you see for February 15th is what our 15-day prediction was on January 31st and so on. The orange area below is the difference between the two, the prediction and the outcome. You can think of it as our ‘prediction error’.
I would say that we are tracking the market relatively well. On most days, our error is around 1%. Not bad, particularly given that February was more volatile than the previous few months and also one during which the market rally has stalled somewhat.
Now again, this is too short of a time span for a statistically significant analysis, and our weekly vote count went up nearly 7x over this period, which is unlikely to be a constant state of events, but so far the results are encouraging. Note that these are also completely unfiltered results. Every vote cast has been used. Over time, I want to see if we can change how our average is computed by making it smarter. Perhaps, we will give more consistent users a higher weight. Perhaps, we won’t account for votes of those who are voting for the first time. Neither would impact how the pay offs are calculated, but our averages might provide better results.
As of this writing, our 15-day indicator is predicting the S&P500 to be flat vs. last Friday’s close in two weeks’ time. Let’s see if that happens.
Here is how we’ve done with the Bitcoin:
Our Bitcoin predictions were just a little more range bound than the underlying market (i.e. the Bitcoin itself). Most of the time, the error stayed within 5% and would have avoided the largest spike had the results been filtered. Still, overall, I would again say that we have been tracking the market relatively well given the magnitude of its swings.
No comments:
Post a Comment