Tracking KenPom
This post will examine the accuracy of the KenPom rankings and
predictions, and try to evaluate the performance of Michigan's
basketball season in comparison.
I've been following along with Brian/Tim's basketball previews and was wondering how accurate the KenPom predictions have been. I'll graph the predictions versus the outcomes, and try to adjust the predictions based on the current rankings (versus rankings at the time). I will also include a "baseline" program for analysis and comparison to our manic/depressive performance this season.
Numbers:
Michigan is currently ranked 85/47offensively/defensively according to KenPom. Compare that to the competition:
And prediction/results for those games:
Simple numerical average of (kenpom - actual) gives -0.85, which shows pretty good prediction value.
Showing the results graphically:
The orange line shows how close the kenpom prediction was at the time.
Now, we will look at the current rankings to try to get a better feel for the prediction value. Assuming that a better team will beat a worse team, we will estimate margin of victory based on relative ranking.
The last column is expected margin of victory, if the teams played today. Graphing the RDP versus actual gives this:
The games with big gaps would be upsets, but overall the prediction percentage is .61, that is, the percentage of games that the current rankings would predict correctly, win or lose.
Now let's compare that chart to a control, Michigan State. MSU's rank is 28/31. The data in question:
and chart:
Now the prediction rate is .92 (12/13).
So what does all this show? I think it shows the value of KenPom's system when used on a good team. Or, conversely, the inconsistency of Michigan this season - beating teams they shouldn't beat, losing to teams they should beat. I'm not a gambler, so I didn't take into account the value of covering against the spread, I'm simply looking at this as a fan and judging based on wins/losses. As far as wins and losses, this system seems very accurate. I may look into tweaking the ranking calculation to better match the results, but I think the basic idea is pretty solid.
I've been following along with Brian/Tim's basketball previews and was wondering how accurate the KenPom predictions have been. I'll graph the predictions versus the outcomes, and try to adjust the predictions based on the current rankings (versus rankings at the time). I will also include a "baseline" program for analysis and comparison to our manic/depressive performance this season.
Numbers:
Michigan is currently ranked 85/47offensively/defensively according to KenPom. Compare that to the competition:
| team | current offense rank | current defense rank |
| minnesota | 38 | 53 |
| osu | 10 | 19 |
| ill | 67 | 40 |
| psu | 73 | 113 |
| iowa | 141 | 162 |
| minnesota | 38 | 53 |
| wisc | 11 | 15 |
| nw | 27 | 163 |
| iowa | 141 | 162 |
| msu | 28 | 31 |
| purdue | 33 | 5 |
| wisc | 11 | 15 |
| uconn | 66 | 28 |
And prediction/results for those games:
| team | kenpom prediction | actual difference | kenpom - actual |
| minnesota | 2 | -16 | 18 |
| osu | 12 | 11 | 1 |
| ill | 1 | 6 | -5 |
| psu | -8 | 4 | -12 |
| iowa | -3 | -2 | -1 |
| minnesota | 9 | -7 | 16 |
| wisc | 5 | 8 | -3 |
| nw | 3 | 13 | -10 |
| iowa | -11 | -14 | 3 |
| msu | 3 | 1 | 2 |
| purdue | 11 | 10 | 1 |
| wisc | -9 | 16 | -25 |
| uconn | -1 | -5 | 4 |
Simple numerical average of (kenpom - actual) gives -0.85, which shows pretty good prediction value.
Showing the results graphically:
The orange line shows how close the kenpom prediction was at the time.
Now, we will look at the current rankings to try to get a better feel for the prediction value. Assuming that a better team will beat a worse team, we will estimate margin of victory based on relative ranking.
| team | rank average | michigan rank - team rank | ranking difference prediction |
| minnesota | 45.5 | 20.5 | 2.05 |
| osu | 14.5 | 51.5 | 5.15 |
| ill | 53.5 | 12.5 | 1.25 |
| psu | 93 | -27 | -2.7 |
| iowa | 151.5 | -85.5 | -8.55 |
| minnesota | 45.5 | 20.5 | 2.05 |
| wisc | 13 | 53 | 5.3 |
| nw | 95 | -29 | -2.9 |
| iowa | 151.5 | -85.5 | -8.55 |
| msu | 29.5 | 36.5 | 3.65 |
| purdue | 19 | 47 | 4.7 |
| wisc | 13 | 53 | 5.3 |
| uconn | 47 | 19 | 1.9 |
The last column is expected margin of victory, if the teams played today. Graphing the RDP versus actual gives this:
The games with big gaps would be upsets, but overall the prediction percentage is .61, that is, the percentage of games that the current rankings would predict correctly, win or lose.
Now let's compare that chart to a control, Michigan State. MSU's rank is 28/31. The data in question:
| team | actual difference | ranking difference prediction |
| osu | 7 | 1.5 |
| ind | -14 | -14.65 |
| psu | -12 | -6.35 |
| purdue | 8 | 1.05 |
| ill | 5 | -2.4 |
| wisc | 14 | 1.65 |
| nw | -9 | -6.55 |
| mich | -1 | -3.65 |
| minn | -2 | -1.6 |
| iowa | -7 | -12.2 |
| ill | -10 | -2.4 |
| minn | -13 | -1.6 |
| iowa | -18 | -12.2 |
and chart:
So what does all this show? I think it shows the value of KenPom's system when used on a good team. Or, conversely, the inconsistency of Michigan this season - beating teams they shouldn't beat, losing to teams they should beat. I'm not a gambler, so I didn't take into account the value of covering against the spread, I'm simply looking at this as a fan and judging based on wins/losses. As far as wins and losses, this system seems very accurate. I may look into tweaking the ranking calculation to better match the results, but I think the basic idea is pretty solid.
Small nitpick: why is iowa listed twice in the charts?
EDIT: nice analysis, though. I always wondered how statistically significant KenPom predictions were. Thanks!
we played them twice.
Couple of points.
1, I think in the first graph, you can't get to your average by using positive and negative numbers. A basic principle of signal averaging is that random noise averages out to zero. That's more of what I see in those kenpom - actual numbers for Michigan. If you just did the absolute value of the kenpom - actual you would get how far "off" he was for each game, regardless of whether it was an upset or not. That would put his average prediction for Michigan at 7.76. Meaning that his predictions were, on average, 7.76 points off. I don't think that's very good. In fact, I'd be willing to bet that most people who follow basketball with a decent amount of effort could do as well or better.
2, I don't think you can look back at games a month or two ago using current rankings. There is so much fluidity to the game, I just don't think that will end up being representative of much.
I agree that the average difference is kind of a useless number. I think the right/wrong percentage is the data of value from this exercise, and it's easier to see in a bar graph than a line graph. Now, I'm not sure I've proven anything other than "Good teams beat bad teams, most of the time."
I figured 2. would be a question. What I was trying to show was that the current positions would represent likely outcome, not taking into account outside interference, like OSU or Purdue losing good players. It was intended to reflect what we know now about the game, rather than what we knew then. If a team has fallen apart, then an earlier "upset" wouldn't be as big a deal, since now we know/expect them to lose.
By the way, for potential diary-makers, I did the whole thing in Google Docs and just pasted the result in here, worked like a charm.
Agreed, although the variance of the data is probably even more important to consider. In other words, is KenPom consistently bad or inconsistently less bad at predicting Michigan games? Looking at the data, and given Michigan's play this year, I imagine it's the latter.
that this is the type of debates we have. Statistical significance and other stuff that relates a class that handed me my lowest grade at UofM. I dig that about this crowd.
Hey Brian, how do we stack up against other blogs?
The accuracy of KenPom's predictions are something I've always privately thought were sort of poor, but I'm glad someone actually took the initiative (boo me) to put some analysis behind it.
As a previous poster noted, I think "averaging" the +'s and -'s of his performance gives a somewhat jumbled number as he could have been wrong in our opposition's favor by 25 points in one game, and wrong in our favor by 25 the next game, and, in that system, some up with a "perfect" prediction record. More useful, I think, would be to average the total of the variance of each game. IE, he was off by 6 points one game, 20 the next, etc.
Also, I think you may be giving the system a slight free pass when just saying "it predicts winners 92% of the time in the case of MSU", because oftentimes, predicting winners and losers isn't that difficult. For instance, if you started with the premise that "The Home team will win", you'd have something like a 70% success rate. If you got more complex and started choosing out road game where, say, MSU or Purdue were playing at Indiana, Iowa, or Penn State, your prediction rate would climb higher.
So, I guess we need a comparison against another prediction system, as you allude to. For instance, is KemPom more accurate than the spread Vegas puts out?
I agree. Variance is the real key in discerning Pomeroy's accuracy and comparing Pomeroy to Vegas lines would be interesting to say the least. Still, great post, thank you for doing this.
Comments