Checking in on bowl predictions: SP+ & FEI

Submitted by Blue@LSU on December 18th, 2023 at 10:14 PM

Well, I’ve finished all my grading and will start my drive north on Thursday. So with a few extra days on my hands, and now that we’re 7 games into the bowl season, I thought it might be a good time to check in to see how well our favorite advanced stats have fared with their bowl game projections.

Although I have fun looking at these data (and maybe other people here will enjoy it too), I must confess that my primary motivation is that I’m hoping for some insights (even if just an inkling) into how well the SP+ and FEI predictions will hold up for the game against Alabama. You see, the advanced stats have Michigan as 8.4 point (SP+) and 10.5 point (FEI) favorites against ‘Bama and I’m wishing like hell that they are correct.

Before looking at the results to date, it’s important to keep two major caveats in mind. First, there have only been 7 games so far. Second, these predictions are based on each team’s performance throughout the season. However, the models do not know, and cannot account for, injuries or players that have opted out, say, to prepare for the NFL or because they’ve entered the transfer portal. In other words, the teams that are the basis for these models are, in some cases, not the same teams that will be playing the games.

If anyone is interested in the data for the SP+ or FEI predictions, they can be found here and here.

THE RESULTS

The graph shows the predicted winner and predicted margin of victory (blue dot) for SP+ and FEI, along with the actual margin of victory (red dot) for the predicted winner. Negative values for the actual margin of victory indicate incorrect predictions, while positive values indicate correct predictions. The distance between the predicted and actual margin of victory represents the projection error. 

So for example SP+ predicted ULL to beat Jackson State by 1 point. The blue dot represents the predicted margin of victory for ULL (+1). The red dot indicates the actual margin of victory for ULL (-3). The distance between the two dots (4 points) represents the prediction error. 

Just glancing at the graph, it doesn’t appear that either SP+ or FEI are particularly close in terms of predicting the actual margins of victory. If we just look at correctly predicting winners, FEI has a clear advantage with a 6-1 record compared to a 4-3 record for SP+. 

Personally, I’m a fan of graphs but I know that some people prefer to look at tables. And since I aim to please…

The table gives pretty much the same information as the graphs. But I also provide the average prediction error for SP+ and FEI. In general, there’s a pretty modest difference in the average prediction errors for both sets of advanced stats.  

So what can we say so far? 

  • Well, the vast majority, 12/14, of the predictions so far have been coin flips (less than 3 points). And while FEI has been more accurate in predicting the winner, their prediction error has only been marginally smaller than SP+. I mean, sure, they accurately predicted Fresno State to beat NMSU. But they predicted a 1-point victory in a game that turned out to be a 27-point victory. Can we really say that’s more accurate than SP+’s prediction that NMSU would win (or Fresno State would lose) by .6 points? 
  • I’ve got a problem with micronumerosity, and my doctor told me the only solution is more football. Luckily we still have 34 bowl games left to play. 

I'll try to keep these updated with one more post on the regular board before January 1. If anyone is interested to follow along at home, here’s the schedule and predictions for the games ahead.

 

 

 

P.S. I know I said in my last diary that I would take a break from making data posts. I guess I just can't help myself. 😊

Go Blue!

Comments

HighBeta

December 18th, 2023 at 10:35 PM ^

First? Drive safe. Morons, drunks, and crazies drive too, frequently. Avoid them at all costs. So, safe travels!

Second? Yes, we need higher N to better gauge the predictors' prediction successes.

Third? Yes, these are not the same players whose performance was used to build their models.

Fourth? It's fascinating to compare predictive modeling engines.

Fifth? Actual margins seem, subjectively, less important than the W/L result. This is a philosophical debate that stats heads can mink hole at great length.

Last? Great work. Again. Thank you!

Blue@LSU

December 18th, 2023 at 11:15 PM ^

Thanks!

I'm one of those strange people that loves working with data but is also always skeptical of data. One of the reasons I like looking at the margins is that it (in a way) gives us a measure of uncertainty, which again goes back to my skepticism (I'm a fan of Nassim Taleb). But I do think I probably take it too far sometimes. 😊

Happy Holidays, HighBeta!

HighBeta

December 19th, 2023 at 1:32 AM ^

Strange? It's not strange to be visually presenting the unpredictability of human responses (coach play calling) to a complex series of events (current score, time remaining, etc.).

My contention is that the more you work with data, the greater your skepticism grows of patterns (other than a certain randomness) in same. 👍🙂

Happy Holidays to you too!

 

Yeoman

December 18th, 2023 at 11:06 PM ^

Standings so far in a confidence pool based on the four major ranking services and last week's line at BetMGM:

  1. Sagarin 7-0  99-0  1.000
  2. FEI        6-1  82-3    .965
  3. SP+      6-1  67-4    .944
  4. Massey 6-1  109-16 .872
  5. MGM     4-3  69-42   .622

FEI missed on Cal, SP+ missed on Miami, Massey missed on Louisiana. The moneyline misses at MGM were Georgia Southern -3.5, New Mexico State -3.5, Old Dominion -2.5.

Funny that all the misses so far were one-offs; if you picked the consensus favorite you're perfect. Or you could have just gone with Sagarin.

Yeoman

December 18th, 2023 at 11:34 PM ^

I'll try to get a diary up on this later, but the short answer is it's like the ESPN Bowl Mania contest (the confidence version; they do three different contests now). You pick a winner for each bowl, and you assign points from 1-41 based on your confidence in the result.

So SP+ had Miami at 0.7 worse than AppSt. That was the fourth smallest spread so that game was worth four points.WKU was an 8.7 point favorite at SP+; that was the 11th biggest spread so that game was worth 31. Etc. The first column is the straight-up w/l record so far, the second column is the confidence points won and lost so far, the third is a sort of weighted winning percentage.

Hope that makes sense for now. Wish I could have gotten a diary posted before bowl season started but it didn't happen.

Yeoman

December 18th, 2023 at 11:26 PM ^

If anyone's curious how this looks for the next few games:

  • UTSA is a unanimous favorite, but not by as much as the MGM line of -13.
  • Syracuse is a unanimous favorite by more than the MGM line of -3.
  • Everybody agrees on UCF.
  • MGM had Arkansas State -1, everybody else has NIU.

Mostly what's happened so far is that the lines have been adjusted for missing players, and then the teams played at the level they did all year even without the missing players. Not sure that works when we get to OSU etc.

Blue@LSU

December 18th, 2023 at 11:38 PM ^

and then the teams played at the level they did all year even without the missing players.

This is fascinating. Intuitively I would've guessed that a lower-ranked team would be at a greater disadvantage if they lost an important player (at least compared to a team that can plug in another 4* or higher).

It's going to be interesting to see what kind of team OSU fields against Mizzou. There's no way I would bet on that game.  

Yeoman

December 19th, 2023 at 10:21 AM ^

It's a small sample size so far. But Ohio U. lost their quarterback to the transfer portal and dominated the LOS anyway. (I'm not sure if the QB that played was their third, or if he started the year #2 and got hurt.) WKU's QB might be a late-round draft pick; he opted out and his backup (who was already leaving for the portal but wanted to play the game anyway) threw five TDs.

Koop

December 19th, 2023 at 4:14 AM ^

For those interested, Sagarin's model predicts a Michigan win over Alabama by (roughly) 3 points when treating the Rose Bowl as a neutral site. 

(Sagarin's site appears to show more than one model, but each shows a Michigan victory by between 3 and 4 points.)

Can't say I understand Sagarin's methodology--just reporting what his site shows.

And--not complaining.

Blue@LSU

December 19th, 2023 at 12:41 PM ^

Yeah, I was having a hard time figuring out how to show the prediction vs. the result. I settled on showing the actual margin of victory for the team that was predicted to win. The absolute values for the red dots are the same, but they are negative if that team lost (a negative margin of victory) and positive if they won. 

Hemlock Philosopher

December 19th, 2023 at 10:47 AM ^

Love these. I don't think they take into account the portals and sits though, which are huge in bowl games... also the weather in that MOH/App game was a huge factor... I keep an eye on these picks as well as the portal news to make picks ATS. So far: decent at 5-2. 

plamonge

December 19th, 2023 at 6:42 PM ^

I love this. One thought that came to mind: When the predicted score differential is larger, does the predictive power increase. You'd think so, but who knows. I wondered this because out FEI score difference is 10.5 point vs. most on the first table, which as you said are near 50/50. 

Blue@LSU

December 19th, 2023 at 11:09 PM ^

That's a good question.

Intuitively, I do think it will start to be more informative when the predicted margins start to get larger. In those cases, the models are clearly saying "team x is better than team y" which makes the simple correct/incorrect prediction more meaningful. If the models incorrectly predicting winners in games where they had a team wining by 8+ points, then it might be a sign of a problem (again, taking into account opt-outs and other unmeasured factors).