November 8th, 2018 at 5:13 PM ^

This is really interesting. Thanks for putting it together. It seems the ACC is the most overrated conference with the CFP overweighting and underweighting the SEC. The B1G just gets screwed.  


November 8th, 2018 at 6:28 PM ^

Yes, the ACC is egregiously overrated.

I keep saying it, but the SEC isn't that bad.   Miss St and Auburn are underrated.  And it's understandable that Kentucky and LSU are overrated.  They've had a lot of really good wins, they were just close so they're like ND in that they have good resumes, but not great predictive analytics.


November 8th, 2018 at 9:33 PM ^

So, this rating comparison isn't really fair to the committee, since wins and losses are obviously the most important input variable to the committee, but in some of the computer models are not even considered.  So the analysis is flawed.  Matching the CFP rankings with ESPN's strength of record measurement would be a more sound method to analyze bias in the rankings.


November 9th, 2018 at 9:44 AM ^

There is no doubt that the committee set up the playoff berths last week to encompass a soft landing for Alabama heading into their showdown in Baton Rouge last Saturday, before they dismantled a 4th-ranked LSU which featured the nation's 93rd ranked team in total offense.

So, the shutout of the Tiger offense and complete inability to move the ball seems completely understandable and mostly disguised by the fact that these are supposedly the top teams besides Georgia in the holier than thou SEC. There is no question Alabama with the nation's most productive offense and 11th best defense is a legitimate death star.

But LSU was completely overrated at No. 4 and neatly removed from the Alabama safety net spot left available for either a fail safe loss by ND or the death star. I think the committee has propped up the other SEC and ACC teams to artificially support their Final Four, regardless of how they actually perform on both sides of the ball without respect to record. In short, they are playing historical favorites to back their top choices.

LSU was just complete  committee cannon fodder when Michigan had already earned the 4th spot even before destroying Penn State. 

And the reason that getting the higher seed is important starting out, is that it makes both the competition to get a higher seed and occupy a fail safe position as the committee has already assured both Alabama and Clemson they are in regardless of what happens going forward. Clemson has BC left. Alabama has Auburn and Georgia.

The committee is looking at matchups for its three big games, which explains why ultimately they are propping up the SEC and ACC because historically those conferences have produced their national championship entrants since the new format began.  History is the underlying point in  their choices.

You have to understand their motivation is TV ratings as its relates to assigning the credentials of the best four teams and not saddling the nation with another Alabama woodshed beating of ND in the biggest game of the year.


November 9th, 2018 at 2:41 PM ^

Based on this analysis of methods that use margin of victory, which is the best indicator of team quality, they are underrated by 6 spots.

You are citing a cherry-picked sample of their wins and losses results with no reference to margin or how those games were won or lost, and you're leaving out the fact they beat a good Washington team at a neutral-ish site.


November 8th, 2018 at 5:18 PM ^

I'd like to point out that while CFP says they are looking for the best teams, they are actually looking at resumes. The statistical models used are all predictors, i.e. they ARE looking for the best teams, not resumes. Bill Connelly created Resume S&P+ and ESPN split FPI into FPI (predictor) and SOR (resume) because S&P+ and FPI shouldn't be compared to the rankings.

Now, they still don't line up perfectly, but Resume S&P+ has Syracuse at 29 and SOR has them at 27, for an average of +15 instead of +34. Kentucky is 22 and 7 for an average of +3.5. NC State is 37 and 15 for an average of +12. Boston College is 28 and 18, for an average of +6. I think that pretty well illustrates the point that this is comparing apples to oranges.

I know that Resume S&P+ is using the scores of the games, but I don't know if SOR is or if they are capping them differently. I don't believe the committee is using MOV, so that could be the remaining difference here.


November 8th, 2018 at 5:47 PM ^

After Joel Klatt pointed this out, I started looking into it and got super pissed off about it. I don't like how Resume S&P+ takes points into account (caps at 50, would prefer lower cap or a scale). I also don't trust ESPN because I feel they are likely biased and there isn't much insight into how they do things. So, I created my own ranking system based on S&P+ rankings and wins/losses without MOV, that I'm pretty happy with, to try to prove the bias.

  1. Alabama
  2. Notre Dame
  3. Michigan
  4. Clemson
  5. LSU
  6. Georgia
  7. Central Florida
  8. Oklahoma
  9. Ohio State
  10. Kentucky
  11. Washington State
  12. West Virginia
  13. Cincinnati
  14. Utah State
  15. Army
  16. Boston College
  17. NC State
  18. Fresno State
  19. Penn State
  20. Boise State
  21. Texas
  22. Iowa State
  23. Syracuse
  24. Florida
  25. Buffalo

The ACC all seem to be within 3 spots of the CFP by this method. Florida is still +9 and UCF is -5, but I can chalk that up as outliers.

So, to answer your question, I tried to prove that to be the case, but ended up showing that the CFP is ranking resumes pretty well if they aren't looking at MOV. Any debate about their accuracy should be based around using MOV in some way as that will move the list closer to best teams instead of best resumes.


November 8th, 2018 at 7:08 PM ^

What's clear about your method is you're heavily rewarding raw W/L record.  That's how you're getting so many non-P5 teams in your ranking: Utah State, Army, Boise, Buffalo.

And that's fine if you want to reward simply winning games regardless of who those wins are against.  But that's the only justification for including the ACC teams so highly.  Their overall bodies of work are no more impressive than Army, Utah State, Boise State and Buffalo but since they're in a P5 conference, they're being ranked despite not having beaten anyone of real quality while all those G5 teams aren't being ranked.

So you can't justify the ACC ranks based on your method when the committee isn't treating the G5 the same way.  They're definitively biased towards the ACC or against the G5 or both.  The ACC teams are getting credit for "P5" wins over bad teams in their own conference and bad non-conference teams that make up 1/3 of their schedules.

And this really drills into the biggest problem with the CFP and its oversimplification of resume analysis: they put too much weight on "wins over P5 teams with winning records", regardless of how those winning records were achieved.  It's a totally arbitrary distinction, kind of like the basketball committee treating top 50 or top 100 wins as a magic number.

The huge problem with elevating wins over P5 teams with winning records is that the ACC and SEC only play 8 conference games.  Which means they can schedule 4 tomato cans and end up with more teams with winning records in their conferences and hence more wins against teams with winning records.

Now, the committee only has to pick the top 4 teams so there's a lot less margin for error than if they were seeding a large tournament like the basketball committee does.

I don't think it's caused them to make a mistake in the top 4 yet.  Both of the biggest controversies involved a one-loss non-divisional champ making it over a two loss conference champ (OSU over PSU in 2016 and Bama over OSU in 2017) and in both cases they did get the "better" team in the tournament.  I think they should have valued the conference titles more to make those achievements more meaningful, but under the criteria of "best" teams, they probably did get those selections correct.

What will be interesting is with the ACC being so overrated, would a one-loss ND team that hasn't looked all that good still have a chance? Luckily, probably not since they only get Syracuse and not BC or NC State.

As an aside: Resume S&P+ has some very weird results that I pointed out on the post yesterday about ND.  IMO, it's a bad resume measure because it uses margin and hence is too much like a "quality" measure and even at that it has some weird results.

As an example, ND has the best SoS out of any of the undefeated teams per Connelly.  And that's reflected in them being #1 in ESPNs Strength of Record metric. So they have the best pure "resume" in terms of Ws and Ls and who those came against when you ignore how they won those games.

They're certainly not top 4 when you consider score margin.  The predictive metrics have them ranging from 6th (S&P+) and 13th (Sagarin's predictor) out of the most commonly cited metrics (FEI, FPI, etc).

Somehow Connelly's Resume S&P+ has ND 18th (!!) which is far worse than his "predictive" measure where they're 6th.  That makes absolutely no sense at all.


November 9th, 2018 at 10:37 AM ^

I think we are on the same page at a high level on all of this. A couple of things, though.

I don't take W/L record into account at all. A team losing to the top 4 teams in S&P+ and beating 5-9, giving them a 5-4 record, would have a ranking of 3rd. Similarly, a team beating a completely average team, like Northwestern, 9 times would be 6th. The way that I look at it is that we might know that a team is good, but building a resume is akin to showing your work in math in the sense that you shouldn't get full credit if you're not playing anyone. I don't think that Bama should be getting any more credit for UCF's schedule than what UCF is getting. If you're a better team, play better opponents and show it. I weight so that two top 5 teams playing has a lot of upside and little downside, e.g. if Michigan had beat ND, Michigan would be the #1 team in the country ahead of Bama right now and at the end of the season and ND would be 9th right now with a chance to climb a few spots depending on losses.

As for the ACC teams, they may not have played anyone, but neither have a lot of teams. Who do you move up ahead of Boston College, for example? MSU? I don't even have them ranked. Sure, they beat PSU, but they lost to ASU and Northwestern. Penn State? Maybe, but it would be on the backs of their wins against Appalachian State and Iowa. Texas beat Oklahoma, but lost to Maryland. Your best options are Fresno State and Buffalo, which you already said would be weighting W/L too heavily.

And I agree with you about Resume S&P+. I think that he created an alternate S&P+ that doesn't really seem like a good predictor or resume evaluation, at least compared to what the committee is trying to do. On the Resume-Predictor spectrum, it seems to be just a hair inside Predictor, which really doesn't give it much use.


November 9th, 2018 at 2:52 PM ^

But you do take win loss record into account by the way you're effectively rewarding wins and punishing losses.

And your example proves the point.  If a team loses to teams 1-4 and beats teams 5-9, the ideal place to rank them would be 5th, behind the teams they lost to and ahead of the teams they beat (which I guess would bump them down to 6-10 but I'm unclear if your ranks are before ranking the team in question?).

But you're saying you'd rank them 3rd.  Does that mean you're bumping the previously ranked 3rd and 4th teams, whom they lose to, down to 4th and 5th respectively?

Your results are actually pretty close to Colley Matrix and other ELO methods that are purely W/L based and not at all margin based (Colley doesn't take H/A into account).  So it's a fine pure resume ranking but a pretty poor quality ranking.

I think we are in agreement that the committee's actual behavior as it relates to picking the top 4 is more about resume than quality.  And so you're justifying the ACC's inclusion in the top 25 along the same lines.  And I would agree that their inclusion of those bad ACC teams would be fair IF THEY WERE CONSISTENT with their evaluation of resumes.  But they aren't giving G5 teams like Army, Buffalo, Utah State the same treatment as those weak ACC teams with similarly good W/L records.  They've all racked up wins against mediocre competition but since the G5 teams are in the wrong conferences, they're not in the top 25.


November 8th, 2018 at 8:46 PM ^

Good post --- IMO, you are doing things right IMO from a methodology POV.  "Strength of resume" should incorporate the advanced analytics somehow (e.g., exactly how good is that team you beat or lost too), while also (2) grading only on a "binary won or loss basis", while not considering MOV.

My own ratings that try do the same thing you are trying - basically the same results.  Exact same top #12, just a handful of teams flip-flopped by 1 spot.


November 9th, 2018 at 11:03 AM ^

Thanks. I tinkered around a lot to get to this. I wanted to come up with a simple algorithm that is still mathematically sound. I had certain goals and this is the only way that really fit all of the goals. While I don't know that it's necessarily the best way to rank teams for the playoff, this becomes a pretty close proxy for what the committee is trying to do and lets me spot check for bias.

As for the mathematically sound part, I found some cool features of this method. For example, the top 24 teams currently have a positive score. Bill C complains a lot about 25 being an arbitrary cutoff and it changing every week. He's right about the latter, which is something that I tried to solve with this method, but top 20-25 seems like a natural cutoff. If 130 teams were ranked on a bell curve, 20 would be at least 1 standard deviation above average. I've also noticed that the top 14 have a score of at least 1 and the top 5 have a score of at least 4. Bama is the only team that has a chance to reach a score of 9 by the end of the season. So, there are these natural tiers built into it.

I started working on it a week or two ago, after Joel Klatt started spouting off about it. I think I'll wait to see how this formula progresses through the season, but I might sub in some other rankings like FPI, Sagarin (especially his recency one), or a composite. In the offseason, I think I might try to build my own predictor, not with beating the spread in mind, but as my own ranking system based on game theory rather than arbitrary cutoffs, e.g. determining garbage based on the game itself rather than an arbitrary cutoff like 21 points.


November 8th, 2018 at 5:38 PM ^

Except the CFP rankings and S&P are ranking very different things. One is predictive of future performance, the other is measuring accomplishments to this point. So this doesn't really prove anything other than teams like BC and Syracuse have outperformed what statistics would suggest they should have done so far this season.


November 8th, 2018 at 6:43 PM ^

Thanks for posting this!


I also created my own stats by taking out the stuff I didn't like, taking the good stuff, adding 97 and multiplying by 4:

1. Michigan

2. Michigan

3. Michigan

4. Michigan

5. Michigan

6. Michigan

7. Michigan

8. Michigan

9. Michigan

10. Michigan


I think Klatt and Communist Football might be better at this than I am.

Arb lover

November 8th, 2018 at 8:03 PM ^

Yeah but the committee will just look at this list and say "see, Michigan is just about exactly where we have them placed at one spot off... what's the problem?"


November 8th, 2018 at 9:20 PM ^

These metrics affirm my belief that all that matters is winning the Big Ten. Aside from the OSU homers who officiate, there is something regal with settling it on the field. Beat everyone on your side and the best from the other. No political bias or corporate influence. Control your own fate.


November 8th, 2018 at 9:29 PM ^

I still haven’t let the reality set in yet. Two years ago I expected us to go to the playoffs. Then Iowa happened. Yet, I went to Columbus expecting to win vs a team that sadly was better. Even then, I held out hope Washington and Clemson would lose in their CCG and we’d get in.

At this point two years later, the cards are stacked in our favor greater than they were two years ago. I still don’t want to believe it yet. Go into Columbus and win and we are in. This team doesn’t have Barrett, Samuel, Bosa, Hooker, Conley, Ward, McMillan, Lattimore, etc. On top of that we have a better team with a better line, QB, and RB.

November 24th needs to come, ASAP


November 8th, 2018 at 10:47 PM ^

Any of you watching this barn burner game between 14th ranked NC State about to get torched by Wake Forest? If WF can make a last minute drive, they’ll win. Sheesh. Purdue or Indiana would absolutely mop both of these ACC teams. 

UPDATE: the beastly demon deacons, with a record of 4-5 have beaten the overhyped Wolfpack. I’ll bet anyone on this blog $1 that NC State somehow stays in the top 25 or that Wake Forest might crack the top 4.