There has been a lot of talk about the CFP rankings recently with Michigan's chances for a playoff berth hinging on how the committee ranks teams. Joel Klatt has had several interviews where he talks about how ACC and SEC teams are being overrated to try to bolster the resumes of Alabama and Clemson to ensure that they get into the playoff regardless of whether the lose a game. He then goes on to compare their CFP rankings to FPI as a way of showing how overrated these teams are. This was a misstep on his part in my opinion as he is comparing apples to oranges. I've seen a lot of people follow his lead on here, reddit, and twitter. I've tried to head this off in the comments, but to no avail. Hopefully, something with a wider audience can help.
To start, let's take a look at ESPN's thoughts on FPI vs. the committee rankings:
To help determine the “most deserving” teams for the College Football Playoff, ESPN’s Stats & Information Group has developed some in-depth analytical tools to evaluate résumés. These metrics look “backward” at what a team has accomplished to date and are therefore categorically different from our Football Power Index. FPI is a forward-looking system that evaluates who is “most powerful” and helps predict specific matchups as well as the rest of the season. Some of ESPN’s own break down the difference between “best” (which we measure using FPI) and “most deserving” (where the résumé metrics come in) here.
It goes on to describe SOR (Strength of Record), which was created for the purpose of comparing resumes similar to what the committee is doing. They explicitly state that FPI and SOR are very different measures for looking at the "best" teams vs. "most deserving". So, to continue, there needs to be some discussion about what the goals of the committee is trying to do. ESPN has some more thoughts on this:
Despite the committee’s mantra of selecting the “four best teams in the country,” it appears that in the first two years of playoff selection, the committee favored team accomplishment over team strength. So if you are trying to predict what the committee will do, take a look at strength of record, because seven of eight teams to make the playoff ranked in the top four of that metric before playoff selection. Then FPI can be used to predict which teams will ultimately come out on top.
To put it plainly, the committee is looking at "most deserving" teams Klatt and a lot of others online are talking about "best". This is the misstep. We all know that it is possible for the better team to lose a game. Pretty much any predictor that I've seen has Michigan as a better team than Notre Dame, but Notre Dame won the head-to-head matchup, so they are more deserving as long as they keep winning out. Purdue put the beatdown on OSU, but I don't think that a 5-5 team is better than a 9-1 team, nor do I think that they are the more deserving.
So, what we're looking at is a spectrum of evaluation. On the left, you have the qualitative, backward-looking resume rankings judging "most deserving" by looking at your SOS without MOV and, on the right, you have the quantitative, forward-looking predictors judging "best" by looking at your stats and not even looking at the score, let alone winner or loser of the game. On the left, you have CFP and SOR. On the right you have S&P+, FEI, FPI, Sagarin, Massey, etc.
Bill Connelly tried to bridge the gap between the two sides of the spectrum when he created Resume S&P+. This method starts with S&P+ and adds in some MOV to try to generate a resume. On our spectrum above, where 0 is "most deserving" and 10 is "best", this is probably an 8. There are some major flaws in this methodology. For example, Notre Dame is a top 5 team by just about any resume ranking and a top 10 team by just about any predictor, but Resume S&P+ has them at 14. Penn State, with 3 losses, is listed at 8 despite not being in the top 10 on either end of the spectrum. In my opinion, this is probably due to a high weight on SOS and not enough on how you performed against that schedule. I think this makes it not a great tool for looking at resumes and a useless tool for predictions, which leaves it without a use in ranking teams. I think where it might have use is in its outliers as those teams are likely being over/under ranked due to their SOS.
The argument that SEC or ACC teams are ranked too high seem to be largely based on a mismatch of philosophies. To me, the question comes down to this: what is the purpose of the playoff? Is it to reward the 4 best/most deserving teams, to find the best/most deserving team? While it may seem backwards, if the goal is to find the best team, pitting the 4 best teams per S&P+ is not the best way to do so, unless the assumption is that the better team always wins. Hence, the reason for looking at "most deserving" teams. With the lack of data points that Bill Connelly refers to in the Resume S&P+ article, what makes sense, from an analytics perspective, is to try to take the most deserving teams (the teams that played the harder schedule and still won) and pit them against each other. This is what the CFP is set up to determine. The best team may not win, and we may still not know with statistical certainty which team is the best, but there will be no doubt that the team winning the national championship is the most deserving.
So, if you feel that the S&P+ is closer to how teams should be ranked than a standard resume ranking, that's fine. Just know that your argument should be that the criteria for selection is wrong as opposed to the CFP being a bad implementation of the criteria. However, if your argument is that teams are ranked too high/low based on the criteria specified for selection, use the proper tool to show that which is another resume ranking tool such as SOR.
For what it's worth, Syracuse (+9), Kentucky (-5), Boston College (+6), and Miss St (+10) are the biggest differences between CFP and SOR.