A long, long time ago I did some analysis on the correlation between defensive performance and player experience. The results showed, somewhat surprisingly, no correlation. However, the consistent knock was that the analysis leaves out player talent level. This article will hopefully lay that issue to rest. We will look at the talent level of defenses throughout the NCAA, examine the correlation between the average rivals star rating of the defensive players on the two-deep and the defensive performance, and conclude that it is quite significant. We will weep a little at Michigan's poor performance, but end up hopeful that from the ashes of 2010, a salvageable defense can rise in 2011.
As before, I'm scraping the depth chart data from rivals for the 97 teams for which it is available, and for this analysis, using only the defensive players. As before, I calculate "experience" from the players academic year (fr = 1, so = 2, jr = 3, se = 4, gs = 5, rs += 1).* I then take a simple average to come up with an experience score for the defense.
Next, I scraped the rivals recruiting database to get star ratings. Unfortunately, even though both data sources are from rivals, they have a significant amount of mismatches, so I had to go back through and find ratings for several hundred players, in cases where, for example, rivals had the full name in the two-deep but only the first initial in the recruiting database. Yes, this was a massive pain in my ass**, and it is why this analysis doesn't include offensive data - I didn't feel like looking up another several hundred players***. Players that I couldn't find got 1 star. (eg, the Michigan 1 stars are: Moundros, Kovacs, Leach) Finally, with stars for every player, I took an average for each defense.
"Defensive Performance" comes from Football Outsiders. I'm back to using the S&P+ out of personal preference.
First lets look at a plot of experience vs talent for defenses in the NCAA last year:
Each point is a team, their average "talent" as measured by rivals star ratings is on the X axis, and "experience", as measured by average years on the team, is on the Y axis. The blue lines cross at Michigan's point. As you can see, Michigan's defense last year averaged right around 3 stars, which is not the worst in the NCAA but certainly not the best.
Lets summarize this data with a bunch of tables:
Teams with most talented defenses:
Teams with least talented defenses:
Talent on Big 10 teams + Nebraska and ND:
Teams nearest to Michigan by Euclidian distance:
|Team||Stars avg.||Exp. avg.|
Huh. . . that's actually some pretty good company.
Ok, next, lets look at how talent correlates with defensive performance:
Finally - a good correlation! After staring at those experience vs performance shotgun blasts, this is beautiful. Clearly, having a more talented defense leads to better performance. In a linear regression model with defensive talent (avg. rivals stars) and defensive experience (avg. years on team) as predictors and defensive performance (Football Outsiders' S&P+) as the target, talent is a significant predictor (p = 3.49e-11) of defensive performance with a large effect size (each avg star increases a defense's S&P+ score by 14.8). R2 is so-so, at 0.38. However, as we saw in previous analyses, years on the team is still not a significant predictor (p = 0.84). This underscores the extreme importance of recruiting.
The blue lines cross on Michigans point. Teams with the same level of talent turned in much better defensive performances, and teams with similar defensive performances pulled it off with much less talent. Blerg.
If we take the red line (best fit line) as a gauge of the performance a team should be able to get from a group of players with a given telant level, we can look at who is overperforming and underperforming that prediction by looking at the distance of the actual performance from the red line.
Top 10 outperforming their talent level:
|Team||Gain over Predicted|
Top 10 underperforming their talent level:
|Team||Loss from Predicted|
Big 10 + Nebraska and ND:
|Team||Distance from Expected|
I've said it before, I'll say it again: Blerg.
Interestingly, even after including talent in the regression, experience ("years on the team") is still nowhere close to being a significant predictor of defensive performance. Getting older is not guaranteed to make your team appreciably better, but getting talent on the field does. There is also clearly a range of outcomes available at each level of talent, exemplefied by Boise State and TCU (assuming FO S&P+ really does account for strength of schedule). We might attribute this to a factor not included in the regression analysis, eg. "coaching".
All told, there are no excuses in Michigan's average experience or talent level that can account for the defensive performance in 2010. In terms of average talent and experience, this team resembles some of the best teams in the country, including the national champions. Hopefully, this means there is no reason that there can't be a huge turnaround in 2011.
Unfortunately, SD State doesn't have a depth chart on rivals, so they weren't included in this analysis. It might be interesting to compare their performance to expectations - maybe I'll do that for my next diary.
* Yes, I know this counts "years on the team" and not "years as a starter", quit telling me that.
** I discovered a bug in rivals recruit search. Go here: http://rivals.yahoo.com/ncaa/football/recruiting/recruit-search and put Greg Banks in the first and last name boxes, and select "any year" in the drop down. Hit search. Now go back and search for Greg Bank.
*** Though if anyone wants to help me compile this data I'd be open to that, we could use more excellent articles like this one: http://mgoblog.com/diaries/recruiting-bias-and-accuracy