Experience, Recruiting and Defensive Performance
A long, long time ago I did some analysis on the correlation between defensive performance and player experience. The results showed, somewhat surprisingly, no correlation. However, the consistent knock was that the analysis leaves out player talent level. This article will hopefully lay that issue to rest. We will look at the talent level of defenses throughout the NCAA, examine the correlation between the average rivals star rating of the defensive players on the two-deep and the defensive performance, and conclude that it is quite significant. We will weep a little at Michigan's poor performance, but end up hopeful that from the ashes of 2010, a salvageable defense can rise in 2011.
Approach
As before, I'm scraping the depth chart data from rivals for the 97 teams for which it is available, and for this analysis, using only the defensive players. As before, I calculate "experience" from the players academic year (fr = 1, so = 2, jr = 3, se = 4, gs = 5, rs += 1).* I then take a simple average to come up with an experience score for the defense.
Next, I scraped the rivals recruiting database to get star ratings. Unfortunately, even though both data sources are from rivals, they have a significant amount of mismatches, so I had to go back through and find ratings for several hundred players, in cases where, for example, rivals had the full name in the two-deep but only the first initial in the recruiting database. Yes, this was a massive pain in my ass**, and it is why this analysis doesn't include offensive data - I didn't feel like looking up another several hundred players***. Players that I couldn't find got 1 star. (eg, the Michigan 1 stars are: Moundros, Kovacs, Leach) Finally, with stars for every player, I took an average for each defense.
"Defensive Performance" comes from Football Outsiders. I'm back to using the S&P+ out of personal preference.
Results
First lets look at a plot of experience vs talent for defenses in the NCAA last year:
Each point is a team, their average "talent" as measured by rivals star ratings is on the X axis, and "experience", as measured by average years on the team, is on the Y axis. The blue lines cross at Michigan's point. As you can see, Michigan's defense last year averaged right around 3 stars, which is not the worst in the NCAA but certainly not the best.
Lets summarize this data with a bunch of tables:
Teams with most talented defenses:
| Team | Stars avg. |
| florida | 4 |
| alabama | 3.909090909 |
| usc | 3.909090909 |
| texas | 3.863636364 |
| florida state | 3.818181818 |
| notre dame | 3.818181818 |
| miami-fl | 3.772727273 |
| oklahoma | 3.772727273 |
| ohio state | 3.727272727 |
| georgia | 3.590909091 |
Teams with least talented defenses:
| Team | Stars avg. |
| florida int'l | 1.111111111 |
| florida atlantic | 1.315789474 |
| army | 1.318181818 |
| smu | 1.684210526 |
| nevada | 1.818181818 |
| colorado state | 1.863636364 |
| indiana | 1.863636364 |
| ulm | 1.863636364 |
| western kentucky | 1.863636364 |
| houston | 1.866666667 |
Talent on Big 10 teams + Nebraska and ND:
| Team | Stars avg. |
| notre dame | 3.818181818 |
| ohio state | 3.727272727 |
| penn state | 3.272727273 |
| michigan | 3 |
| illinois | 3.090909091 |
| nebraska | 3.045454545 |
| michigan state | 2.863636364 |
| minnesota | 2.863636364 |
| wisconsin | 2.590909091 |
| northwestern | 2.545454545 |
| iowa | 2.409090909 |
| purdue | 2.318181818 |
| indiana | 1.863636364 |
Teams nearest to Michigan by Euclidian distance:
| Team | Stars avg. | Exp. avg. |
| michigan | 3 | 2.789473684 |
| arizona state | 3.045454545 | 2.818181818 |
| arizona | 3.045454545 | 2.863636364 |
| ole miss | 3.090909091 | 2.818181818 |
| auburn | 3.090909091 | 2.727272727 |
| georgia tech | 2.863636364 | 2.772727273 |
| ok state | 3.090909091 | 2.681818182 |
| arkansas | 3.181818182 | 2.863636364 |
| nebraska | 3.045454545 | 2.590909091 |
| miss. state | 2.818181818 | 2.681818182 |
Huh. . . that's actually some pretty good company.
Ok, next, lets look at how talent correlates with defensive performance:
Finally - a good correlation! After staring at those experience vs performance shotgun blasts, this is beautiful. Clearly, having a more talented defense leads to better performance. In a linear regression model with defensive talent (avg. rivals stars) and defensive experience (avg. years on team) as predictors and defensive performance (Football Outsiders' S&P+) as the target, talent is a significant predictor (p = 3.49e-11) of defensive performance with a large effect size (each avg star increases a defense's S&P+ score by 14.8). R2 is so-so, at 0.38. However, as we saw in previous analyses, years on the team is still not a significant predictor (p = 0.84). This underscores the extreme importance of recruiting.
The blue lines cross on Michigans point. Teams with the same level of talent turned in much better defensive performances, and teams with similar defensive performances pulled it off with much less talent. Blerg.
If we take the red line (best fit line) as a gauge of the performance a team should be able to get from a group of players with a given telant level, we can look at who is overperforming and underperforming that prediction by looking at the distance of the actual performance from the red line.
Top 10 outperforming their talent level:
| Team | Gain over Predicted |
| boise state | 53.08772727 |
| tcu | 33.45090909 |
| utah | 23.26409091 |
| ohio state | 19.71 |
| iowa | 19.35818182 |
| smu | 17.48 |
| miss. state | 17.43409091 |
| boston college | 16.16318182 |
| south carolina | 15.61545455 |
| stanford | 15.33681818 |
Top 10 underperforming their talent level:
| Team | Loss from Predicted |
| kansas | -22.19545455 |
| texas-el paso | -21.23454545 |
| michigan | -19.32894737 |
| minnesota | -17.88181818 |
| virginia | -17.76636364 |
| memphis | -17.37857143 |
| georgia tech | -17.29954545 |
| usc | -16.72 |
| northwestern | -13.96 |
| new mexico | -13.858 |
Big 10 + Nebraska and ND:
| Team | Distance from Expected |
| ohio state | 19.71 |
| iowa | 19.35818182 |
| nebraska | 13.73136364 |
| wisconsin | 13.61090909 |
| illinois | 11.08909091 |
| notre dame | 8.942727273 |
| purdue | 8.308181818 |
| nevada | 7.542727273 |
| michigan state | 3.183181818 |
| penn state | -3.197727273 |
| indiana | -8.669090909 |
| northwestern | -13.96 |
| minnesota | -17.88181818 |
| michigan | -19.32894737 |
I've said it before, I'll say it again: Blerg.
Conclusions
Interestingly, even after including talent in the regression, experience ("years on the team") is still nowhere close to being a significant predictor of defensive performance. Getting older is not guaranteed to make your team appreciably better, but getting talent on the field does. There is also clearly a range of outcomes available at each level of talent, exemplefied by Boise State and TCU (assuming FO S&P+ really does account for strength of schedule). We might attribute this to a factor not included in the regression analysis, eg. "coaching".
All told, there are no excuses in Michigan's average experience or talent level that can account for the defensive performance in 2010. In terms of average talent and experience, this team resembles some of the best teams in the country, including the national champions. Hopefully, this means there is no reason that there can't be a huge turnaround in 2011.
Unfortunately, SD State doesn't have a depth chart on rivals, so they weren't included in this analysis. It might be interesting to compare their performance to expectations - maybe I'll do that for my next diary.
* Yes, I know this counts "years on the team" and not "years as a starter", quit telling me that.
** I discovered a bug in rivals recruit search. Go here: http://rivals.yahoo.com/ncaa/football/recruiting/recruit-search and put Greg Banks in the first and last name boxes, and select "any year" in the drop down. Hit search. Now go back and search for Greg Bank.
*** Though if anyone wants to help me compile this data I'd be open to that, we could use more excellent articles like this one: http://mgoblog.com/diaries/recruiting-bias-and-accuracy
...I think he's got it!
There is also clearly a range of outcomes available at each level of talent, exemplefied by Boise State and TCU [and Michigan and UTEP and Kansas] (assuming FO S&P+ really does account for strength of schedule). We might attribute this to a factor not included in the regression analysis, eg. "coaching".
DOTW candidate, I'd say.
Very interesting stuff. Thanks for putting the time in and posting. I'm going to nitpick with one line, but I enjoyed this.
I think you overstated your case a bit when you said, "Clearly, having a more talented defense leads to better performance." I'm sure that's true, but your model doesn't account for a whole bunch of variables that could leave your estimators biased. For example, more talented teams generally will have better coaches and facilities (since they spend more money, have people capable of bringing in that talent, etc.). If you gave Florida and Florida International equal defensive talent this year, I suspect the Florida defense would significantly outperform the Florida International defense. Florida's coaches are better, its weight room is better, its offense leaves the defense in better situations, etc. If I'm right about that, there's probably a decent amount of omitted variable bias in there. What you're seeing could be as much about all of the other advantages as it is about the talent difference.
These things shouldn't cause such serious problems for the experience analyses, since experience and program resources are probably much less correlated.
The R-squared is 0.38 which basically means that 38% of the variation in defenses is due to talent level. That is likely the most significant factor but coaching is probably what accounts for much of the remaining variance (which the OP clearly states, it's just nearly impossible to quantify that).
Differences in weight rooms, facilities, etc. probably has very little to do with the variantian in defences. Those things are important mostly for the recruiting aspects, in that it makes it easier to land top talent. If you took a talented team with a stellar coach, they would be an elite defense with the CCRB as their weight room and Elbel for a practice field. I'm not saying those things don't matter, but it's probably a marginal effect, especially considering that most, if not all, Div 1A schools have suffient weight and practice facilities.
Someone should correct me if my stats knowledge is rusty, but I'm making a different point.
His R-squared tells him how much of the variation can be explained by his "talent" variable (not some underlying true talent level). I'm not saying anything about the remaining 62%. I think that could be coaching, luck, or all kinds of things. I'm saying that a good part of the 38% might not really be about talent. That variable is catching a bunch of stuff (coaching, facilities, hostile home environments, etc.), since schools that attract high-end talent are very different from schools that don't. Without controlling for that other stuff - and I'm not sure how to do it - I think this overstates the effects of greater talent. I suspect that the R-squared on talent would drop, for example, if we could somehow include coaching skill as a control variable.
Having said that, I never think about stats in terms of R-squared, so I might be off with this.
Oh, right. Thanks. I'll stand by my original point about omitted variable bias (and overestimating the effects of talent), but I don't know what the hell I'm talking about with respect to R-squared.
But when you clarified your point, I think you just made it be about explaining how schools get talent. Understanding this doesn't change the measure of talent that was selected, which is the Rivals star system. In other words, the fact that OSU recruits more 5 stars than other teams and ends up with a higher talent average as a result may be their really cool weight room, it still doesn't change the fact that they have more talent recruited on average.
Also guessing that the variable that could explain more of the remaining R squared is coaching doesn't help assign weight rooms, etc. For the variable "coaching" to be added to the formula, all you need is an agreed upon metric for "coaching".
Now some would argue that Win/Loss record is the metric for coaching ability. But that is too broad, since it also is affected by talent level.
A better metric for coaching, in my mind, would be the measure of NFL recruits that a coaching program achieves from the talent that they previously recruited. I think someone else has already tried an approach at evaluating the schools who over and under achieve at turning their High School Star Talent into NFL Star Talent. Again you could point to facilties, but I think that is just a basic tool that the coach uses. Yes I would agree that the best coaches get paid the most from the schools with the most money, etc. But then again, Boise State is one of the over achievers. I'm sure that comparitive to other schools their size, they have better facilities, but do they really have better facilities than Michigan? Do you think that delta is more significant than who is specifically coaching the talent to use those facilities?
I think this would be very valuable analysis, since this site seems to have a very strong focus on both recruiting and "schemes". And early in the Rodriguez years I remember the younger non-coach demographic getting into comment battles with the older coaching demographic about scheme versus talent.
I find it very shocking that last year Michigan had basically average talent and bottom of the barrel defensive performance.
This analysis seems to indicate that with Talent being a key indicator (now granted there was no assigning of scheme to the schools, but I think the top schools and over achievers are broad enough to cover the spread from Pro-Style to errr, spread style), that a wise coach would try and adjust their scheme to match their talent.
The fact that Brady Hoke's staff have repeated that over and over is the biggest thing giving me hope for next year.
Assuming that coaching thing explains most of that remaining 60+%
Yeah, you're absolutly right, I shouldn't have worded that statement to imply causation.
One of the inputs I'd like to see added is strength of opposition (in offense).
While Boise State may perform very well compared to the talent they have, to put into perspective the quality of opposition they are playing is important IMHO.
Of course - this is is just a suggestion and doesn't discount your analysis in any way.
I think he's okay here. Football Outsiders incorporates the quality of a team's opponents into its S&P ratings, so that's accounted for in ebv's outcome variable.
Thanks for the info.
Great link - cheers
Two things.
1. You have us ranked above two other Big 10 teams with higher star averages on defense (Illinois and Nebraska). That changes your conclusion somewhat: in actuality, we didn't have elite-level talent, though we did still underperform. Basically we had a middle-of-the-pack defense in terms of evaluated talent, and a bottom-of-the-barrel defense in terms of performance.
2. One thing you might want to consider, if you update this, is the average years of experience on (Edit: thanks to the poster below for making this more clear) the starting lineup. A redshirt senior 3* whose started for 3 years is not the same as a true freshman 3*. We started a bunch of young kids in 2010, whereas Michigan State did not. I wonder if the results would change if you controlled for this?
Good work, very interesting stuff. It would be interesting to see if the overachievers and 'underpreformers' had significant variations in experience from the national mean.
FYI- Your name (ebv) is the acronym for the Epstein-Barr virus- the major cause of mono, an endemic African lymphoma called Burkitt's and nasopharyngeal carcinoma (nose cancer). Sincerly, your friendly neighborhood pathologist. Cheers.
or its his initials. 50-50 could go either way.
One thing to add:
Differing abilities across levels of the secondary could account for some of the variation in the data. So, for example, our secondary was terrible last year - lots of painful 3rd and 10+ yard conversions. That wasted any good play by our defensive line. A really bad line could have the same effect on a good secondary by never getting to the QB.
So, imbalanced ability across levels of the defense could cause underperformance while a perfectly balanced defense would result in better than average performance.
Obviously, this would tremendously complicate the analysis and I'm not expecting anyone to run it, but I thought I would mention it anyway.
Yeah, that makes a lot of sense. Football Outsiders actually breaks down defensive performance into passing and rushing, but I think the responsibilities are too intertwined to split the team's talent / experience into pass and rush.
One thing I might do is to look at the standard deviation of talent on the field - that might get at your point in a round-about way.
when 1 and 2 star talent replaces 3-5 star talent? The attrition argument was based on the weak-link-in-the-chain hypothesis, so knowing how much damage having some percentage of relatively untalented players would be is the angle of attack there, right?
Though honestly, I have to say I don't expect that to change the results that much. Teams like Boise and MissSt are probably going to have a similar number of 1-2 star types as Michigan had. The argument will still come down to how expressive those star ratings were in the particular instances we're talking about. I don't think it's too difficult to accept that 1-2 star types for Boise and Iowa are different from 1-2 star types for Michigan. M is very unlikely to be playing players who were overlooked given the respect they're afforded in the recruiting game. Not so much for BSU and Iowa.
Either way, it's still clear that there was a pretty critical coaching malfunction. Whatever your argument, I think this post makes it evident that there's almost no way the talent was so bad that they merely played to their talent level. GERG was a debacle.
Thanks for doing all the hard work. Knocked it out of the park.
I've been thinking about doing this analysis since you put out your initial post on experience, but I've been too lazy. Glad you got around to it. I've been in the camp that has been arguing that we need better players much more than we need to give the current guys time to develop (all the people claiming that the defense would be better next year with the young guys having another year of experience).
Like you, I would expect the defense to get up to at least the talent expectations next year with Mattison. Will be a couple years til the coaching and talent gets up to elite levels.
First, there is plenty of variance in experience to be able to pick up correlation if it was there. You must not have read the post from a while back where he explains that he uses the depth charts from rivals (which are often just one or two players deep) so he's not using guys that dont see the field.
Would it be possible to use "years in the two-deep" instead of "years on the team"? A player in the two-deep has a different experience than one who is not, both in practice and during games.
Maybe the main thing that "years on the team" shows is that age is not a significant factor? That is interesting -- maybe what it shows is that most kids who ever make into the two-deep in a D-I program are highly disciplined, so maturity doesn't matter as much as one might expect.
and clearly a great effort. Amazing.
If the errors are due largely to coaching, then we'll see if the new staff does much better. My guess is the offense takes a small step back and the defense a sizable step forward, albeit just to the middle of the 1A pack. May not show up in the W/L since we were a "lucky" team last year. Anywhere from 6 to 8 wins is the reasonable expectation.
Some of the error is just pure chance so you'd have to hope for a regression to the mean there. But we've been hoping for that for a couple years.
Comments