Experience, Recruiting and Defensive Performance

Submitted by ebv on May 15th, 2011 at 9:17 PM

A long, long time ago I did some analysis on the correlation between defensive performance and player experience. The results showed, somewhat surprisingly, no correlation. However, the consistent knock was that the analysis leaves out player talent level. This article will hopefully lay that issue to rest. We will look at the talent level of defenses throughout the NCAA, examine the correlation between the average rivals star rating of the defensive players on the two-deep and the defensive performance, and conclude that it is quite significant. We will weep a little at Michigan's poor performance, but end up hopeful that from the ashes of 2010, a salvageable defense can rise in 2011.


As before, I'm scraping the depth chart data from rivals for the 97 teams for which it is available, and for this analysis, using only the defensive players. As before, I calculate "experience" from the players academic year (fr = 1, so = 2, jr = 3, se = 4, gs = 5, rs += 1).* I then take a simple average to come up with an experience score for the defense.

Next, I scraped the rivals recruiting database to get star ratings. Unfortunately, even though both data sources are from rivals, they have a significant amount of mismatches, so I had to go back through and find ratings for several hundred players, in cases where, for example, rivals had the full name in the two-deep but only the first initial in the recruiting database. Yes, this was a massive pain in my ass**, and it is why this analysis doesn't include offensive data - I didn't feel like looking up another several hundred players***. Players that I couldn't find got 1 star. (eg, the Michigan 1 stars are: Moundros, Kovacs, Leach) Finally, with stars for every player, I took an average for each defense.

"Defensive Performance" comes from Football Outsiders.  I'm back to using the S&P+ out of personal preference.


First lets look at a plot of experience vs talent for defenses in the NCAA last year:

Each point is a team, their average "talent" as measured by rivals star ratings is on the X axis, and "experience", as measured by average years on the team, is on the Y axis.  The blue lines cross at Michigan's point.  As you can see, Michigan's defense last year averaged right around 3 stars, which is not the worst in the NCAA but certainly not the best.

Lets summarize this data with a bunch of tables:


Teams with most talented defenses:

Team Stars avg.
florida 4
alabama 3.909090909
usc 3.909090909
texas 3.863636364
florida state 3.818181818
notre dame 3.818181818
miami-fl 3.772727273
oklahoma 3.772727273
ohio state 3.727272727
georgia 3.590909091


Teams with least talented defenses:

Team Stars avg.
florida int'l 1.111111111
florida atlantic 1.315789474
army 1.318181818
smu 1.684210526
nevada 1.818181818
colorado state 1.863636364
indiana 1.863636364
ulm 1.863636364
western kentucky 1.863636364
houston 1.866666667


Talent on Big 10 teams + Nebraska and ND:

Team Stars avg.
notre dame 3.818181818
ohio state 3.727272727
penn state 3.272727273
michigan 3
illinois 3.090909091
nebraska 3.045454545
michigan state 2.863636364
minnesota 2.863636364
wisconsin 2.590909091
northwestern 2.545454545
iowa 2.409090909
purdue 2.318181818
indiana 1.863636364


Teams nearest to Michigan by Euclidian distance:

Team Stars avg. Exp. avg.
michigan 3 2.789473684
arizona state 3.045454545 2.818181818
arizona 3.045454545 2.863636364
ole miss 3.090909091 2.818181818
auburn 3.090909091 2.727272727
georgia tech 2.863636364 2.772727273
ok state 3.090909091 2.681818182
arkansas 3.181818182 2.863636364
nebraska 3.045454545 2.590909091
miss. state 2.818181818 2.681818182

Huh. . . that's actually some pretty good company.

Ok, next, lets look at how talent correlates with defensive performance:


Finally - a good correlation! After staring at those experience vs performance shotgun blasts, this is beautiful.  Clearly, having a more talented defense leads to better performance.  In a linear regression model with defensive talent (avg. rivals stars) and defensive experience (avg. years on team) as predictors and defensive performance (Football Outsiders' S&P+) as the target, talent is a significant predictor (p = 3.49e-11) of defensive performance with a large effect size (each avg star increases a defense's S&P+ score by 14.8).  R2 is so-so, at 0.38.  However, as we saw in previous analyses, years on the team is still not a significant predictor (p = 0.84).  This underscores the extreme importance of recruiting.

The blue lines cross on Michigans point.  Teams with the same level of talent turned in much better defensive performances, and teams with similar defensive performances pulled it off with much less talent.  Blerg.

If we take the red line (best fit line) as a gauge of the performance a team should be able to get from a group of players with a given telant level, we can look at who is overperforming and underperforming that prediction by looking at the distance of the actual performance from the red line.


Top 10 outperforming their talent level:

Team Gain over Predicted
boise state 53.08772727
tcu 33.45090909
utah 23.26409091
ohio state 19.71
iowa 19.35818182
smu 17.48
miss. state 17.43409091
boston college 16.16318182
south carolina 15.61545455
stanford 15.33681818


Top 10 underperforming their talent level:

Team Loss from Predicted
kansas -22.19545455
texas-el paso -21.23454545
michigan -19.32894737
minnesota -17.88181818
virginia -17.76636364
memphis -17.37857143
georgia tech -17.29954545
usc -16.72
northwestern -13.96
new mexico -13.858


Big 10 + Nebraska and ND:

Team Distance from Expected
ohio state 19.71
iowa 19.35818182
nebraska 13.73136364
wisconsin 13.61090909
illinois 11.08909091
notre dame 8.942727273
purdue 8.308181818
nevada 7.542727273
michigan state 3.183181818
penn state -3.197727273
indiana -8.669090909
northwestern -13.96
minnesota -17.88181818
michigan -19.32894737

I've said it before, I'll say it again: Blerg.


Interestingly, even after including talent in the regression, experience ("years on the team") is still nowhere close to being a significant predictor of defensive performance.  Getting older is not guaranteed to make your team appreciably better, but getting talent on the field does.  There is also clearly a range of outcomes available at each level of talent, exemplefied by Boise State and TCU (assuming FO S&P+ really does account for strength of schedule).  We might attribute this to a factor not included in the regression analysis, eg. "coaching".

All told, there are no excuses in Michigan's average experience or talent level that can account for the defensive performance in 2010.  In terms of average talent and experience, this team resembles some of the best teams in the country, including the national champions.  Hopefully, this means there is no reason that there can't be a huge turnaround in 2011.

Unfortunately, SD State doesn't have a depth chart on rivals, so they weren't included in this analysis.  It might be interesting to compare their performance to expectations - maybe I'll do that for my next diary.


* Yes, I know this counts "years on the team" and not "years as a starter", quit telling me that.

** I discovered a bug in rivals recruit search. Go here: http://rivals.yahoo.com/ncaa/football/recruiting/recruit-search and put Greg Banks in the first and last name boxes, and select "any year" in the drop down. Hit search. Now go back and search for Greg Bank.

*** Though if anyone wants to help me compile this data I'd be open to that, we could use more excellent articles like this one: http://mgoblog.com/diaries/recruiting-bias-and-accuracy



May 15th, 2011 at 9:42 PM ^

...I think he's got it!

There is also clearly a range of outcomes available at each level of talent, exemplefied by Boise State and TCU [and Michigan and UTEP and Kansas] (assuming FO S&P+ really does account for strength of schedule).  We might attribute this to a factor not included in the regression analysis, eg. "coaching".

DOTW candidate, I'd say.

turd ferguson

May 15th, 2011 at 9:44 PM ^

Very interesting stuff.  Thanks for putting the time in and posting.  I'm going to nitpick with one line, but I enjoyed this.

I think you overstated your case a bit when you said, "Clearly, having a more talented defense leads to better performance."  I'm sure that's true, but your model doesn't account for a whole bunch of variables that could leave your estimators biased.  For example, more talented teams generally will have better coaches and facilities (since they spend more money, have people capable of bringing in that talent, etc.).  If you gave Florida and Florida International equal defensive talent this year, I suspect the Florida defense would significantly outperform the Florida International defense.  Florida's coaches are better, its weight room is better, its offense leaves the defense in better situations, etc.  If I'm right about that, there's probably a decent amount of omitted variable bias in there.  What you're seeing could be as much about all of the other advantages as it is about the talent difference.

These things shouldn't cause such serious problems for the experience analyses, since experience and program resources are probably much less correlated.


May 15th, 2011 at 11:59 PM ^

The R-squared is 0.38 which basically means that 38% of the variation in defenses is due to talent level. That is likely the most significant factor but coaching is probably what accounts for much of the remaining variance (which the OP clearly states, it's just nearly impossible to quantify that).  


Differences in weight rooms, facilities, etc. probably has very little to do with the variantian in defences.  Those things are important mostly for the recruiting aspects, in that it makes it easier to land top talent.  If you took a talented team with a stellar coach, they would be an elite defense with the CCRB as their weight room and Elbel for a practice field.  I'm not saying those things don't matter, but it's probably a marginal effect, especially considering that most, if not all, Div 1A schools have suffient weight and practice facilities.

turd ferguson

May 16th, 2011 at 12:17 AM ^

Someone should correct me if my stats knowledge is rusty, but I'm making a different point.

His R-squared tells him how much of the variation can be explained by his "talent" variable (not some underlying true talent level).  I'm not saying anything about the remaining 62%.  I think that could be coaching, luck, or all kinds of things.  I'm saying that a good part of the 38% might not really be about talent.  That variable is catching a bunch of stuff (coaching, facilities, hostile home environments, etc.), since schools that attract high-end talent are very different from schools that don't.  Without controlling for that other stuff - and I'm not sure how to do it - I think this overstates the effects of greater talent.  I suspect that the R-squared on talent would drop, for example, if we could somehow include coaching skill as a control variable.

Having said that, I never think about stats in terms of R-squared, so I might be off with this.


May 16th, 2011 at 1:00 AM ^

R2 never drops when you add variables to your model. What may change is the statistical significance of the "talent" variable. What you suggest is closer to what is called omitted variable bias. If there exists a variable which (1) is not included in the model (2) is correlated with the talent variable (3) is correlated with success as a defense, then your estimate of the relationship between talent and success will be biased. In the examples you give, the bias is likely to be positive, which means we would overestimate the magnitude of the coefficient on the talent variable. But that's not the same as the R2.

Blue in Seattle

May 16th, 2011 at 4:04 PM ^

But when you clarified your point, I think you just made it be about explaining how schools get talent.  Understanding this doesn't change the measure of talent that was selected, which is the Rivals star system.  In other words, the fact that OSU recruits more 5 stars than other teams and ends up with a higher talent average as a result may be their really cool weight room, it still doesn't change the fact that they have more talent recruited on average.

Also guessing that the variable that could explain more of the remaining R squared is coaching doesn't help assign weight rooms, etc.  For the variable "coaching" to be added to the formula, all you need is an agreed upon metric for "coaching".

Now some would argue that Win/Loss record is the metric for coaching ability.  But that is too broad, since it also is affected by talent level.

A better metric for coaching, in my mind, would be the measure of NFL recruits that a coaching program achieves from the talent that they previously recruited.  I think someone else has already tried an approach at evaluating the schools who over and under achieve at turning their High School Star Talent into NFL Star Talent.  Again you could point to facilties, but I think that is just a basic tool that the coach uses.  Yes I would agree that the best coaches get paid the most from the schools with the most money, etc.  But then again, Boise State is one of the over achievers.  I'm sure that comparitive to other schools their size, they have better facilities, but do they really have better facilities than Michigan?  Do you think that delta is more significant than who is specifically coaching the talent to use those facilities?

I think this would be very valuable analysis, since this site seems to have a very strong focus on both recruiting and "schemes".  And early in the Rodriguez years I remember the younger non-coach demographic getting into comment battles with the older coaching demographic about scheme versus talent.

I find it very shocking that last year Michigan had basically average talent and bottom of the barrel defensive performance. 

This analysis seems to indicate that with Talent being a key indicator (now granted there was no assigning of scheme to the schools, but I think the top schools and over achievers are broad enough to cover the spread from Pro-Style to errr, spread style), that a wise coach would try and adjust their scheme to match their talent.

The fact that Brady Hoke's staff have repeated that over and over is the biggest thing giving me hope for next year.

Assuming that coaching thing explains most of that remaining 60+%





May 15th, 2011 at 10:03 PM ^

One of the inputs I'd like to see added is strength of opposition (in offense).

While Boise State may perform very well compared to the talent they have, to put into perspective the quality of opposition they are playing is important IMHO.

Of course - this is is just a suggestion and doesn't discount your analysis in any way.

Eye of the Tiger

May 16th, 2011 at 1:12 AM ^

Two things.  

1. You have us ranked above two other Big 10 teams with higher star averages on defense (Illinois and Nebraska).  That changes your conclusion somewhat: in actuality, we didn't have elite-level talent, though we did still underperform.  Basically we had a middle-of-the-pack defense in terms of evaluated talent, and a bottom-of-the-barrel defense in terms of performance.  

2. One thing you might want to consider, if you update this, is the average years of experience on (Edit: thanks to the poster below for making this more clear) the starting lineup.  A redshirt senior 3* whose started for 3 years is not the same as a true freshman 3*.  We started a bunch of young kids in 2010, whereas Michigan State did not.  I wonder if the results would change if you controlled for this?


May 15th, 2011 at 10:39 PM ^

Good work, very interesting stuff. It would be interesting to see if the overachievers and 'underpreformers' had significant variations in experience from the national mean.

FYI- Your name (ebv) is the acronym for the Epstein-Barr virus- the major cause of mono, an endemic African lymphoma called Burkitt's and nasopharyngeal carcinoma (nose cancer). Sincerly, your friendly neighborhood pathologist. Cheers.


May 15th, 2011 at 11:25 PM ^

One thing to add:

Differing abilities across levels of the secondary could account for some of the variation in the data.  So, for example, our secondary was terrible last year - lots of painful 3rd and 10+ yard conversions.  That wasted any good play by our defensive line.  A really bad line could have the same effect on  a good secondary by never getting to the QB.

So, imbalanced ability across levels of the defense could cause underperformance while a perfectly balanced defense would result in better than average performance.

Obviously, this would tremendously complicate the analysis and I'm not expecting anyone to run it, but I thought I would mention it anyway.



May 16th, 2011 at 10:30 AM ^

Yeah, that makes a lot of sense.  Football Outsiders actually breaks down defensive performance into passing and rushing, but I think the responsibilities are too intertwined to split the team's talent / experience into pass and rush.

One thing I might do is to look at the standard deviation of talent on the field - that might get at your point in a round-about way.  


May 16th, 2011 at 12:26 PM ^

when 1 and 2 star talent replaces 3-5 star talent?  The attrition argument was based on the weak-link-in-the-chain hypothesis, so knowing how much damage having some percentage of relatively untalented players would be is the angle of attack there, right?

Though honestly, I have to say I don't expect that to change the results that much.  Teams like Boise and MissSt are probably going to have a similar number of 1-2 star types as Michigan had.  The argument will still come down to how expressive those star ratings were in the particular instances we're talking about.  I don't think it's too difficult to accept that 1-2 star types for Boise and Iowa are different from 1-2 star types for Michigan.  M is very unlikely to be playing players who were overlooked given the respect they're afforded in the recruiting game.  Not so much for BSU and Iowa.

Either way, it's still clear that there was a pretty critical coaching malfunction.  Whatever your argument, I think this post makes it evident that there's almost no way the talent was so bad that they merely played to their talent level.  GERG was a debacle.

Thanks for doing all the hard work.  Knocked it out of the park.


May 16th, 2011 at 12:06 AM ^

I've been thinking about doing this analysis since you put out your initial post on experience, but I've been too lazy.  Glad you got around to it.  I've been in the camp that has been arguing that we need better players much more than we need to give the current guys time to develop (all the people claiming that the defense would be better next year with the young guys having another year of experience).  

Like you, I would expect the defense to get up to at least the talent expectations next year with Mattison.  Will be a couple years til the coaching and talent gets up to elite levels.


May 16th, 2011 at 12:52 AM ^

Thats really cool! Thanks a lot for doing all this work!

It's not too surprising that you get no statistically significant association between experience and success because you have next to no variation in your experience measure. By taking the mean over the entire defense, you squeeze every team toward the average number of years that athletes spend on scholarship. Since all teams tend to take the same number of athletes every year, they'll all have similar mean experience,hence the flat distribution of cases (excluding outliers) in your experience vs ratings graph.

The information we really need to have is how much experience does the starting line have. Really, the experience that really matters is the experience that's at the front of the depth chart. The mean experience for the whole Michigan defense will have changed a bit between this and last season, but the experience of the starting players will have change proportionately more. The latter is what we care about, and it is not reflected in your analysis.

Thanks for the hard work!


May 16th, 2011 at 10:06 PM ^

First, there is plenty of variance in experience to be able to pick up correlation if it was there.  You must not have read the post from a while back where he explains that he uses the depth charts from rivals (which are often just one or two players deep) so he's not using guys that dont see the field.


May 16th, 2011 at 11:25 AM ^

Would it be possible to use "years in the two-deep" instead of "years on the team"? A player in the two-deep has a different experience than one who is not, both in practice and during games.

Maybe the main thing that "years on the team" shows is that age is not a significant factor? That is interesting -- maybe what it shows is that most kids who ever make into the two-deep in a D-I program are highly disciplined, so maturity doesn't matter as much as one might expect.


May 17th, 2011 at 3:28 PM ^

The question that would be fascinating to answer is to what extent the errors are correlated. M fans want to knlow is whether Michigan is going to underperform again.


May 17th, 2011 at 9:39 PM ^

If the errors are due largely to coaching, then we'll see if the new staff does much better.  My guess is the offense takes a small step back and the defense a sizable step forward, albeit just to the middle of the 1A pack.  May not show up in the W/L since we were a "lucky" team last year.  Anywhere from 6 to 8 wins is the reasonable expectation.

Some of the error is just pure chance so you'd have to hope for a regression to the mean there.  But we've been hoping for that for a couple years.