A few weekends ago I did some analysis that tried to answer the question: “Do good defenses need experience?” In short, if experience is vital to a team’s performance, then we expect to see a positive correlation between a team’s experience and performance. Instead, the correlation was very weak, indicating that experience is not a major factor in defensive performance.
However, there were a number of major caveats with the approach I took, so I’ve redone some of the comparisons with the hope of addressing some of the issues. I’ve also added the same comparison for offense.
I’ve changed the approach to help eliminate some of the caveats to my last analysis. Many people pointed out that players who never see the field probably skew the results. I’m now scraping data from the depth charts on rivals.com, to eliminate that bias. Not every team has a depth chart listed, and some were in a different format that I did not take the time to parse, but the data includes 97 teams (essentially the ones with a themed team site on rivals.)
As a bonus, rivals includes information on redshirts, which were left out of the previous analysis. I calculate an experience score from the player’s academic year (Freshman = 1, Sophomore = 2, Junior = 3, Senior = 4, Grad Student* = 5, Redshirt = +1) so a player’s score is roughly the number of years they’ve been on the team.** Then, because each team has a different number of players on their depth chart, I take an average. The resulting team score has the advantage of being easily interpreted as the average number of years the players who make up a unit have been on the team (assuming they started on the team as a freshman). I’ve split players by position, so each team will receive an offensive and defensive experience score. Finally, I matched up the defensive and offensive FEI scores from Football Outsiders (which takes strength of schedule into account) with the experience scores and use linear regression to test for correlation.
Of course, there are still some big caveats to this approach. The data still does not include injuries and transfers so this may be unfairly attributing experience to teams in some cases. The analysis also doesn’t account for talent (recruiting guru rankings) in any way. Finally, we’re trusting Football Outsiders to provide realistic performance measurements. If their rankings aren’t accurate, none of this analysis will be either.
Lets start with something new: the offense:
In the graph, each point is a team. Its offensive experience score is on the x-axis and its offensive performance score (Football Outsiders Offensive FEI) is on the y-axis. Larger numbers are better for FEI, so good teams are at the top. The results are almost the definition of no correlation. Experience is not a significant predictor of offensive performance (p = 0.18) and R2 is a miserable 0.02. We can only conclude that there is a lot more to offensive success than simply experience.
The blue lines cross on Michigan’s point. Despite the sophomore quarterbacks, Michigan fielded one of the most experienced offenses, with over a junior’s experience on average. Performance-wise, we’re in good shape with the second best offense in the nation.
The maroon lines cross on Mississippi State’s point. They have significantly less experience than Michigan, averaging ~ 2.5 years. They’re also performing much worse; many teams with the same level of experience have better offensive output.
Moving on to defense – I’ve multiplied the football outsiders defensive FEI score by -1, so good performances are again towards the top of the graph.
Again, the correlation is not significant (p = 0.5) and the R2 is very low (0.04). This is actually different from the previous analysis, where we saw a significant positive correlation with a tiny effect size.***
Again, the blue lines cross at Michigan’s point. Again, Michigan has one of the more experienced units, fielding over juniors on average. Mississippi State (maroon lines) fields a lot less experience, with about 2.73 years on average. However, despite the relative lack of experience, Mississippi State turned in a much better defensive performance this year than did Michigan.
Update - I've added a line for Michgan after the following player substitutions: Patterson for Watson (sr rs +1), Vinopal for Emillen (fr -1), and Demens for Jones (so rs +1) , removed Floyd (which I'm iffy about since he did play much of the year) (-3) and add in Avery, Christen, and Talbott (all fr +1). The result would be just below a junior's experience: 65 / 22 ~ 2.95. I've added the dashed, verticle line for that experience level. If you want to check my math, I posted the raw data for Michigan in the comments.
One final note – the pattern of elite units (either offense or defense) having middling experience holds for both sides of the ball, i.e. the top right of both graphs is empty. The NFL draft is probably the simplest explanation for this.
I am surprised by the nearly complete lack of correlation in the data.
Is it possible that we need more variables to make sense of this data? (Any stat experts know whether you can underestimate an effect by having too few variables?) It’s also possible that an interaction between several variables might provide a good explanation for our data; for example we might need experience, NFL-worthy talent and great coaching to produce an elite defense, but none of the three alone will do it.
Speaking of talent, yes, that information is not included here, and probably should be. However, we aren’t looking for a perfect correlation. If experience accounts for a significant fraction of a team’s performance, we ought to see some kind of correlation, even a weak one, with an R2 that indicates experience explains some portion of the variation in the performance variable. Instead, there’s almost no correlation at all. This really makes it look like performance comes down to coaching and recruiting, and not so much experience, at least in terms of years on the team.
An alternate explanation, put forward by MCalibur in the comments on my last post, is that the concentration of experience on the field is more important than simply gross experience. Perhaps coaches can hide some freshman mistakes behind solid play from battle-hardened seniors. Michigan’s true-freshman filled secondary might leave the coaches with no place to hide their players inexperience. If I continue to pursue this, I’ll try breaking experience down by position.
Update 2 On Steve Sharik's suggestion I repeated the comparison with the median. It cleans things up a little bit, but there still isn't good correltion.
* Don’t worry, there are no redshirt grad students, I checked.
** I got a number of excellent suggestions from bighouseinmate, MCalibur, expatriate, tpilews and probably others, that a more relevant measure of experience would be years as a starter as opposed to years on the team. I agree completely. Unfortunately, that information is not consistently available, so I’m sticking with academic year as a proxy for that until I can find a reliable source for that data.
*** Perhaps most surprisingly, we now see a negative (!) correlation between defensive performance and experience, which, if it were significant, would mean that as your team gains experience it turns out worse performances. However, these results are so close to random, the sign of the correlation should probably be ignored.