Experience vs defensive performance

Submitted by ebv on

Let me start this off by thanking all the players for their incredible efforts on the field this season.  Vincent Smith blocking JJ Watt is the defining image for me.  Vinopal stopping Leshoure for a loss on 3rd and 1 is right up there too.

So what is the purpose of this?

I want to explore how much of our defensive performance we can plausibly blame the coaches for, and what is an inevitable part of a young program.  Is there an uninterruptible progression in football ability from freshman to senior, or can good coaching get a kid to play like someone with 2 or more years of experience?  Everyone seems to have an opinion, but what does the data say?

How will you test that? 

I’ve scraped rosters for every FBS team from ESPN.  For each team, I calculate an experience score by assigning a value to each academic year (Senior = 4, Junior = 3, Sophomore = 2, Freshman = 1)  and summing over defensive players**.  High scores will go to big teams with lots of seniors.  Low scores will go to small teams with lots of freshman.  I then measured the correlation with the Football outsider’s defensive S&P+, an aggregate measure of defensive performance that does not (as far as I can tell) include experience.  If performance is predictable from experience, then there is not much the coaches can do aside from wait for the team to grow up.  However, if there is no correlation, then coaches can influence performance by increasing talent levels (recruiting) and ability (coaching).

What are the results?

The data takes on what appears to be a normal distribution with most teams in the middle in terms of both experience and performance, with just a few at the extremes.  Tulane is a major outlier in experience, with an experience score of 63.  Scout.com’s Tulane season preview starts by saying “Few units in America are greater coach killers than the Tulane D.”  More about them later.  The high end of experience features Navy, Nebraska, Army, Ohio, Kansas State, and Notre Dame, in that order.  Michigan has an experience score of 101 and an S&P+ score of 90.8, locating us near the peak of both distributions.

In the next figure, each point is a team, with its experience score on the x-axis and its defensive performance, as measured by Football Outsiders, on the y-axis.  The plot is roughly cone shaped: elite defenses are impossible without a certain level of experience, while at the same time, experienced defenses can be crappy.  It is interesting to note that the upper right is empty, indicating that a team can, somehow, be too experienced.  Could this be explained by the NFL draft?

There is a positive correlation with p = 0.026, indicating that it is not random.  Every year of experience leads to a gain of 0.2 S&P+ units.   In short, getting older makes the team (a little bit) better.

R2 is very small (0.042), meaning that experience can only explain about 4% of the variation in performance.  Most of a team’s defensive performance is not explained from its experience alone.

The red line shows the best fit linear regression model for the data.  Teams above the line are performing better than we would expect from experience alone, while teams below are worse.  The horizontal yellow line shows Michigan’s defensive performance. It is right about the same as Tulane’s (the leftmost point with an experience score of 63).  Coach Killer you say?  The vertical, yellow lines will be discussed after first covering some caveats.

Huge, conclusion altering caveats:

First, ESPN does not include data on redshirts, which would probably change the experience picture substantially.   Second, we can not separate the portions of a team’s defensive performance that are due to good coaching from those that result from raw talent / recruiting.  Third, ESPN does not give information about scholarships, so it is possible for the experience scores to be biased by players who will never see the field.  Fourth, the data does not reflect injuries so the team on the field may be much less experienced than the roster.  Finally, FO Data doesn’t include yesterday’s games yet, so if there were big suprises (cough, Iowa) thet might change this analysis somewhat.  Try to interpret the data with these limitations in mind (i.e. don’t take this analysis too seriously).

Ok, so what are the vertical, yellow lines?

The right one shows Michigan’s experience score.  Note the wide range of possible defensive performance scores at this level of experience.  Clearly, Michigan has suffered its share of injuries this year, so the vertical, yellow line on the left shows the experience score if we remove the defensive players from the OSU injury report (JT Floyd, Mike Jones, Jared Van Slyke, Mike Williams, Troy Woolfolk) and Vladmir Emilien.  Even removing these players, the defense is still playing below the expectation based on their experience.  Note that we are comparing Michigan with injuries to the rest of the FBS without injuries, essentially assuming that Michigan’s injury situation is a wild outlier – an assumption I’m not sure is true.  However, even making this assumption, the team is still performing below average.

Can now we draw some sweeping conclusions about the level of coaching at every football program in the country?

Finally, I wanted to look at who was doing well relative to their experience.  To do this I calculated an expected S&P+ from the regression model (the red line) and subtracted that value from the actual S&P+ score.  I then looked for teams with a large difference in performance and expectation.   If the difference is positive, it means the team is exceeding the performance expected given their experience.  If it is negative, then the team is underperforming expectations.

Who is doing well?

Top ten teams and the amount they’re exceeding expectations, in S&P+ units.

Boise State

40.21204551

TCU

36.39315477

Ohio State

27.06871772

South Carolina

26.1609196

Miami-FL

25.8609196

Texas A&M

24.43648256

Iowa

20.82149088

Mississippi State

20.78535665

Illinois

20.64202887

West Virginia

19.73483531

Who is doing badly?

Bottom eleven teams, and the amount they’re underperforming expectations, in S&P+ units.

Indiana

-14.34072764

Memphis

-14.7162906

Houston

-15.44072764

UNLV

-16.68240818

New Mexico State

-18.34852576

Ohio

-19.53292952

Army

-20.21403879

Ball State

-20.39575261

Middle Tennessee

-20.40684523

UTEP

-21.6162906

Eastern Michigan

-23.30129892

How’s the Big 10 doing?

Big 10 teams + ND and Nebraska and the amount they’re exceeding or underperforming expectations, in S&P+ units.

Ohio State

27.06871772

Iowa

20.82149088

Illinois

20.64202887

Notre Dame

17.10095289

Wisconsin

12.61369276

Michigan State

10.62313813

Purdue

10.03812981

Penn State

9.070364973

Nebraska

7.785961215

Minnesota

-8.214643349

Northwestern

-8.520189658

Michigan

-11.06906375

Indiana

-14.34072764

How about Michigan?

Michigan is ranked 93, below expectations by about 11 S&P+ units, even when removing the injured players listed above.  That is nearly a full standard deviation (sd = 13.3).  We’re barely outperforming Bowling Green, Baylor and Virginia, and are being narrowly outperformed by Rice, Florida International and Arkansas State.

And our friends at Tulane?

Tulane

-0.988559054

Actually, about what you’d expect, given their dismal experience score.  They're ranked 56th.

Conclusions.

The correlation between experience and defensive performance is poor, which means the coaches should be able to have a great deal of influence over performance.  Michigan is a young team, but plenty of teams that are just as young are outperforming us defensively.  Even if we assume that Michigan’s is the only FBS team with injuries, the performance is still below expectations.  This points to an issue with coaching.

There is the possibility is that Michigan’s experience score is an illusion, and consists mostly of non-scholarship players who will never see the field.  If you know of a data source that has up-to-date rosters and lists which athletes have scholarships, I’d be happy to rerun the analysis, but as it is the numbers make it look like we’re being outcoached, at least defensively, by Tulane.

 

* Since Tulane is an outlier, I reran this analysis without that datapoint.  There was not a substantial impact on the best-fit model.

** Edit 11/29/2010 - When I originally posted this I said "and summing over the whole team".  Actually, the experience scores are only counted for players at defensive positions.

Comments

clarkiefromcanada

November 28th, 2010 at 9:36 PM ^

I think your 4 senior, 3 junior, 2 soph, 1 freshman model for experience and summing over the whole team is a bit overly simplistic. I'd be uncomfortable with any generalization there since, as you rightfully point out, it doesn't account for non scholarship recipients who don't see the field. More problematic, however, would be the lack of accounting for the redshirt year in your calculations. Finally, I think from a statistical model perspective you under represent the distance between a 4 senior and freshman with 1, 2, 3, 4 scoring. I would suggest  this distance between freshman and senior is substantially greater.

Nice effort though; the data just needs clarification and refinement.

bighouseinmate

November 28th, 2010 at 9:52 PM ^

........since one can say, and see the results on the field, that all freshmen are not the same.

Simply stated, a team starting 11 freshmen on defense, with all of them being 5-star players SHOULD outperform a team starting 11 freshmen on defense, with most of them being only 3-star rated players. This isn't a matter of star-gazing but simple fact.

A team like OhioSt., which consists of higher level recruits than UM is pulling in right now, should do better than us, even if they were starting the same age-level of players.

Another point that you fail to address, and I'm not sure how you could, is the impact that depth has on a team's defense. A team with several players at a position, especially one like a DL position, and they have all been in the program for a couple years SHOULD do better than even a team with one multi-year starter, but the backups are all first year players. This is one point that is overlooked when considering the age level of starters. The backups can sometimes be even more important than the starters in that a team that has less dropoff in quality of play when subbing in is better prepared than one that may have an all-american playing but when he is out, the position may as well be vacant.

maizenbluenc

November 29th, 2010 at 8:04 AM ^

OK - but does coaching not effect the level of recruits recruited? Rich and his staff have had 2-1/2 recruiting classes. If we have a young defense, underperforming, with a relative lack of 4 and 5 stars in the young defensive lineup his staff recruited, who's responsible?

Interesting anaylsis. Maybe too narrow in factors considered, but interesting just the same.

Vasav

November 28th, 2010 at 9:58 PM ^

I thought this was really well done, and clearly took a lot of effort. But I do think your numbers miss a a few key points - what the guru ratings are of the players, whether the seniors are starting or not, and where the 5th year seniors/true freshman are playing. A freakish 5* true freshman at free safety could have more of an impact than a fundamentally strong, walk-on 5th year senior at defensive end. And yet the senior DE would appear much stronger than the freshman safety in your model.

That said, even with that info somehow quantified, I think it's still fair to say, as this data shows, this coaching staff has done a subpar job with the personnel we've got.

MGlobules

November 28th, 2010 at 9:59 PM ^

that coaching really is an issue with Michigan; although GERG has had a steep hill to climb and (maybe) been forced to adopt an approach he neither had great insight about or comfort with. . . there just wasn't a great-enough past history of success to merit much confidence in his results here.  

But redshirting clearly could play a big role in results (we might assume that the bigger and better established programs, including those with continuity benefit more from the redshirt process). 

And I would have been more comfortable with your conclusions if you had emphasized the role of not JUST injuries but attrition on results. In the end, I think, we have one too many caveats, and (possibly) too little opportunity to control for the most confounding variables.

Care to spend another day (or so) on this? I'm sure there'll be an appreciative audience.  

MCalibur

November 28th, 2010 at 10:17 PM ^

What does the binned-average trend look like? Basically take the average S&P+ for each bin in your experience score histogram and plot the resulting data.

Age (Freshman, Sophomore, etc) isn’t exactly the same as experience. I think a better indicator for experience would be Total Starts or Years as Starter in my opinion. I’ve done this for QB Progression, so I know how exactly tedious it can be (not that bad), but I think its worth doing if you want to find out real trends (as opposed to supporting a specific, grander, thesis).

Concentration of Youth. I would argue that Michigan’s problem isn’t necessarily youth but how that youth is concentrated. Michigan’s Secondary has 1 player (out of 5) who has any real experience depending on whether you classify Kovacs as a small linebacker or a slower-than-ideal safety. Amongst more traditional DB, Michigan has zero players will game experience prior to this season. That’s a bigger problem than having four young players dispersed though out the all levels of the defense.

Physical development is critical, especially as you get closer to the line of scrimmage. In my opinion that is the bigger concern with playing 18 and 19 year-olds (Freshmen and RS Freshman/Sophomores) against 21 and 22 year-olds. Physical maturity is a big deal in Football and I’d argue that this should be accounted for in your project.

Depth of experience. I agree with the comment that total roster age is not what is interesting, only the experience level of those players that get meaningful PT. The starting line-up is really important, but having adequate replacements matters too. This is something (experience in the two-deep) else I’d add to your experience variable.

Blue in Seattle

November 28th, 2010 at 10:16 PM ^

You state in the beginning that you can't separate coaching from talent.  Yet in your conclusion you ignore this and just state that since experience does not correlate to explain a large portion of performance that the difference is coaching.

Everyone else is mentioning how your data and assumptions can be improved, but the inconsistency of your conclusion given your assumptions is what is most flawed.  All that you can conclude is that if all your assumptions are true, experience only explains 4% of the performance.  But you can't suddenly conclude that everything else is coaching.  Everything else is coaching plus talent AND you have no idea what the split is between talent and coaching.

What would really be excellent is if you attempted to refine your model based on previous analysis done on this blog.  Specifically the decimated defense which is an analysis of both experience and talent level based on the recruiting rankings from high school.  What was missing from that analysis was the connection to a performance metric, e.g. your S&P+

overall I respect the effort and the analysis, it was laid out quite well, and was logical, and clear enough to read and critique,  good job.

 

 

jshclhn

November 28th, 2010 at 10:42 PM ^

I would not be so quick to separate talent and coaching - we are talking about a head coach in his third year with the program, and it is the coaching staff's job to recruit, retain, and develop talent.  

Granted, you do have a point that the coaches aren't the ones on the field playing the game, so you cannot assign 100% of the performance responsibility to the coaches.  There are other factors, but over a long period of time coaching is a huge chunk.

expatriate

November 28th, 2010 at 10:37 PM ^

Recruiting rankings have already been touched on here (the teams that are underperforming may have experience, but they have terrible recruiting classes), so I wanted to address a couple of other issues:

1) Depth is a factor, since M has been acting with very few scholarship defensive players, due to injuries, transfers, and poor recruiting these last few years.  That is a definite factor in a defense's ability to sub in and find that "next man up" when there literally arent enough bodies to bring that "next man up".  That should be accounted for somehow.

2) Do your rosters take defensive starters into account?  I am sure that many of these teams are taking on dead weight with a Doug Dutch type player who never sees the field.  This isn't a problem for M, but it could explain some of the differential for other teams.  For instance, on Bowling Green, if you are a senior you are pretty certainly going to play.  On OSU or Iowa, a hot-shot freshman can be so obscenely good that they take your spot.  This touches a bit into recruiting, but still.

All in all very good analysis, but I feel like its the tip of the iceberg.

Communist Football

November 28th, 2010 at 10:45 PM ^

Many of the earlier commenters have brought up appropriate ways to refine your approach (redshirting, depth, and weighting on starters are good examples -- though I know it would take a ridiculous amount of work for you to compile these), but I appreciate the enormous effort it took you to put this together. I'm certain this will get a lot of well-deserved attention from Brian and the mods.

As much as I have defended RR, I do share Misopogon's opinion that RR deserves a fair share of the blame for the state of the defense: in particular, the DC's lack of authority to run the defense he wants, with the assistants of his choice; and the excessive loyalty to underperforming position coaches (e.g. Hopson and Gibson). Let's hope that, if RR is retained, DB requires him to clean this up.

Michigan4Life

November 29th, 2010 at 1:20 PM ^

with regards to defense is the forcing of running 3-3-5 scheme.  DC has never had to pick their own assistants in any level.  The HC pick the position coaches.  It's up to the DC to communicate with the position coaches on what he want them to do.  Of course, losing out Witty, Dorsey and to name a few are on RR with regards to knowing the academic eligiblity.

colin

November 28th, 2010 at 11:35 PM ^

i'm a little surprised that, even given your caveats, that the effect is so small.  is this because most teams are more or less the same age?  how bad would a team full of freshman be (or a team of all seniors)?  if everyone is very close in age, then the redshirt issue would be very distorting right?

ebv

November 29th, 2010 at 8:53 AM ^

Thanks for the comments everyone.  I'm going to do offense next, and I'll try to take your feedback into consideration as much as possible.

tpilews

December 2nd, 2010 at 11:06 AM ^

I think there needs to be an emphasis on the starters and not the entire roster. You could double, or triple the point value assigned to starters. So a freshman is worth 2 or 3 points and a senior is worth 8 or 12. Doing so would also place a premium on having experienced starters. I like what you've done and appreciate all the work you put in. Very interesting diary. +1 to you for the effort. I look forward to seeing any revisions.

Bluestreak

December 6th, 2010 at 1:58 AM ^

We faced 3 teams which are ranked in the top 10 in the eventual standings. Is this factored into that we were facing much harder opponents than some of the others you listed.

 

That said, my biggest grudge with the current regime is that being Michigan - things shouldn't be so bad in any case - we should have a steady flow of talent at all times. I don't know who is responsible or what happened - it just shouldn't be this bad.

 

P.S. scrap the 3-3-5 and play a scheme which is likely to work in the Big 10. Even my 5 year old nephew says that.