The r^2 is what it is. If you are surprised by how much of the variance it accounts for, that's fine, but that doesn't make it any bigger. Even if we accept the generous 5% number and assume that all the correlation is due to causality (clearly a ridiculous assumption), that still means that 95% of the offensive performance is due to factors other the the interior offensive line's experience.
Offensive Line Experience: Short Ride in a Broken-Down Machine
A moment comes when you first start listening to minimalist music—for some people it comes quickly, for some people it never clicks at all—when your perception of time changes. As a musician famously described his first exposure to a Philip Glass opera; his initial boredom was transformed as...
I began to perceive...a whole world where change happens so slowly and carefully that each new harmony or rhythmic addition or subtraction seemed monumental...
...he said as the rhythmic woodblock...no, it's Adams not Glass...the woodblock crack of the pulling Stanford guard's pads as he thumped the Oregon SAM out of the hole play after play after play after...
NO! I will NOT spend my Thursday evening in an altered state of consciousness. So I started using the media timeouts, and then the time between plays (well, at least when Stanford had the ball, which thankfully was just about always) to work on a project I'd started a few days earlier during the Gameboy diaries, pulling participation reports for all 125 FBS teams and pulling roster/bio information to get the classes of their starters on the o-line.
And some of you people think huddles serve no purpose.
Honestly, the Horse Wasn't Dead When I Started
The results are here, usefully tabled in a spreadsheet to save some work for the next sap that starts on one of these projects.
Of course, as I sat down at my computer to do some regression analysis on the data I opened the blog and saw Gandalf's diary covering most of what I was planning to do (and doing a better job of it I might add). But I was taking a slightly different tack and found a couple of wrinkles, so for the sake of the eight of you that are still interested I'll continue on....
First a couple of comments about the dataset (feel free to skip the rest of this section, but it might be important if anyone uses the data for further analysis). Gandalf took his data from depth charts at the ourlads.com scouting site; mine come from the starting lineup listed in each school's participation report in the official game stats for their most recent game against FBS competition (sometimes coaches play with their lineup for games they're treating as exhibitions, give a start to a loyal walk-on for example, so if the most recent game was against a Delaware State I pulled the lineup for the week prior).
The official reports have the virtue, or defect, of being precise accounts of who was on the field. Sometimes that was a problem because everyone doesn't actually use five offensive linemen all the time. Idaho started a game with four, presumably spreading the field with covered, ineligible tight ends and wide receivers. Somebody else came out heavy and listed six. There were also some schools that simply listed their linemen as “OL” without assigning specific positions.
Where possible I straightened those situations out by using the schools' published depth charts. When that didn't work either I looked at third-party depth charts and did my best to reconcile them with the actual starters. It's possible there are a couple of players out of position here, but I don't think it's material.
For teams, usually pistol teams, that flop their line, I assumed the tight end would line up to the right and assigned the quick tackle and guard to the left side and the strong tackle and guard to the right.
For obvious reasons, service academies don't redshirt players. If an academy lineman's bio showed a year in which he didn't see game action, I counted that year as a redshirt and subtracted the year from his class. The point after all was to look at experience, not remaining eligibility.
Additive and Multiplicative Measures of Experience
My starting point was two proposals in the Gameboy diaries. Gameboy himself proposed assigning a value to each player (one point for each year, half a point for a redshirt) and adding them (well, averaging them, which of course is the same thing but for scale). That average appears in the spreadsheet as the GLEM (Gameboy Line Experience Metric).
In a comment to one of the diaries reshp1 suggested an alternative: assigning a value to each player based on experience (conceived as the probability that the player in question will successfully carry out his assignment) and multiplying those values and subtracting the product from one to get the probability that an assignment will be busted on a given play. That probability appears in the spreadsheet as the RBI (Reshp Bust Index). It's basically the weakest-link theory with the additional recognition that anyone might turn out to be the weakest link on a given play.
I focused on the latter metric because conceptually it makes sense to me and because it wasn't treated in Gandalf's diary. Reshp1 pulled the probabilities out of the air, or his hat, or somewhere, but the analysis doesn't seem to be sensitive to the particular choices here. The values are in a lookup table on page 2 of the spreadsheet if anyone wants to play around with alternatives.
Before I go on, a sanity check on Reshp1's metric—a list of the ten youngest lines:
- UCLA (7-2, 4-2)
- Idaho (1-9)
- California (1-9, 0-7)
- Wake Forest (4-6, 2-5)
- Eastern Michigan (2-8, 1-5)
- Western Kentucky (6-4, 2-3)
- Tulane (6-4, 4-2)
- Maryland (5-4, 1-4)
- Arkansas (3-7, 0-6)
- Michigan (6-3, 2-3)
Not a list you want to be on; those are some bad teams right there, combining for a 16-37 record in their respective conferences and that's flattering because it leaves out independent Idaho, who's probably the worst of the lot. (You can point to UCLA if you like as proof that, if everything goes right, you can survive starting multiple freshmen. Arkansas fans are probably pointing to Michigan and saying the same thing.)
The Running Game
Sanity check #2 is to redo Gandalf's work, but with Reshp's metric. Here's a graph of yards per carry vs. RBI:
That looks familiar. R2 is .058; the correlation coefficient is -.24 (these coefficients will all be negative because RBI is smaller for more experienced lines). And if we strip out the tackles and just look at the interior?
R2 is .084, the correlation coefficient is -.29, and it's not a coincidence that this looks an awful lot like Gandalf's chart using “youngest interior lineman”.
Weakest link, check. Experience matters more on the interior than at the tackles, check.
But what I really wanted to do was to look at the impact of o-line experience on an offense as a whole. To do that I've used the offensive component of the Fremeau Efficiency Index, which looks at all offensive drives (except for clock-kills and garbage-time drives) and compares the results to expectations based on the starting field position. By its nature it's pace-adjusted and independent of the effect of the team's defense; they also apply a strength of schedule adjustment.
Here's the chart:
R2 is .026, the correlation coefficient is –.16. The effect’s not as large, but a young line impacts the whole offense, not just the run game.
It made some sense that in the running game experience would matter more in the interior than at the tackles since it's an interior lineman that makes the line calls and the assignments tend to be more complicated inside. It wasn't so clear that this would still hold when the passing game was added in:
but that's what we find. The correlation is greater when we only look at the interior. R2 is .048, the correlation coefficient is -.22.
It's on the interior that experience really matters. And Michigan's interior RBI ranks 123rd of 125 FBS teams.
How Large an Effect?
A lot was made in Gandalf's diary, and especially in the comments, about the low R2 values here, which were seen as a demonstration of the relative unimportance of experience vs. other factors, like coaching.
I see it differently. This is an extremely diverse universe of teams we're looking at here. There are differences between Michigan and Eastern, or between Ohio State and Ohio U., that can't ever be overcome by something as simple as inexperience on the line. A lot of the scatter in these charts is just a matter of big programs being big and small programs being small. Given those enormous differences in baseline levels of the various FBS teams it's amazing to me that we could see anything like 5-8% of a performance difference being credited to any one team demographic, especially when the difference is measured using an SOS-adjusted metric like Fremeau.
And the slopes of these trend lines aren't small. The expected oFEI difference between 2012 Michigan and 2013 Michigan is .32; the actual difference is .197. The expectation, just correcting last year's performance for the youth on the field this year, was for a worse offense than we've actually seen.
Put another way, if you use that trend line to adjust for this year's lack of experience, add the missing .32, Michigan's offense goes to 19th in the nation, right behind Stanford and Louisville. UCLA turns into Oregon. Eastern becomes Bowling Green and maybe English keeps his job. Everybody's happy.
Good Teams are All Alike, Every Bad Team is Bad in its Own Way
I thought I'd try to get a handle on that by comparing each team's performance to the baseline they've established historically. I've averaged the oFEI's for each program for the five-year period from 2008-2012, then calculated the deviation of this year's performance from that average.
Basically, we're now looking at year-to-year deviations in performance within each program.
On the one hand, this gets rid of the scatter due to the vast discrepancy in baseline performance expectations from the top to the bottom of the division.
On the other hand, this also filters out any effect from programs like Wisconsin whose strength largely comes from the fact that they always field powerful, experienced lines. There's not much year-to-year variance there—they're always old, always good.
So it's possible we won't see any bigger correlation here than before...
...what happened? R2 is .009. Two-thirds of the effect is now gone. (A result, by the way, that's consistent no matter what metrics I use for line experience.) Apparently, only a third of the effect we’re looking at is a matter of one-off bad seasons due to a young line; most of the effect is systematic, inherent in particular programs. It's almost as if there were a correlation between poor past performance and current youth, and that's because there is:
There's the missing two-thirds. Historically (well, over the last five years anyway) bad teams are on the left, good programs on the right. There's less current youth (lower Bust Index) as you move right.
A look back at the teams listed earlier provides a clue. It's a mix of historically bad programs like Eastern, struggling FCS converts like Idaho, and programs that have suffered some sort of recent calamity, the kind that makes you decide to hire John L. Smith to be your substitute teacher for a year. Some had horrible recruiting, some had retention problems…each one has had its peculiar issues but every one of them is a program in disarray—some recovering, some not. Teams don’t field multiple freshmen because they want to; they do it because things fell apart.
We'll know more if someone does the study suggested in the comments to Gandalf's diary, looking at overall roster depth instead of just the age of the starters, but I think what's happening here is that the Wisconsin effect is the dominant effect in the study. Good programs don't suffer from youth on their lines because (a) it doesn't happen to them and (b) when it does, it's not a sign of weakness. When Andrus Peat finds his way to the top of the depth chart as a sophomore it's because he's beaten out multiple upperclassmen and won the position. When Kyle Bosch find his way to the top of the depth chart it's by default; the juniors and seniors he's supposed to be competing against aren't on the roster.
I think the next thing I might try, if I were of a mind to keep flogging this, is to do something so straightforward and blunt as to look for a correlation between offensive efficiency and the number of scholarship upperclass o-linemen on a roster (more telling than the percentage, I would guess).
"95% of the offensive performance is due to factors other the the interior offensive line's experience"
Not quite--it means that 95% of the variance among teams is due to factors other than the interior offensive line's experience.
That's no surprise. Most of the variance is the enormous gap between top-end and bottom-end programs. Alabama's worst team in the database was #16 out of 120. New Mexico State's best was still in the bottom 25. There's nothing either school could possibly do to match the other, and discrepancies like that dominate any variance analysis.
To see what I mean, do an analysis on a more homogenous group of teams, like a single conference. The R2 will be 3-5 times bigger...and the variance is still dominated by gaps like OSU/Purdue or Bama/Kentucky that don't have anything to do with temporary conditions in either program.
What you're saying is that if you controlled for other variables, the r^2 for offensive line experience would go up. Which is a perfectly reasonable hypothesis, but not shown by any of your analysis. As a reader, I can only judge by the numbers you present me with. If you can control for other variables and get the r^2 for interior offensive line experience to quadruple, that would be both interesting and compelling; but just asserting that it's so is neither. It seems as if you did a lot of work to get this data, but then you hand-wave to get the conclusion you want.
If sure if Yeoman did the analysis on similar teams you'd then complain about sample size. Pity for you that this post didn't support your thesis.
Let's do the B1G since that's close to home. oFEI vs. int RBI.
R2 was .05 for all of FBS; for the B1G it's 0.16.
That's still not a homogenous group of programs of course--they've been playing football for a century and I don't think there's ever been a 5-year stretch where Indiana was at the level of OSU and Michigan (they might have occasionally caught up with one during an especially dire stretch, but they've never been ahead of both). I don't know how to find a sample that doesn't suffer from that effect that wouldn't run into serious sample size issues. Maybe have a fairly small group of similar-level programs and look at them over multiple years?
Suggestion: build a sample out of recruiting data. Take the teams that have appeared at least X amount of times in Rivals' Top 25 recruiting classes since 2002 and use just those teams in your study. You'll end up with some variance (Alabama and Illinois will both be in there) but not so much as that between Bama and Illinois. And then at least you're working with a sample of athletes who are relatively close to each other in expectation.
Whether or not our guys are winning jobs or forced into action is kind of irrelevant at this point. This season (which I think showed a lot of promise if this one area of the squad could have been fixed) is basically a lost cause. The issue now is what to expect going forward. Year 3 always is going to be strongly effected by what was left behind, but years 4 and 5 are all on the new guy. What is our team going to look like?
In Jim Harbaugh's 3rd year at Stanford (offense was very good, shitty D left them at 8-5), he had two RS freshmen starting on the o-line (Martin and DeCastro). Those guys were both HM all-conference and freshman all-Americans. He'd also signed Andrew Luck (played well as a RS freshman) and Chris Owusu in that very first full class he brought in.
In Nick Saban's third year at Alabama, he won a national title with a RS freshman from his first recruiting class (Barrett Jones), a JUCO transfer from his second, and a RS sophomore starting on the o-line. By this time Julio Jones, Trent Richardson, Eddie Lacy, AJ McCarron and probably a bunch of other future stars were already on campus as well.
Pete Carroll won a national title in 2004 (year 4) with a true sophomore and a RS freshman starting on the offensive line (Ryan Kalil and Sam Baker, both from his 2nd full recruiting class). Those two classes also brought in Mike Williams, Reggie Bush, and LenDale White at the skill positions.
Right now none of the guys from the 2012 o-line class at Michigan looks like a future star. Kalis got benched and Magnuson is struggling (probably) playing out of position. There is no Martin/DeCastro/Jones/Kalil/Baker in that class. That class also didn't contain a QB. And the only RB we took was Drake Johnson. The one star on offense (Funchess) spent a year and a half trying to block bigger guys instead of catching passes.
Pretty much all our eggs are in the Shane Morris and Derrick Green baskets at QB and RB. And any star power up front will almost certainly have to come from the 2013 o-line class. A lot of things have to go right real soon for this team to improve after absorbing the losses of Lewan/Schofield/Gallon/Fitz and then Devin. Just saying "Wait for the recruits" seems like wishful thinking at this point, unless we're resigned to being a notch below teams like Alabama, USC, and even Stanford.
I don't think anything is irrelevant. The data show that Michigan's struggles are close to the norm to be expected with what they have in their line men's room. That isn't to excuse coaches. It just gives us an idea of why we suck at blocking, not the total picture.
As far as next year, we will be better. At least 4 of our 5 starters will have some in game experience, and if not, at least they will have been beaten out by younger guys. This will go a long way. We probably won't be world beaters, but we wont be a negative play factory either.
I ask Yeoman to highlight the teams that are this year starting something close to what we will be starting next year (RsSo, So, Jr, RsSo, RsSo). It would be interesting to see where they fall on the scatterplot.
We'll fine. Although I think the OP is very good, and details a lot Michigan's issues, I think a follow up analysis would be very interesting. Meaning, what happens developmentally to a young group of OLinemen that are pressed into service "too soon" due to depth issues. To me, it would seem that the "trial by fire" of playing early (too early) will, in the long run, pay off with a highly experienced OL in the near future.
Lost their two starting guards due to injuries and were having try outs for walkons. They ran fairly well against us.
A lot of it is coming down to coaching and S&C, or else our recruits were way overrated?
According to my coworker you're actually only supposed to use linear regression when you have at least five independent variables as a part of your model. The correlation numbers are more accurate to look at in this case since correlation is supposed to show how related two variables are to each other. If you want to show causation there is another model I'd have to look up, Unitarian or something, that could be used to show how much one independent variable is the cause of one dependent variable.
I'm sorry, nothing you've said here is remotely accurate.
RS Frosh starters on 1997 line are proof you CAN win with the right linemen and right coaching.
I don't doubt that younger is NOT better in most cases, but Hoke and staff had two off-season years (starting with afterglow of the 2011 season) to get ready for THIS season, knowing that guys like Barnum, Omameh, Mealer (mediocre at best but at least upperclassmen) would be gone and needed to be replaced for kickoff vs. CMU. How do you get guys ready with that much lead-time? COACHING!
I'd say if we can't run the ball, and our TB's can't block, then put in our best receiving RBs (Norfleet, Hayes,etc.) and try to through the ball 70% of the time. But please have our coaching staff QUIT PRETENDING it can run the damn ball. Five guys out in patterns means Devin only has to worry about 6 guys rushing.
It's been two many years since I had that Stats class at U of M, but wonder how the chart would look if we viewed efficiency vs. combination of experience and recruiting star ratings.
In that world, we'd been FCS level at best when you account for how poorly highly recruited guys have done.
It is possible to win with youth on the line, as your Backus/Hutchknson example shows. I think it helps if those two players are good enough the eventually be drafted in the top 20, play ten years in the NFL, and be arguably the best guard to ever play football. The point is that it is rare. The data show this.
Your complaints about coaching are valid, though. I guess the argument is that, as this season has shown, we aren't really pretending to be a downhill running team. We started the season running the stretch primarily, which is not really a downhill play. Lately, we have all but abandoned the run, and the results have been worse because we cannot pass protect. At least we scored when we were going 27-for-27™.
I personally don't think all of our woes are due to youth, but I do think it has played a part. In order for a lineman to be successful he needs several things:
1. Physical tools - strength/size/agility
3. understanding of the offense
All three of those are things that are likely to increase with time in program. Yes, there will be physically mature kids who either came in with good techniqu or are quick learners and can learn both the technique and the playbook quickly, but it shouldn't be a shock that many can't and need time.
Even if someone could statistically conclude something along the lines of 33 percent of 4/5 star lineman who weigh 300 lbs or more by their redshirt sophomore year are rrady to play we still couldn't statistically look at a coach wiho couldn't turn 1 out of 3 such players into adequate starters as a good or bad coach because such a small sample size is meaningless.
Hutchison and Backus are generational players,
Hutchison is a first ballot Hall of Famer.
I'm starting to feel like during the RR years we had all these fancy charts that made us feel better about the future. I just want to see this team steamroll their schedule next year, go 11-1 or better (as their talent indicates they should), get to the Big Ten title game, and get to the freakin' Rose Bowl and set up two monstrous years that are the Shane Morris era. I love the work you did on these charts, but at this point if you need a chart like this to convince you that this offensive line is massively and epically young you are deluded.
but conclusions cannot be made by any of those statistics. We live in a multivariate world - there are suppressors, variable interactions, spurious effects, etc. not to mention the reliability of the independent and dependent variables.
I don't think Yeoman's arguing that experience is the only factor that governs offensive success, just that it plays a small role. Do you think this is wrong? And if so, do you have any multivariate data that lacks suppressors, has only invariable interactions, nonspurious effects, and demonstrably reliable independent and dependent variables to back up this assertion? Obviously if there are specific errors or biases those should be corrected, but just calling it useless seems a bit harsh and unproductive.
sure why some here can't concede that the interior OL youth is a huge problem. Borges is expected to scheme around it but is not credited for having done so in the games the Wolverines put up all kinds of points. Add to that a first year starter QB who was spooked by his own turnovers before all hell broke loose with the OL and you have what we have.
Mattison said they are a couple of plays away from being a 1 loss team. Yeah, it would be smoke and mirrors to a degree but let's quit pretending that we have the most inept staff in football.
I'm all ears. What kind if plays can one call that do not rely heavily on blocking? Really, what are they? If you can name 2 plays of such a nature, I will want to Fire Borges. I still think an offense based on two plays would suck, but I'll go for it. Two plays.
Pretty much any screen is less reliant on blocking then most plays, especially bubble screens as they stay away from the interior of the defense. Anything where you roll the QB out also helps a little, moving the pocket around can minimize blocking problems, as can quick throws.
I mean, look, it's not entirely on Borges, but to say "there are no plays which minimize the importance of blocking" is....just fucking dumb. Now, can Michigan run those plays? No, not even with the minimized emphasis on blocks. Part of that is still on the coaches though, because it's their job to get players ready, and they haven't done that.
I think even the most ardent supporters of our coaching staff realize that Funk needs to go, and at the very least Hoke should take a long look at whether Borges gives this team the best chance to compete for conference titles in the future. It's not just about drawing plays up that work on a whiteboard, it's about getting players to run them well, and this offensive staff has done a piss poor job of preparing the players for gameday, and has made just enough boneheaded playcalling decisions to get people to complain about them.
But then the interior of the line isn't gonna know what to do next year as well
Trade a mediocre year this year for several years of great is the coaches motto it seems.
Interesting stuff. I like the comparison between the deviation from baseline efficiency and experience/inexperience. I'm surprised that cuts the r-squared down so much. Also, I'm glad your basic look at experience and YPC produced the same results! That could have been embarrassing...
But point Brian.
Please help me understand what next year will be about when Lewan and Schofield depart????
It's possible Bryant gets healthy and comes on, or Miller puts on some strength and wins the job back, but let's assume it's Braden-Kalis-Glasgow-Bosch-Magnuson.
So depending on which of the two theories you buy:
(1) the Bust Index for the entire line will improve from 75% to 65%, which would improve oFEI by about .06 and move us (all else being equal which of course it isn't) up about ten spots, or
(2) the Bust index for the interior will improve from 69% to 46%, which would improve oFEI by about .175 and move us up about about 20 spots.
Even if you don't buy into the part about experience not mattering at the tackles, there's every reason to expect the line to be better next year. Lewan's a major talent of course, but there'll only be one guy on the line in less than his third year in the program and he'll be a returning starter. If there's another, besides Bosch, he'll have won the job instead of getting it by default.
The roster hole won't really be healed until 2015 but this year was the low point.
Analysis aside (which I enjoyed thoroughly), some other points to consider…The impact of teams transitioning from a west-coast/pro-style offense to a read option/spread then back to a pro-style offense would seem to have more of a pronounced effect on unit cohesiveness and unit chemistry than would be revealed by a simple analysis of the individual parts and their experience level correlated to performance. So if team chemistry and familiarity with a defined offensive system has more influence on future success of the offense, then we should see a greater improvement in the offensive unit once the parts settle into the defined system and get to play together more. This would also indicate that as the players from the RR era leave and hybrid offenses disappear then much higher levels of achievement can be expected than the pure correlative experience data would suggest. Does that make any sense to anyone?