Texas blog Barking Carnival takes a look at the numbers and finds there is some relationship between switching to a fast-paced no huddle offense and a decline in your defensive efficiency numbers that year:
(Note this isn't saying you can't have a good defense running HUNH, but that when you switch to it, you should expect a defensive decline that season)
Setting the Scene
no animals were harmed in the making of this diary
Apology. I am sorry to all those who are cringing at seeing another experience diary. I had originally conceived of this as being a two part study, with the first looking at the running game and the second looking at the passing game. Despite the other diaries, both of which were useful in their own ways, I think there is still some horse meat to be gleaned from the carcass of o-line experience as it relates to pass protection. The horse might be dead, but that doesn’t mean it’s useless. Oh, and there's another one since I originally wrote this thing up.
Previous Work. In the first diary, I attempted to demonstrate that o-line experience does indeed play a role in governing a team’s ability to run the ball. R-squared values ranged between about 0.05 and 0.10 depending on how we defined “experience,” suggesting that about 5-10% of the variation in YPC across FBS can be explained by a team’s experience along the o-line. Clearly other factors are also important to running the ball well, but experience isn’t meaningless. Further manipulations to the data set suggested that interior line experience is more important than tackle experience and that the “your line is only as good as its weakest link” argument does hold water.
Some of the comments from that diary questioned why we don’t move one or both of our experienced and talented tackles to the interior if that is where it really seems to matter. Transaction costs of moving around linemen aside, the question is valid in general terms. Why not put your best linemen on the inside if those are the most important positions? A variety of answers could be given to this question – for example, exterior and interior line positions could have different ideal body types with regard to height, weight, strength, and agility – but the most obvious response is that tackles are more influential in the passing game. Our best linemen play at tackle because they protect our quarterback.
Questions. This is a proposition that can be tested statistically, and that’s what I aim to do in the second part of this study. My metric for o-line success in the passing game is sack percentage (i.e., the percentage of pass attempts on which your QB is sacked), since it’s the o-line’s job to keep the QB clean. Using the same essential methodology as the last study, I aim to answer four questions:
- Does o-line experience help prevent sacks?
- Is tackle experience or interior line experience more important in protecting the quarterback?
- Does the “weakest link” theorem hold for the passing game?
- What else could influence sack percentage?
Data. This study looks at 123 FBS teams (Georgia State and UTSA are omitted since their info wasn’t on ESPN). Sack percentage stats come from ESPN, and the experience data comes from the scouting site Ourlads. Star rankings that come at the end of this study are taken from Rivals. Photos come from MGoBlog's flickr account and are attributable to Bryan Fuller. Check out the previous diary for basic definitions of the statistics that are used.
This is long, so buckle up. Feel free to jump to the conclusions if you don’t want the nitty gritty. All the data are summarized in a…chart? Chart.
Questions and Answers
Probably cropped out: massive amounts of backside pressure
Question 1: Does o-line experience help prevent sacks?
Let’s start by taking a broad look using average experience in years of the offensive line. The relationship between experience and sack percentage is plotted below. Click on the graph to see the same sack percentage data plotted against total number of starts. The plot is oriented so that up is good (i.e., your QB isn't getting sacked that often) and down is bad (i.e., you're looking like Michigan against MSU and Nebraska).
Although the trend line makes it look as though there is an inverse relationship between sack percentage and experience (i.e., sacks go down as experience goes up), which is what we’d expect, the r-squared is relatively low (0.02) and the p-value (0.10) suggests the trend may not be statistically significant. If we plot the same relationship but use starts as our metric for experience, the relationship becomes even more spurious with an r-squared of 0.01 and a p-value of 0.28 (click on above graph to see scatter plot). On the whole, total or average o-line experience doesn’t seem to be a great predictor of the o-line’s ability to keep the quarterback from getting sacked.
Question 2: Is tackle experience or interior line experience more important in protecting the quarterback?
I don't care who matters more as long as we keep it under 7 sacks a game
We saw in the previous study that interior line experience was more important to run game success than tackle experience. We’d likely expect the opposite to be true for the passing game based on the premium put on left tackles both in college and in the pros. Average tackle experience and sack percentage is plotted below.
This is unexpected. The correlation is spurious. The r-squared is less than 0.01, and the p-value is 0.67, both of which suggest there is no correlation between tackle experience and sack percentage. The trend line actually rises slightly, which would indicate that sack percentage rises as tackles get more experienced, which makes no sense at all, even if the correlation was statistically significant. On the whole, tackle experience does not appear to be a good predictor of your team’s ability to not give up sacks.
Could the interior of the line be more important in the passing game as well? Click for enlarged scatterplot with teams divided by conference and all BCS teams labeled.
Now we’re getting somewhere. The trend line makes intuitive sense. The percentage of sacks you give up goes down as your interior linemen become more experienced. The r-squared is 0.05, implying interior line experience can explain about 5% of the variance in sack percentage. The p-value is 0.02 suggesting that results are statistically significant. The slope of the trend line suggests that an extra year of average interior line experience is worth a drop of almost 1 percent in sack percentage. If you extrapolate that over the course of a season, that’s about 3 to 5 fewer sacks. Not a huge difference, but if your team matures over the course of several years from starting freshman to starting seniors, that adds up to reducing sacks by about 1 per game.
One possible critique here is that “average” tackle experience is not the correct measure. Teams often put one of their better run blockers at right tackle and their best pass protector at left tackle. Thus instead of looking at the average, we should just look at the correlation between left tackle experience and sack percentage.
We don’t even need a graph here. We get almost the exact same trend as when the tackle experience data are averaged, and the slope of the trend line suggests that sack percentage slight increases as tackles get older. That is intuitively incorrect. A low r-squared value (<0.01) suggests left tackle experience doesn’t matter very much and a high p-value (0.31) implies statistical insignificance. This is admittedly somewhat baffling and definitely unexpected.
Question 3: Does the “weakest link” theorem hold for the passing game?
weakest link only in age, not awesomeness
When looking at the run game, the data suggested that the youngest member of the interior line was a better predictor of success than average experience of the interior line. In the passing game, the “weakest link” is a little less weak.
Unlike with the run game, average interior line experience appears to serve as a better predictor of sack percentage than does the “weakest link” along the interior of the line. The r-squared here is 0.02 and the p-value is 0.12, suggesting that the significance is marginal at best. It's not that the weakest link is a terrible predictor, just that the average experience of the entire interior line serves as a slightly better indicator of sack percentage.
At this point we can draw some basic conclusions from the first three questions. Total or average o-line experience only seems to be a marginal predictor of a team’s ability to keep their quarterback from getting sacked. Tackle experience, whether averaged or just taken as the left tackle, appears to have no relationship whatsoever with sack percentage. Just like with the run game, interior line experience seems to be the most salient characteristic with regard to o-line experience for predicting success in pass protection.
Question 4: What else could influence sack percentage?
One of the main critiques from the last study was that we’re living in a multivariate world and other potentially influential factors should be included in the analysis. I’m still working on getting myself up to speed regarding multivariate analysis, so I’m tentative to try and do too much with that now. We can, however, look to see how some other variables correlate with sack percentage.
Offensive Line Talent
4 star, 4 star, 5 star...sack?
Talent is one obvious potential factor in governing pass protection success. The chart below shows the correlation statistics for the star rankings of the o-line with regard to sack percentage. I’ve omitted the graphs here because none of these produce any correlation of statistical significance.
|Average O-Line Star Ranking||0.01||0.30||Low|
|Average Tackle Star Ranking||<0.01||0.52||Low|
|Average Interior Line Star Ranking||0.01||0.22||Low|
|Left Tackle Star Ranking||0.01||0.32||Low|
Surprisingly, star ranking of offensive linemen doesn’t seem to correlate very strongly with sack percentage. My guess is that this is due to star ranking of offensive linemen correlating closely with the difficulty of defense that a given team plays. For example, Alabama has talented linemen, but they play against tough defenses in the SEC. Toledo, on the other hand, has crappy linemen, but they play against week defenses in the MAC. Moreover, there tends to be very little variation in star rankings with non-BCS schools – almost everyone is a 2 star – so this may be obscuring some of the impact that talent (i.e., star rating) has on pass protection.
This can be accounted for, to some extent, by looking at only the BCS schools, where there will be more variation among offensive line star ratings and more consistency in the level of teams played. The chart below shows the correlation statistics for offensive line talent and pass protection success.
click to zoom with all BCS teams labeled
|Average O-Line Star Ranking||0.03||0.18||Low|
|Average Tackle Star Ranking||<0.01||0.64||Low|
|Average Interior Line Star Ranking||0.05||0.07||Marginal|
|Left Tackle Star Ranking||<0.01||0.87||Low|
Once again, it appears as though the interior of the line is the most crucial for preventing sacks. This corresponds well with the experience data presented in the first three questions. If we’re operating under the hypothesis that the interior of the line is more important than the tackles with regard to pass protection, which the experience data suggest, then we’d expect talent to matter more on the interior than it does at the tackles as well. It turns out that this is exactly the result we get. Whether looking at experience or talent, the interior seems to be the key to success.
I don’t know much about multivariate regression, but when you take both experience and talent of the interior of the offensive line into account for predicting sack percentage, an r-squared of 0.09 is produced. This is almost double the r-squareds produced by regressing experience and sack percentage and talent and sack percentage, and it suggests that these two factors work in tandem to determine the success of the offensive line regarding pass protection.
Unleashing the Dragon
A team’s tendency to throw deep, thus necessitating a longer drop and more time in the pocket, could be another influential factor governing sack percentage. I thought that yards per completion would be the best measure of a team’s tendency to throw deep, since yards per attempt could be equally as high for teams that throw quick, short passes, but complete a high percentage of them. Either way, we can look at both metrics.
click to see scatterplot of ypa and sack percentage
|Passing Depth Metric||R-Squared||P-Value||Significance?|
|Yards per completion||0.01||0.31||Low|
|Yards per attempt||0.01||0.20||Low|
These measures do not correlate particularly well with sack percentage. Yards per completion gives us the trend we’d expect – that sacks go up as yards per completion go up, but the explanatory value is weak as the p-value suggests insignificance. When doing a multivariate regression with yards per completion and interior line experience against sack percentage, the r-squared only rises to 0.056 from 0.05. It doesn’t add much explanatory value. Using YPA instead of yards per completion actually produces a trend where it appears that increased yards per attempt facilitate a decrease in sack percentage. That doesn’t make a lot of sense and the correlation and statistically insignificant anyway.
4 star talents, 5 star smiles
A third possible factor governing sack percentage is the skill of the quarterback. Perhaps sacks are less a matter of how good the line is and more a matter of how good the QB is. To measure this I look at each BCS quarterback’s star rating and their Rivals rating (4.9-6.1) to see whether their high school talent correlates with how often they end up sacked. You can’t really use any college stats as a measure of their talent, because those can be directly influenced by the play of the offensive line, and we’re trying to isolate QB talent as a separate and independent variable here.
click to zoom
There’s really not much here. Whether you go by star ranking (2-5 stars) or by the more precise Rivals rating (4.9 – 6.1), there’s no significant relationship between a quarterback’s talent and his ability to remain upright. R-squareds for both metrics are <0.01, and p-values are 0.75 – 0.80. On an individual level, the skill of a single quarterback might help him avoid sacks, but taken broadly across all BCS schools, quarterback talent doesn’t seem to be a factor.
all 10 linemen on FSU's 2-deep are upperclassmen /Miami Herald
Probably the most common critique in the previous diary was that depth should be taken into account. You can do this different ways: average or total experience on the 2-deep, the oldest player at each position on the 2-deep, or the percentage of upperclassmen on the 2-deep. For this study I'm using the last of these definitions, the percentage of upperclassmen. I'm defining "upperclass" as students who have been on campus for two years prior to this season. So redshirt sophomores and true juniors are both considered "upperclassmen," while true sophomores are not. The graphs below show the trends for both the line as a whole and the interior of the line.
click to see all teams labeled - Duke also has an all-upperclass 2-deep
The correlation is unexpectedly poor. The graphs above show line depth both across the entire o-line and just the interior of the line. In both cases, the trend line suggests that the more upperclassmen you have, the more sacks you give up. This doesn't pass the common sense test, and r-squareds for both are low (0.01) and p-values are high (>0.30) implying that the correlation is not statistically significant. It doesn't appear to be a matter of defining "upperclassmen" either. If you run the same regression using average depth on the line, you get the same spurious results. While line depth might be an excuse for any given team, across the entire FBS the experience of your starters seems to matter much more than the experience on the entire depth chart.
Modest but significant. Despite using a completely different metric for o-line success, sack percentage instead of YPC, the conclusions of this study are eerily similar to the previous one. Let’s begin with the (hopefully) obvious caveats. Offensive line experience explains a modest, though significant, amount of the variation in sack percentage across all FBS schools. We’re talking about 5% of the variation here, so there are clearly a lot of other factors that go into determining how good a team is at protecting their quarterback.
In a way, this study is much less about Michigan than it is about college football in general. The success or failure in pass protection for a single team can often be explained by factors that are specific to that team. For instance, Devin Gardner is essentially the Ben Roethlisberger of college football, refusing to throw the ball away or to be tackled until approximately half the other teams defenders are draped all over him. This certainly contributes to Michigan’s high percentage of sacks, but it is a difficult variable to account for and measure across all of college football.
That being said, offensive line experience does stand out as a particularly salient characteristic for explaining a team’s sack percentage. Although we’d assume that experience at the tackle positions would be more important in the passing game, the results of this study suggest that once again the interior of the line is what matters most. In contrast to the previous study, the “weakest link” (i.e., the youngest interior linemen) is not as good of a predictor as the average experience of the interior of the line.
Taking a comparative approach by looking at experience alongside other potentially influential factors provides some context for how important experience actually is. The chart below plots each of the metrics I’ve looked at in this study along with their r-squared and p-values.
|Independent Variable||Unit of Measurement||Data Set||R-Squared||P-Value||Significance?|
|Avg. O-Line Experience||Years||FBS||0.02||0.10||Marginal|
|Total O-Line Experience||Starts||FBS||0.01||0.28||Low|
|Avg. Tackle Experience||Years||FBS||0.01||0.67||Low|
|Avg. Interior Line Exp.||Years||FBS||0.05||0.02||High|
|Left Tackle Experience||Years||FBS||<0.01||0.31||Low|
|Average O-Line Talent||Rivals Stars||FBS||0.01||0.30||Low|
|Average Tackle Talent||Rivals Stars||FBS||<0.01||0.52||Low|
|Avg. Interior Line Talent||Rivals Stars||FBS||0.01||0.22||Low|
|Left Tackle Talent||Rivals Stars||FBS||0.01||0.32||Low|
|Average O-Line Talent||Rivals Stars||BCS||0.03||0.18||Low|
|Average Tackle Talent||Rivals Stars||BCS||<0.01||0.64||Low|
|Avg. Interior Line Talent||Rivals Stars||BCS||0.05||0.07||Marginal|
|Left Tackle Talent||Rivals Stars||BCS||<0.01||0.87||Low|
|Throwin' Deep A||Yards per Completion||FBS||0.01||0.31||Low|
|Throwin' Deep B||Yards per Attempt||FBS||0.01||0.20||Low|
|QB Talent A||Rivals Stars||BCS||<0.01||0.80||Low|
|QB Talent B||Rivals Rating||BCS||<0.01||0.78||Low|
|Total Line Depth||Upperclassmen %||FBS||0.01||0.33||Low|
|Interior Line Depth||Upperclassmen %||FBS||0.01||0.43||Low|
This provides some much needed perspective. IME this really highlights the importance of experience, and especially the importance of the interior line. Interior line experience correlates more strongly with sack percentage than does a team’s tendency to throw the ball deep (at least when measured by yards per completion), and it serves as a better predictor than average talent of an entire offensive line (at least when measured by star ranking). This is really interesting! If I was a betting wizard, and I am, I would have bet on average o-line talent being a much better predictor of success than experience. Also, although the experience of the starting interior linemen does correlate significantly with sack percentage, depth along the offensive line does not.
The factor that comes closest to interior line experience in terms of predicting sack percentage is the talent of the interior of the line. This should strengthen our confidence in the conclusion that the interior line is the more crucial than the tackles in keeping the quarterback clean. As previously touched upon, when we combine interior line experience and interior line talent as two predictors of sack percentage and run a multiple regression, the r-squared returned is approximately 0.09. This isn’t huge by any means, but it serves as a better measure of success in pass protection than any single metric we’ve looked at so far.
Why don’t the best linemen play on the interior? This was one of the main questions raised during the last study, and the assumption was that teams play their best lineman at tackle in order to protect their quarterback. This study suggests that the interior of the line is more influential in accomplishing that task. There are a couple potential explanations. QB injuries and fumbles could still be most common from blind side hits, and team’s put their best guy there in order to mitigate these disasters. I haven’t tested this but I imagine it’s something that could be done statistically. It could also be that the best linemen play at left tackle because that’s the most important line position in the NFL, where one might assume that tackles matter more (hence their bloated salaries). If you look at the relationship between left tackle talent in the NFL (as measured by salary) and sack percentage, however, you get a pretty spurious correlation.
The line does trend up a bit suggesting that higher paid left tackles allow fewer sacks, but the r-squared is only 0.01 and the p-value is 0.66. It appears that left tackles aren’t much better at predicting pass protection success in the NFL than they are in college. (This is obviously more complicated than I’ve presented here. For example, teams could spend more on left tackles to fix problems that are inherent in the rest of their line or in their offensive system, thus producing a trend where teams with higher paid left tackles actually have higher sack percentages. This study is about college though, so I’m just leaving this for now).
I guess I just don’t know, man. The argument about protecting the quarterback from taking blind side hits makes intuitive sense to me, but the data all suggest that the interior is a more important factor in pass protection. If anyone’s got any quantitative study on why it makes more sense to play your best lineman at left tackle, or that tackles are, in fact, more crucial to pass protection, I’d be interested to see it.
What does this mean for Michigan? Let’s reemphasize that the experience data explain a relatively small proportion of the variance in sack percentage and that for any single team, and for any given team, team-specific explanations probably outweigh the statistical ones suggested by youth or talent. That being said, Michigan is very young where it appears to matter most. They are, however, talented – at least according to their star rankings. If these players develop at an average rate, then our line should make some serious strides by the time it’s full of talented upperclassmen on the interior. This is somewhat disheartening for this year but should provide some hope for the future.
This hope, of course, is based on the expectation of reasonable player development. We don’t need the best coaches in the world, since they tend to recruit already talented and physically gifted players, but we do need to develop those players on par with the rest of college football. I have no idea whether Borges and Funk have histories of successful o-line development, but it might be something worth looking into. The potential is there, however, to have a very successful o-line with regard to both the running and passing game as these kids become upperclassmen.
This study isn’t meant to indict or absolve any of the coaches, and it really does say more about college football as a whole than Michigan in particular. I do, however, think it’s interesting to see how Michigan’s production compares to other schools given a specific level of experience. We’re pretty far below the trend line even when experience on the interior is accounted for, and especially when talent along the interior line is taken into consideration. I think that Devin Gardner’s inner Ben Roethlisbergerness has something to do with this, as does Al’s predilection for two man routes where both receivers go deep. Experience, especially on the interior, does seem to play a role though. I don’t think it’s really possible to accurately assign percentages of blame (it’s really just a guessing game), but until we get that sack percentage out of the FBS basement, rest assured, there will be plenty of blame to go around.
the past the future (let's hope)
Happy MGoThanksgiving to all!
the 36th image that comes up when you google "turkey football"
yes, I am taking this as a sign we beat Ohio
A moment comes when you first start listening to minimalist music—for some people it comes quickly, for some people it never clicks at all—when your perception of time changes. As a musician famously described his first exposure to a Philip Glass opera; his initial boredom was transformed as...
I began to perceive...a whole world where change happens so slowly and carefully that each new harmony or rhythmic addition or subtraction seemed monumental...
...he said as the rhythmic woodblock...no, it's Adams not Glass...the woodblock crack of the pulling Stanford guard's pads as he thumped the Oregon SAM out of the hole play after play after play after...
NO! I will NOT spend my Thursday evening in an altered state of consciousness. So I started using the media timeouts, and then the time between plays (well, at least when Stanford had the ball, which thankfully was just about always) to work on a project I'd started a few days earlier during the Gameboy diaries, pulling participation reports for all 125 FBS teams and pulling roster/bio information to get the classes of their starters on the o-line.
And some of you people think huddles serve no purpose.
Honestly, the Horse Wasn't Dead When I Started
The results are here, usefully tabled in a spreadsheet to save some work for the next sap that starts on one of these projects.
Of course, as I sat down at my computer to do some regression analysis on the data I opened the blog and saw Gandalf's diary covering most of what I was planning to do (and doing a better job of it I might add). But I was taking a slightly different tack and found a couple of wrinkles, so for the sake of the eight of you that are still interested I'll continue on....
First a couple of comments about the dataset (feel free to skip the rest of this section, but it might be important if anyone uses the data for further analysis). Gandalf took his data from depth charts at the ourlads.com scouting site; mine come from the starting lineup listed in each school's participation report in the official game stats for their most recent game against FBS competition (sometimes coaches play with their lineup for games they're treating as exhibitions, give a start to a loyal walk-on for example, so if the most recent game was against a Delaware State I pulled the lineup for the week prior).
The official reports have the virtue, or defect, of being precise accounts of who was on the field. Sometimes that was a problem because everyone doesn't actually use five offensive linemen all the time. Idaho started a game with four, presumably spreading the field with covered, ineligible tight ends and wide receivers. Somebody else came out heavy and listed six. There were also some schools that simply listed their linemen as “OL” without assigning specific positions.
Where possible I straightened those situations out by using the schools' published depth charts. When that didn't work either I looked at third-party depth charts and did my best to reconcile them with the actual starters. It's possible there are a couple of players out of position here, but I don't think it's material.
For teams, usually pistol teams, that flop their line, I assumed the tight end would line up to the right and assigned the quick tackle and guard to the left side and the strong tackle and guard to the right.
For obvious reasons, service academies don't redshirt players. If an academy lineman's bio showed a year in which he didn't see game action, I counted that year as a redshirt and subtracted the year from his class. The point after all was to look at experience, not remaining eligibility.
Additive and Multiplicative Measures of Experience
My starting point was two proposals in the Gameboy diaries. Gameboy himself proposed assigning a value to each player (one point for each year, half a point for a redshirt) and adding them (well, averaging them, which of course is the same thing but for scale). That average appears in the spreadsheet as the GLEM (Gameboy Line Experience Metric).
In a comment to one of the diaries reshp1 suggested an alternative: assigning a value to each player based on experience (conceived as the probability that the player in question will successfully carry out his assignment) and multiplying those values and subtracting the product from one to get the probability that an assignment will be busted on a given play. That probability appears in the spreadsheet as the RBI (Reshp Bust Index). It's basically the weakest-link theory with the additional recognition that anyone might turn out to be the weakest link on a given play.
I focused on the latter metric because conceptually it makes sense to me and because it wasn't treated in Gandalf's diary. Reshp1 pulled the probabilities out of the air, or his hat, or somewhere, but the analysis doesn't seem to be sensitive to the particular choices here. The values are in a lookup table on page 2 of the spreadsheet if anyone wants to play around with alternatives.
Before I go on, a sanity check on Reshp1's metric—a list of the ten youngest lines:
- UCLA (7-2, 4-2)
- Idaho (1-9)
- California (1-9, 0-7)
- Wake Forest (4-6, 2-5)
- Eastern Michigan (2-8, 1-5)
- Western Kentucky (6-4, 2-3)
- Tulane (6-4, 4-2)
- Maryland (5-4, 1-4)
- Arkansas (3-7, 0-6)
- Michigan (6-3, 2-3)
Not a list you want to be on; those are some bad teams right there, combining for a 16-37 record in their respective conferences and that's flattering because it leaves out independent Idaho, who's probably the worst of the lot. (You can point to UCLA if you like as proof that, if everything goes right, you can survive starting multiple freshmen. Arkansas fans are probably pointing to Michigan and saying the same thing.)
The Running Game
Sanity check #2 is to redo Gandalf's work, but with Reshp's metric. Here's a graph of yards per carry vs. RBI:
That looks familiar. R2 is .058; the correlation coefficient is -.24 (these coefficients will all be negative because RBI is smaller for more experienced lines). And if we strip out the tackles and just look at the interior?
R2 is .084, the correlation coefficient is -.29, and it's not a coincidence that this looks an awful lot like Gandalf's chart using “youngest interior lineman”.
Weakest link, check. Experience matters more on the interior than at the tackles, check.
But what I really wanted to do was to look at the impact of o-line experience on an offense as a whole. To do that I've used the offensive component of the Fremeau Efficiency Index, which looks at all offensive drives (except for clock-kills and garbage-time drives) and compares the results to expectations based on the starting field position. By its nature it's pace-adjusted and independent of the effect of the team's defense; they also apply a strength of schedule adjustment.
Here's the chart:
R2 is .026, the correlation coefficient is –.16. The effect’s not as large, but a young line impacts the whole offense, not just the run game.
It made some sense that in the running game experience would matter more in the interior than at the tackles since it's an interior lineman that makes the line calls and the assignments tend to be more complicated inside. It wasn't so clear that this would still hold when the passing game was added in:
but that's what we find. The correlation is greater when we only look at the interior. R2 is .048, the correlation coefficient is -.22.
It's on the interior that experience really matters. And Michigan's interior RBI ranks 123rd of 125 FBS teams.
How Large an Effect?
A lot was made in Gandalf's diary, and especially in the comments, about the low R2 values here, which were seen as a demonstration of the relative unimportance of experience vs. other factors, like coaching.
I see it differently. This is an extremely diverse universe of teams we're looking at here. There are differences between Michigan and Eastern, or between Ohio State and Ohio U., that can't ever be overcome by something as simple as inexperience on the line. A lot of the scatter in these charts is just a matter of big programs being big and small programs being small. Given those enormous differences in baseline levels of the various FBS teams it's amazing to me that we could see anything like 5-8% of a performance difference being credited to any one team demographic, especially when the difference is measured using an SOS-adjusted metric like Fremeau.
And the slopes of these trend lines aren't small. The expected oFEI difference between 2012 Michigan and 2013 Michigan is .32; the actual difference is .197. The expectation, just correcting last year's performance for the youth on the field this year, was for a worse offense than we've actually seen.
Put another way, if you use that trend line to adjust for this year's lack of experience, add the missing .32, Michigan's offense goes to 19th in the nation, right behind Stanford and Louisville. UCLA turns into Oregon. Eastern becomes Bowling Green and maybe English keeps his job. Everybody's happy.
Good Teams are All Alike, Every Bad Team is Bad in its Own Way
I thought I'd try to get a handle on that by comparing each team's performance to the baseline they've established historically. I've averaged the oFEI's for each program for the five-year period from 2008-2012, then calculated the deviation of this year's performance from that average.
Basically, we're now looking at year-to-year deviations in performance within each program.
On the one hand, this gets rid of the scatter due to the vast discrepancy in baseline performance expectations from the top to the bottom of the division.
On the other hand, this also filters out any effect from programs like Wisconsin whose strength largely comes from the fact that they always field powerful, experienced lines. There's not much year-to-year variance there—they're always old, always good.
So it's possible we won't see any bigger correlation here than before...
...what happened? R2 is .009. Two-thirds of the effect is now gone. (A result, by the way, that's consistent no matter what metrics I use for line experience.) Apparently, only a third of the effect we’re looking at is a matter of one-off bad seasons due to a young line; most of the effect is systematic, inherent in particular programs. It's almost as if there were a correlation between poor past performance and current youth, and that's because there is:
There's the missing two-thirds. Historically (well, over the last five years anyway) bad teams are on the left, good programs on the right. There's less current youth (lower Bust Index) as you move right.
A look back at the teams listed earlier provides a clue. It's a mix of historically bad programs like Eastern, struggling FCS converts like Idaho, and programs that have suffered some sort of recent calamity, the kind that makes you decide to hire John L. Smith to be your substitute teacher for a year. Some had horrible recruiting, some had retention problems…each one has had its peculiar issues but every one of them is a program in disarray—some recovering, some not. Teams don’t field multiple freshmen because they want to; they do it because things fell apart.
We'll know more if someone does the study suggested in the comments to Gandalf's diary, looking at overall roster depth instead of just the age of the starters, but I think what's happening here is that the Wisconsin effect is the dominant effect in the study. Good programs don't suffer from youth on their lines because (a) it doesn't happen to them and (b) when it does, it's not a sign of weakness. When Andrus Peat finds his way to the top of the depth chart as a sophomore it's because he's beaten out multiple upperclassmen and won the position. When Kyle Bosch find his way to the top of the depth chart it's by default; the juniors and seniors he's supposed to be competing against aren't on the roster.
I think the next thing I might try, if I were of a mind to keep flogging this, is to do something so straightforward and blunt as to look for a correlation between offensive efficiency and the number of scholarship upperclass o-linemen on a roster (more telling than the percentage, I would guess).
Rerum omnium magister usus.
Experience is the teacher of all things.
Julius Caesar, Commentarii de Bello Civili, Book 2, Chapter 8
Setting. As the 2013 football season rolls on, the problems in Michigan’s run game have become more and more glaring. This has led to much ballyhooing and debate as to the main causes of Michigan’s ground game woes. The most basic argument is whether our problems are caused by weakness on the line or at the running back position. Brian’s UFRs come into play here, and while Fitz and the gang haven’t been perfect by any means, the play-by-play breakdowns seem to suggest that the problem lies with our offensive line. A phalanx our line is not.
Identifying the line as the problem, however, has not really made anyone very content. Rather it’s sparked a debate between the baby blamers – those who see Michigan’s youth as the source of their problems – and the crappy coach contingent – those who find fault with our coaches development of our o-line talent, not to mention play calling.
Previous Work. The MGoCommunity has already produced some solid work on this topic. The Mathlete’s preseason study looked at other teams who had offensive lines with an 1st round NFL pick combined with 2+ freshmen. Although he eventually admitted that we were still probably a year away, his comparison grouped us with the likes of Alabama, Oregon, and Stanford. GuloGulo’s diary from back in September looked at the relationship between average o-line experience in the Big 10 and success mainly defined as yards per play. After the first four weeks of the season, he concluded that we were about average in both experience and success. Gameboy’s recent diaries have shown that Michigan’s line is relatively young whether you take a “number of years in the program” or “number of previous starts” approach.
Questions. In this study I want to delve a little deeper into what we mean by “experience,” what we mean by “success,” and how those two variables are actually related. I will attempt to answer four questions:
- Can offensive line experience explain run game success?
- Are years or starts a better measure of experience?
- Is interior line experience more important that tackle experience?
- Is average experience a better measure than the weakest link?
Definitions. Neither experience nor success have single and obvious definitions. With regard to the o-line, success could be defined by wins, yards per play, yards per rush, sack percentage, or play-by-play results a la UFR. For the first part of this study, I use yards per carry as my metric for success. Experience, likewise, can be defined in a variety of ways, including the number of years in the program, the number of starts, or the number of snaps. This analysis primarily uses the number of years in the program as its measure for experience. This isn’t because it’s necessarily the best measure – we’ll test that in a bit – but rather because it’s the measure that’s easiest to find information about. Redoing this study with a start- or snap-focused measure of experience would be a worthwhile endeavor. In the graphs below, the year of the players are equated with numbers, so that freshman = 1, red shirt freshman = 1.5, sophomore = 2, red shirt sophomore = 2.5, and so on.
Data. The data for this study are drawn from this 2013 season. All 125 FBS teams are included. The YPC stats come from ESPN and the experience info comes from the a scouting site. Because this isn’t necessarily about Michigan’s o-line this year, but rather about the general relationship between offensive line experience and success, data from any recent season should apply though. Giving this thing some time depth would probably improve its efficacy. The stats are current as of 11/6/2013. All the images are from the MGoBlog flickr account; Bryan Fuller gets the credit, I believe.
This is a primarily quantitative study, but I’m in no way a statistician. My background is in Classics, as in Greek and Roman studies, so although I’ve tried brush up on my stats, there’s certainly the possibility that these metrics aren’t employed or interpreted perfectly. Feel free to correct me.
With that said, it’s probably useful to give a brief overview of the statistical measures in an attempt to describe what they actually tell us. I’m looking at 4 main metrics: correlation coefficient, r-squared, p-value, and slope of the linear trend line.
Correlation coefficient: The correlation coefficient quantifies the degree of linear relation between two variables. The coefficient ranges from -1.0 to 1.0, and the larger the absolute value of the coefficient, the stronger the relationship. This will provide a single number for the strength of the relationship between o-line experience and yards per carry.
R-squared: The r-squared provides a measure for the amount of variance in one variable that can be explained by another variable. This will be used to assess how much of the variance in yards per carry can be explained by o-line experience. It’s important to note here that there are obviously many other factors other than experience that govern running game efficiency (coaching, scheme, running back skill, etc.). A low r-squared doesn’t necessarily mean that experience is unimportant, just that other factors are also important.
P-value: The p-value let's us know whether our results are statistically significant; more specifically it provides a measure to assess whether we should discard the null hypothesis. In this case, the null hypothesis is that there is no relationship between o-line experience and running game success. The p-value ranges from 0 to 1. A small p-value, < 0.05, suggests that we reject the null hypothesis, while a large p-value suggests we retain it. If p-values are low, we should have faith in the relationship between experience and success; if they are high we should feel less confident about that relationship.
Slope of linear trend line: The trend lines in the graphs below show the linear relationship between experience and success. The slopes of that lines indicate the extent to which we’d expect YPC to change as a result of a change in experience. For example, if the slope was 0.5, the data would suggest that an extra year of average o-line experience is worth ½ of a yard per carry.
Question 1: Can offensive line experience explain run game success?
Good habits formed during youth make all the difference. - Aristotle
The scatter plot below depicts the relationship between average offensive line experience in years and yards per carry. Click for enlarged scatterplot with all BCS teams labeled.
The data broadly confirm what we’d expect. This is good! This means that we’re right in claiming our youth is (partially) the problem. The older your offensive line is, on average, the more yards per carry that team produces. The correlational coefficient is 0.16 for this data set, indicating that there’s a slightly positive correlation between offensive line experience and yards per carry. The r-squared is small, however, suggesting that only about 3% of the variation in teams’ yards per carry can be attributed to the experience of the offensive line. A p-value of 0.07 is marginal, meaning that it’s not particularly clear whether we should interpret these results as significant or not. Let’s start by giving experience the benefit of the doubt though, and for the time we can conclude that experience does indeed influence ground game success. The slope of the linear trend line suggests that an extra year of average experience is worth about 1/3 of a yard per carry.
At first glance, there does seem to be a positive correlation between o-line experience and YPC, although there is still a lot of variance in YPC that cannot be explained by experience.
Question 2: Are years or starts a better measure of experience?
One of the arguments against the approach taken in question one is that an offensive lineman’s number of previous starts is a better measure of experience than the number of years he’s been in the program. Let’s take a look; the graph below plots this alternate measure of experience against yards per carry. Click to enlarge and see Oregon and Wisconsin put up 6.7 YPC despite having less total starts along the o-line than Michigan.
The relationship between the number of previous OL starts and yards per carry generally mirrors the pattern produced when the number of years in the program is taken into consideration. The correlation coefficient is actually slightly higher (0.23 compared to 0.16), suggesting that starts is indeed a slightly better measure than years in the program for the purpose of predicting o-line success. The r-squared suggests that previous starts can explain about 5% of the variance in yards per carry, and a p-value of 0.01 indicates that these results are indeed significant. The slope of the line suggests that each extra start is worth about 1/100 of a yard per carry, meaning that 50 extra stars is worth about ½ a yard, and 100 extra starts is worth about a full extra yard per carry.
Now that the data show that “number of starts” is probably a better measure of offensive line success, I’m going to revert to “number of years in the program” as my main metric of experience. This is simply due to the convenience of the data. If someone can get number of starts for all the programs, that should improve things. Perhaps another day.
Question 3: Is interior line experience more important than tackle experience?
Why doesn't Lewan make everything okay?
One of the most common arguments against using the average or total experience of the entire offensive line is that all spots along the line are not created equal. Lewan being an awesome LT doesn’t help our RG Mags getting crushed by the NT. Essentially, interior line experience is more important than tackle experience. But does it really matter whether your experience comes on the interior or exterior of the line?
Let’s start with tackle experience first. The graph below shows the relationship between the average experience of each team’s tackles and their YPC.
Check out Michigan and Purdue with their bookend fifth year senior tackles. This doesn’t bode well for a positive relationship. Looking across the entire spectrum of the FBS, there appears to be no correlation between the experience of a team’s tackles and their ability to run the ball successfully. Once again, this is good news for us. It’s not that we’re not taking advantage of our great tackles, it appears that on the whole, tackle experience just doesn’t influence ground game success all that much. The correlation coefficient is a measly 0.02, the r-squared is <0.01, and the p-value is 0.81, which is incredibly high. The slope of the trend line suggests a very, very slight decrease in YPC as tackles increase in age, which doesn't make any sense at all.
This is really interesting actually, as all metrics suggest there is essentially no connection between tackle experience and yards per carry. If tackles aren’t the cause of the correlation between total experience and YPC, then it must be the interior of the line, right? Click to enlarge and see us at the children's table with UCLA and Purdue.
It appears as though the “our interior line is full of infants” excuse is actually a pretty good one. With a correlation coefficient of 0.22, the relationship between these two variables is stronger than when offensive line experience as a whole is averaged (in years) and an order of magnitude stronger than the correlation between tackle experience and YPC. The r-squared indicates about 5% of YPC variation can be explained by experience along the interior of the line, and a p-value of 0.01 suggests these results are significant. The slope of the trend line suggests an extra year, on average, is worth about 1/3 of a yard per carry.
If you extrapolate that out over the course of a season, that’s about 150-200 extra yards of rushing per year (Michigan had 502 rushing attempts in 2012 according to ESPN). Interior line experience does seem to be a big deal. Also, we’re one of the 3 youngest teams out of 125 FBS teams in terms of interior line experience. That is young indeed.
Question 4: Is average experience a better measure than the weakest link?
The foundation of every state is the education of its youth. - Diogenes
Thus far the data have shown that interior line experience is a better predictor of running game success than total offensive line experience. The next question is whether average interior line experience is a better predictor of success than the “weakest link” along the line. In this case we’ll call the youngest person on the interior of the line the weakest link. This really has nothing to do with their ability, it’s just a measure of their experience in the program. Click to enlarge and see Auburn averaging 6+ YPC while starting a true freshman interior lineman.
It looks as though there is something to the “weakest link” argument. The correlation coefficient in this case is 0.29, which stands as our strongest correlation yet between some measure of experience and yards per carry. The r-squared indicates that this measure can explain about 8% of YPC variation, and a p-value of 0.01 suggests that these results are indeed significant. The slope here once again suggests that an extra year is worth about 1/3 of a yard per carry.
The fact that the age of a team's youngest interior offensive lineman is a better predictor of run game success than its average offensive line experience, or even the average experience of just the interior line, is rather unexpected. This should bode well for Michigan's future along the line as we gain experience and depth in future seasons.
First off, offensive line experience leaves a lot of the variance in yards per carry unexplained. So even though this study supports the conclusion that offensive line experience does indeed influence success in the running game, there are clearly many other factors that also play a role.
In this study, experience has been measured in in two ways, both as “years in the program” and as “number of starts.” While both serve as decent predictors of success in the running game as judged by YPC, number of starts seems to be the better measure. Unfortunately, it’s also the measure that is more time-intensive to track. When looking at the outer vs. interior line, the data suggest that success on the ground is much more closely tied to the experience of the interior line than it is to either the tackles or even the average experience of the line as a whole. Surprisingly, tackle experience seems to be completely irrelevant as a predictor of run game success. Finally, the level of experience of the least-experienced person on the interior line serves as an even better metric for predicting running game efficiency. The “weakest link” argument appears to hold water.
Unit of Measurement
|R-Squared||P-Value||Effect on YPC|
|Total Experience||Years||0.16||0.03||0.07||Extra year = +1/3 yard|
|Total Experience||Starts||0.23||0.05||0.01||Extra 10 starts = +1/10 yard|
|Interior Line Experience||Years||0.22||0.05||0.01||Extra year = +1/3 yard|
|Youngest Interior Lineman||Years||0.29||0.08||0.01||
Extra year = +1/3 yard
What does this mean for Michigan? As Gameboy showed us in his diaries, Michigan is young along the O-Line, whether you’re judging by years in the program or by number of starts. What I hope to have demonstrated here is that (a) being young really does matter, and (b) we’re especially young where it matters most (i.e., tied for 2nd youngest on the interior OL out of 125 FBS teams).
Borges and Funk in happier times
There’s been a lot of heat on Borges and Funk recently, and it’s appropriate to ask whether this study indicts or absolves them. Unfortunately, I think the data tend to side step the question. The fact that o-line experience does seem to influence YPC, and especially the finding that interior line experience seems to be of utmost importance, combined with Michigan’s position with regard to these measures (i.e., they fall almost exactly along the linear trend line in both the interior line experience graph and in the weakest link graph), would initially suggest that the line is performing about as expected.
This doesn't let the staff off the hook. The relatively low r-squared values would indicate that there is a lot more than just experience that goes into producing a successful running attack. Coaching, both in terms of scheme and player development, is probably one of the most influential factors in governing run game success, and this study doesn’t attempt to measure or control for that aspect of the game. Moreover, this study doesn't account for talent along the offensive line, which would probably suggest Michigan is underperforming relative to the recruiting rankings. Strength of schedule is also omitted. Having played CMU, Akron, UConn, Indiana, and Nebraska, adding this variable could also raise our team's expected YPC, and in doing so lower our performance relative to expectations.
According to the eyeball test, the apparent regression along the offensive line would seem to indicate that there are some seriously problematic coaching issues. There are several BCS programs with similar youth-related issues on the interior line, both when experience is averaged (e.g., UCLA) and when experience is defined by the youngest interior lineman (e.g., Notre Dame, Arkansas, and Auburn), and these programs still manage to perform significantly better than us in terms of yards per carry. When viewed within the context of the entire FBS, however, the data suggest that Michigan’s youth is a real and influential issue.
On the bright side, this should give us hope for future seasons. As our interior line matures, both in terms of average experience and in terms of its weakest link, we should improve. This only holds, of course, if all the other factors that go into producing a successful offensive line – namely coaching scheme and player development – are on par with the rest of college football. That, unfortunately, is not guaranteed.
Rome wasn’t built in a day, but it did progressively grow bigger and better until it reached a point where it dominated at the point of attack. Let’s hope our offensive line can do the same.
Coming in part 2: Shouldn’t our veteran tackles at least make us better at pass protection?!?
Awww, come on.
 Actually, upon further review, I’m not so sure this is accurate. Over the course of 8 games, Fitz has 5 positive UFR games, 2 negative, and 1 around zero, while the OL has 5 positive UFR games, 2 negative, and 1 around zero. Obviously RB and OL numbers aren’t perfectly commensurate, but this probably suggests the blame should be shared.
Not to undermine Butterfield (LINK) who's numbers appear both impeccable and comforting but some hack named Nate Silver published an article for the New York Times (tumblr site?) giving Michigan a somewhat, err, dimished likelihood of winning the NCAA tournament. Via UMHoops (LINK):
Michigan (up to 3.8 percent from 2.5 percent) Michigan could easily enough have been a No. 1 seed had it played better down the stretch, and it was probably underseeded as a No. 4 even with the losses that it took. In general, however, we’ve found that late-season performance doesn’t tell you that much more than early-season performance when it comes to tournament play – and Michigan’s slump has not extended into the postseason. It was a break for the Wolverines to play in Auburn Hills, Mich., but their domination of a tough Virginia Commonwealth team on Saturday was nevertheless impressive, and they should be thought of as the equivalent of a strong No. 2 seed right now.
It is worth noting that if the 3.8% continues to increase it will someday reach 81.45% and all will be right with the world.
Nearly 3 years after publication, Elsevier has retracted a paper for containing no scientific content.
Link to the paper itself.
Entirely ludicrous, yet something I thought this fine community of intelligentsia would appreciate. Perhaps Brian, in his altered state of consciousness, can make sense of it for us.
EDIT: Be sure to read the comments for links to more articles from the same "author"