fair point that
Setting the Scene
no animals were harmed in the making of this diary
Apology. I am sorry to all those who are cringing at seeing another experience diary. I had originally conceived of this as being a two part study, with the first looking at the running game and the second looking at the passing game. Despite the other diaries, both of which were useful in their own ways, I think there is still some horse meat to be gleaned from the carcass of o-line experience as it relates to pass protection. The horse might be dead, but that doesn’t mean it’s useless. Oh, and there's another one since I originally wrote this thing up.
Previous Work. In the first diary, I attempted to demonstrate that o-line experience does indeed play a role in governing a team’s ability to run the ball. R-squared values ranged between about 0.05 and 0.10 depending on how we defined “experience,” suggesting that about 5-10% of the variation in YPC across FBS can be explained by a team’s experience along the o-line. Clearly other factors are also important to running the ball well, but experience isn’t meaningless. Further manipulations to the data set suggested that interior line experience is more important than tackle experience and that the “your line is only as good as its weakest link” argument does hold water.
Some of the comments from that diary questioned why we don’t move one or both of our experienced and talented tackles to the interior if that is where it really seems to matter. Transaction costs of moving around linemen aside, the question is valid in general terms. Why not put your best linemen on the inside if those are the most important positions? A variety of answers could be given to this question – for example, exterior and interior line positions could have different ideal body types with regard to height, weight, strength, and agility – but the most obvious response is that tackles are more influential in the passing game. Our best linemen play at tackle because they protect our quarterback.
Questions. This is a proposition that can be tested statistically, and that’s what I aim to do in the second part of this study. My metric for o-line success in the passing game is sack percentage (i.e., the percentage of pass attempts on which your QB is sacked), since it’s the o-line’s job to keep the QB clean. Using the same essential methodology as the last study, I aim to answer four questions:
- Does o-line experience help prevent sacks?
- Is tackle experience or interior line experience more important in protecting the quarterback?
- Does the “weakest link” theorem hold for the passing game?
- What else could influence sack percentage?
Data. This study looks at 123 FBS teams (Georgia State and UTSA are omitted since their info wasn’t on ESPN). Sack percentage stats come from ESPN, and the experience data comes from the scouting site Ourlads. Star rankings that come at the end of this study are taken from Rivals. Photos come from MGoBlog's flickr account and are attributable to Bryan Fuller. Check out the previous diary for basic definitions of the statistics that are used.
This is long, so buckle up. Feel free to jump to the conclusions if you don’t want the nitty gritty. All the data are summarized in a…chart? Chart.
Questions and Answers
Probably cropped out: massive amounts of backside pressure
Question 1: Does o-line experience help prevent sacks?
Let’s start by taking a broad look using average experience in years of the offensive line. The relationship between experience and sack percentage is plotted below. Click on the graph to see the same sack percentage data plotted against total number of starts. The plot is oriented so that up is good (i.e., your QB isn't getting sacked that often) and down is bad (i.e., you're looking like Michigan against MSU and Nebraska).
Although the trend line makes it look as though there is an inverse relationship between sack percentage and experience (i.e., sacks go down as experience goes up), which is what we’d expect, the r-squared is relatively low (0.02) and the p-value (0.10) suggests the trend may not be statistically significant. If we plot the same relationship but use starts as our metric for experience, the relationship becomes even more spurious with an r-squared of 0.01 and a p-value of 0.28 (click on above graph to see scatter plot). On the whole, total or average o-line experience doesn’t seem to be a great predictor of the o-line’s ability to keep the quarterback from getting sacked.
Question 2: Is tackle experience or interior line experience more important in protecting the quarterback?
I don't care who matters more as long as we keep it under 7 sacks a game
We saw in the previous study that interior line experience was more important to run game success than tackle experience. We’d likely expect the opposite to be true for the passing game based on the premium put on left tackles both in college and in the pros. Average tackle experience and sack percentage is plotted below.
This is unexpected. The correlation is spurious. The r-squared is less than 0.01, and the p-value is 0.67, both of which suggest there is no correlation between tackle experience and sack percentage. The trend line actually rises slightly, which would indicate that sack percentage rises as tackles get more experienced, which makes no sense at all, even if the correlation was statistically significant. On the whole, tackle experience does not appear to be a good predictor of your team’s ability to not give up sacks.
Could the interior of the line be more important in the passing game as well? Click for enlarged scatterplot with teams divided by conference and all BCS teams labeled.
Now we’re getting somewhere. The trend line makes intuitive sense. The percentage of sacks you give up goes down as your interior linemen become more experienced. The r-squared is 0.05, implying interior line experience can explain about 5% of the variance in sack percentage. The p-value is 0.02 suggesting that results are statistically significant. The slope of the trend line suggests that an extra year of average interior line experience is worth a drop of almost 1 percent in sack percentage. If you extrapolate that over the course of a season, that’s about 3 to 5 fewer sacks. Not a huge difference, but if your team matures over the course of several years from starting freshman to starting seniors, that adds up to reducing sacks by about 1 per game.
One possible critique here is that “average” tackle experience is not the correct measure. Teams often put one of their better run blockers at right tackle and their best pass protector at left tackle. Thus instead of looking at the average, we should just look at the correlation between left tackle experience and sack percentage.
We don’t even need a graph here. We get almost the exact same trend as when the tackle experience data are averaged, and the slope of the trend line suggests that sack percentage slight increases as tackles get older. That is intuitively incorrect. A low r-squared value (<0.01) suggests left tackle experience doesn’t matter very much and a high p-value (0.31) implies statistical insignificance. This is admittedly somewhat baffling and definitely unexpected.
Question 3: Does the “weakest link” theorem hold for the passing game?
weakest link only in age, not awesomeness
When looking at the run game, the data suggested that the youngest member of the interior line was a better predictor of success than average experience of the interior line. In the passing game, the “weakest link” is a little less weak.
Unlike with the run game, average interior line experience appears to serve as a better predictor of sack percentage than does the “weakest link” along the interior of the line. The r-squared here is 0.02 and the p-value is 0.12, suggesting that the significance is marginal at best. It's not that the weakest link is a terrible predictor, just that the average experience of the entire interior line serves as a slightly better indicator of sack percentage.
At this point we can draw some basic conclusions from the first three questions. Total or average o-line experience only seems to be a marginal predictor of a team’s ability to keep their quarterback from getting sacked. Tackle experience, whether averaged or just taken as the left tackle, appears to have no relationship whatsoever with sack percentage. Just like with the run game, interior line experience seems to be the most salient characteristic with regard to o-line experience for predicting success in pass protection.
Question 4: What else could influence sack percentage?
One of the main critiques from the last study was that we’re living in a multivariate world and other potentially influential factors should be included in the analysis. I’m still working on getting myself up to speed regarding multivariate analysis, so I’m tentative to try and do too much with that now. We can, however, look to see how some other variables correlate with sack percentage.
Offensive Line Talent
4 star, 4 star, 5 star...sack?
Talent is one obvious potential factor in governing pass protection success. The chart below shows the correlation statistics for the star rankings of the o-line with regard to sack percentage. I’ve omitted the graphs here because none of these produce any correlation of statistical significance.
|Average O-Line Star Ranking||0.01||0.30||Low|
|Average Tackle Star Ranking||<0.01||0.52||Low|
|Average Interior Line Star Ranking||0.01||0.22||Low|
|Left Tackle Star Ranking||0.01||0.32||Low|
Surprisingly, star ranking of offensive linemen doesn’t seem to correlate very strongly with sack percentage. My guess is that this is due to star ranking of offensive linemen correlating closely with the difficulty of defense that a given team plays. For example, Alabama has talented linemen, but they play against tough defenses in the SEC. Toledo, on the other hand, has crappy linemen, but they play against week defenses in the MAC. Moreover, there tends to be very little variation in star rankings with non-BCS schools – almost everyone is a 2 star – so this may be obscuring some of the impact that talent (i.e., star rating) has on pass protection.
This can be accounted for, to some extent, by looking at only the BCS schools, where there will be more variation among offensive line star ratings and more consistency in the level of teams played. The chart below shows the correlation statistics for offensive line talent and pass protection success.
click to zoom with all BCS teams labeled
|Average O-Line Star Ranking||0.03||0.18||Low|
|Average Tackle Star Ranking||<0.01||0.64||Low|
|Average Interior Line Star Ranking||0.05||0.07||Marginal|
|Left Tackle Star Ranking||<0.01||0.87||Low|
Once again, it appears as though the interior of the line is the most crucial for preventing sacks. This corresponds well with the experience data presented in the first three questions. If we’re operating under the hypothesis that the interior of the line is more important than the tackles with regard to pass protection, which the experience data suggest, then we’d expect talent to matter more on the interior than it does at the tackles as well. It turns out that this is exactly the result we get. Whether looking at experience or talent, the interior seems to be the key to success.
I don’t know much about multivariate regression, but when you take both experience and talent of the interior of the offensive line into account for predicting sack percentage, an r-squared of 0.09 is produced. This is almost double the r-squareds produced by regressing experience and sack percentage and talent and sack percentage, and it suggests that these two factors work in tandem to determine the success of the offensive line regarding pass protection.
Unleashing the Dragon
A team’s tendency to throw deep, thus necessitating a longer drop and more time in the pocket, could be another influential factor governing sack percentage. I thought that yards per completion would be the best measure of a team’s tendency to throw deep, since yards per attempt could be equally as high for teams that throw quick, short passes, but complete a high percentage of them. Either way, we can look at both metrics.
click to see scatterplot of ypa and sack percentage
|Passing Depth Metric||R-Squared||P-Value||Significance?|
|Yards per completion||0.01||0.31||Low|
|Yards per attempt||0.01||0.20||Low|
These measures do not correlate particularly well with sack percentage. Yards per completion gives us the trend we’d expect – that sacks go up as yards per completion go up, but the explanatory value is weak as the p-value suggests insignificance. When doing a multivariate regression with yards per completion and interior line experience against sack percentage, the r-squared only rises to 0.056 from 0.05. It doesn’t add much explanatory value. Using YPA instead of yards per completion actually produces a trend where it appears that increased yards per attempt facilitate a decrease in sack percentage. That doesn’t make a lot of sense and the correlation and statistically insignificant anyway.
4 star talents, 5 star smiles
A third possible factor governing sack percentage is the skill of the quarterback. Perhaps sacks are less a matter of how good the line is and more a matter of how good the QB is. To measure this I look at each BCS quarterback’s star rating and their Rivals rating (4.9-6.1) to see whether their high school talent correlates with how often they end up sacked. You can’t really use any college stats as a measure of their talent, because those can be directly influenced by the play of the offensive line, and we’re trying to isolate QB talent as a separate and independent variable here.
click to zoom
There’s really not much here. Whether you go by star ranking (2-5 stars) or by the more precise Rivals rating (4.9 – 6.1), there’s no significant relationship between a quarterback’s talent and his ability to remain upright. R-squareds for both metrics are <0.01, and p-values are 0.75 – 0.80. On an individual level, the skill of a single quarterback might help him avoid sacks, but taken broadly across all BCS schools, quarterback talent doesn’t seem to be a factor.
all 10 linemen on FSU's 2-deep are upperclassmen /Miami Herald
Probably the most common critique in the previous diary was that depth should be taken into account. You can do this different ways: average or total experience on the 2-deep, the oldest player at each position on the 2-deep, or the percentage of upperclassmen on the 2-deep. For this study I'm using the last of these definitions, the percentage of upperclassmen. I'm defining "upperclass" as students who have been on campus for two years prior to this season. So redshirt sophomores and true juniors are both considered "upperclassmen," while true sophomores are not. The graphs below show the trends for both the line as a whole and the interior of the line.
click to see all teams labeled - Duke also has an all-upperclass 2-deep
The correlation is unexpectedly poor. The graphs above show line depth both across the entire o-line and just the interior of the line. In both cases, the trend line suggests that the more upperclassmen you have, the more sacks you give up. This doesn't pass the common sense test, and r-squareds for both are low (0.01) and p-values are high (>0.30) implying that the correlation is not statistically significant. It doesn't appear to be a matter of defining "upperclassmen" either. If you run the same regression using average depth on the line, you get the same spurious results. While line depth might be an excuse for any given team, across the entire FBS the experience of your starters seems to matter much more than the experience on the entire depth chart.
Modest but significant. Despite using a completely different metric for o-line success, sack percentage instead of YPC, the conclusions of this study are eerily similar to the previous one. Let’s begin with the (hopefully) obvious caveats. Offensive line experience explains a modest, though significant, amount of the variation in sack percentage across all FBS schools. We’re talking about 5% of the variation here, so there are clearly a lot of other factors that go into determining how good a team is at protecting their quarterback.
In a way, this study is much less about Michigan than it is about college football in general. The success or failure in pass protection for a single team can often be explained by factors that are specific to that team. For instance, Devin Gardner is essentially the Ben Roethlisberger of college football, refusing to throw the ball away or to be tackled until approximately half the other teams defenders are draped all over him. This certainly contributes to Michigan’s high percentage of sacks, but it is a difficult variable to account for and measure across all of college football.
That being said, offensive line experience does stand out as a particularly salient characteristic for explaining a team’s sack percentage. Although we’d assume that experience at the tackle positions would be more important in the passing game, the results of this study suggest that once again the interior of the line is what matters most. In contrast to the previous study, the “weakest link” (i.e., the youngest interior linemen) is not as good of a predictor as the average experience of the interior of the line.
Taking a comparative approach by looking at experience alongside other potentially influential factors provides some context for how important experience actually is. The chart below plots each of the metrics I’ve looked at in this study along with their r-squared and p-values.
|Independent Variable||Unit of Measurement||Data Set||R-Squared||P-Value||Significance?|
|Avg. O-Line Experience||Years||FBS||0.02||0.10||Marginal|
|Total O-Line Experience||Starts||FBS||0.01||0.28||Low|
|Avg. Tackle Experience||Years||FBS||0.01||0.67||Low|
|Avg. Interior Line Exp.||Years||FBS||0.05||0.02||High|
|Left Tackle Experience||Years||FBS||<0.01||0.31||Low|
|Average O-Line Talent||Rivals Stars||FBS||0.01||0.30||Low|
|Average Tackle Talent||Rivals Stars||FBS||<0.01||0.52||Low|
|Avg. Interior Line Talent||Rivals Stars||FBS||0.01||0.22||Low|
|Left Tackle Talent||Rivals Stars||FBS||0.01||0.32||Low|
|Average O-Line Talent||Rivals Stars||BCS||0.03||0.18||Low|
|Average Tackle Talent||Rivals Stars||BCS||<0.01||0.64||Low|
|Avg. Interior Line Talent||Rivals Stars||BCS||0.05||0.07||Marginal|
|Left Tackle Talent||Rivals Stars||BCS||<0.01||0.87||Low|
|Throwin' Deep A||Yards per Completion||FBS||0.01||0.31||Low|
|Throwin' Deep B||Yards per Attempt||FBS||0.01||0.20||Low|
|QB Talent A||Rivals Stars||BCS||<0.01||0.80||Low|
|QB Talent B||Rivals Rating||BCS||<0.01||0.78||Low|
|Total Line Depth||Upperclassmen %||FBS||0.01||0.33||Low|
|Interior Line Depth||Upperclassmen %||FBS||0.01||0.43||Low|
This provides some much needed perspective. IME this really highlights the importance of experience, and especially the importance of the interior line. Interior line experience correlates more strongly with sack percentage than does a team’s tendency to throw the ball deep (at least when measured by yards per completion), and it serves as a better predictor than average talent of an entire offensive line (at least when measured by star ranking). This is really interesting! If I was a betting wizard, and I am, I would have bet on average o-line talent being a much better predictor of success than experience. Also, although the experience of the starting interior linemen does correlate significantly with sack percentage, depth along the offensive line does not.
The factor that comes closest to interior line experience in terms of predicting sack percentage is the talent of the interior of the line. This should strengthen our confidence in the conclusion that the interior line is the more crucial than the tackles in keeping the quarterback clean. As previously touched upon, when we combine interior line experience and interior line talent as two predictors of sack percentage and run a multiple regression, the r-squared returned is approximately 0.09. This isn’t huge by any means, but it serves as a better measure of success in pass protection than any single metric we’ve looked at so far.
Why don’t the best linemen play on the interior? This was one of the main questions raised during the last study, and the assumption was that teams play their best lineman at tackle in order to protect their quarterback. This study suggests that the interior of the line is more influential in accomplishing that task. There are a couple potential explanations. QB injuries and fumbles could still be most common from blind side hits, and team’s put their best guy there in order to mitigate these disasters. I haven’t tested this but I imagine it’s something that could be done statistically. It could also be that the best linemen play at left tackle because that’s the most important line position in the NFL, where one might assume that tackles matter more (hence their bloated salaries). If you look at the relationship between left tackle talent in the NFL (as measured by salary) and sack percentage, however, you get a pretty spurious correlation.
The line does trend up a bit suggesting that higher paid left tackles allow fewer sacks, but the r-squared is only 0.01 and the p-value is 0.66. It appears that left tackles aren’t much better at predicting pass protection success in the NFL than they are in college. (This is obviously more complicated than I’ve presented here. For example, teams could spend more on left tackles to fix problems that are inherent in the rest of their line or in their offensive system, thus producing a trend where teams with higher paid left tackles actually have higher sack percentages. This study is about college though, so I’m just leaving this for now).
I guess I just don’t know, man. The argument about protecting the quarterback from taking blind side hits makes intuitive sense to me, but the data all suggest that the interior is a more important factor in pass protection. If anyone’s got any quantitative study on why it makes more sense to play your best lineman at left tackle, or that tackles are, in fact, more crucial to pass protection, I’d be interested to see it.
What does this mean for Michigan? Let’s reemphasize that the experience data explain a relatively small proportion of the variance in sack percentage and that for any single team, and for any given team, team-specific explanations probably outweigh the statistical ones suggested by youth or talent. That being said, Michigan is very young where it appears to matter most. They are, however, talented – at least according to their star rankings. If these players develop at an average rate, then our line should make some serious strides by the time it’s full of talented upperclassmen on the interior. This is somewhat disheartening for this year but should provide some hope for the future.
This hope, of course, is based on the expectation of reasonable player development. We don’t need the best coaches in the world, since they tend to recruit already talented and physically gifted players, but we do need to develop those players on par with the rest of college football. I have no idea whether Borges and Funk have histories of successful o-line development, but it might be something worth looking into. The potential is there, however, to have a very successful o-line with regard to both the running and passing game as these kids become upperclassmen.
This study isn’t meant to indict or absolve any of the coaches, and it really does say more about college football as a whole than Michigan in particular. I do, however, think it’s interesting to see how Michigan’s production compares to other schools given a specific level of experience. We’re pretty far below the trend line even when experience on the interior is accounted for, and especially when talent along the interior line is taken into consideration. I think that Devin Gardner’s inner Ben Roethlisbergerness has something to do with this, as does Al’s predilection for two man routes where both receivers go deep. Experience, especially on the interior, does seem to play a role though. I don’t think it’s really possible to accurately assign percentages of blame (it’s really just a guessing game), but until we get that sack percentage out of the FBS basement, rest assured, there will be plenty of blame to go around.
the past the future (let's hope)
Happy MGoThanksgiving to all!
the 36th image that comes up when you google "turkey football"
yes, I am taking this as a sign we beat Ohio
[Editor's note: frontpaged for obvious reasons. A scheduling mix-up with Brian caused this to get buried earlier, so I'm bringing it back near the top. [How much] Will Michigan rue the loss of Brandon Graham? - Tim]
[Note: I have 2006 fully loaded into the database now and will be included in all future multi-year studies along with 2007-2009.]
We can all agree that sacks and interceptions are good things for the defense and bad things for the offense. But how does a viable pass rush or a ball-hawking secondary affect the performance of the opposing offense on plays where there isn’t a sack or a pick. Likewise, what is the correlation between an offensive line that gives up sacks regularly or a mistake prone quarterback?
Sacks and interceptions have very similar direct impacts on games. From 2006-2009 in games between two D1 teams in competitive game situations (the “universe” for this and most of my analysis) the average defensive unit produced 2.3 ppg worth of sacks and 2.0 ppg worth of interceptions. Sacks have a slightly higher direct value than interceptions (interceptions returns and fumble returns on sacks are not included) but does either of these correlate to a better defensive performance overall.
Chart time? Let’s make it a double.
Not entirely surprisingly, the better a defense is at producing sacks and interceptions, the better it is on downs where neither occur.
For every point per game that a defense generates due to sacks, the overall pass rush generates 1.2 ppg of additional value. Interceptions are also powerful, but not as much so. Each ppg of value a defense generates through interceptions is worth 0.9 ppg of additional value.
This analysis serves to confirm what most football fans already know. Teams that can create interceptions and sacks are good going to be better defensive teams. Whether a strong pass rush/secondary creates pressure on other downs or if strong pass rushes and secondaries are a common occurrence on great defenses is irrelevant. As most of you probably know, defenses that are good at these two things are also good on other downs. So why is this interesting…
The story becomes very different when you look at offenses. The conventional wisdom that was supported for defenses is largely blown up on the offensive side of the ball. Sacks and interceptions may be indicators of great defenses, but they are not symptoms of bad offenses.
The slope of these two charts are about 20% of the gradient of the corresponding defensive charts and virtually flat. On offense, the amount of sacks and interceptions are largely independent of performance. There is obviously the immediate negative effect of the play, but giving up sacks or throwing interceptions show virtually no correlation to success or failure on other downs.
What it means?
For one of side of the ball it merely quantifies conventional wisdom. Good pass defenses get interceptions and sack the quarterback and teams that get interceptions and sack the quarterback are often good pass defenses, even on other plays. The value they create is roughly equal to value created by the big plays.
On offense, it’s a very different story. Interceptions and sacks will always be bad plays for an offense, but their rate of incidence is not strongly correlated to performance on other downs. In fact, if given the choice between a quarterback who threw a lot of picks the prior year but was generally successful otherwise and a quarterback who was very safe but not all that productive, my guess is you will be better of going for the quarterback with the picks.
Special thanks goes to Ty and The Lions in Winter who has been working on a similar line of reasoning for the Lions revamped defensive line.
Potential Future Diaries
Just some ideas I am kicking around or have half started. Let me know what you think about these or any other things you would like to see.
- A follow-up piece on fourth downs digging deeper into how the decision making changes based on the relative strengths of the offense and the opponents defense
- A broader look at “luck”, looking back over the last four years.
- When are extra yards not worth it. The secret dead zones of football.
- Probably not for several months, but a big season preview is in the works.
- Something Carr vs. Rodriguez, now that I have 2006-2007 seasons of data I have two years to compare the two more directly.
- How the best players of the last four years (TEBOW!!!) progressed over the years. Maybe a companion piece on Michigan defenders.
- Any other suggestions? An article a week means I need all the ideas I can get, I’m not afraid to beg!