The Impact of Sacks and Picks on Overall Performance
[Editor's note: frontpaged for obvious reasons. A scheduling mix-up with Brian caused this to get buried earlier, so I'm bringing it back near the top. [How much] Will Michigan rue the loss of Brandon Graham? - Tim]
[Note: I have 2006 fully loaded into the database now and will be included in all future multi-year studies along with 2007-2009.]
We can all agree that sacks and interceptions are good things for the defense and bad things for the offense. But how does a viable pass rush or a ball-hawking secondary affect the performance of the opposing offense on plays where there isn’t a sack or a pick. Likewise, what is the correlation between an offensive line that gives up sacks regularly or a mistake prone quarterback?
Defense
Sacks and interceptions have very similar direct impacts on games. From 2006-2009 in games between two D1 teams in competitive game situations (the “universe” for this and most of my analysis) the average defensive unit produced 2.3 ppg worth of sacks and 2.0 ppg worth of interceptions. Sacks have a slightly higher direct value than interceptions (interceptions returns and fumble returns on sacks are not included) but does either of these correlate to a better defensive performance overall.
Chart time? Let’s make it a double.
Not entirely surprisingly, the better a defense is at producing sacks and interceptions, the better it is on downs where neither occur.
For every point per game that a defense generates due to sacks, the overall pass rush generates 1.2 ppg of additional value. Interceptions are also powerful, but not as much so. Each ppg of value a defense generates through interceptions is worth 0.9 ppg of additional value.
This analysis serves to confirm what most football fans already know. Teams that can create interceptions and sacks are good going to be better defensive teams. Whether a strong pass rush/secondary creates pressure on other downs or if strong pass rushes and secondaries are a common occurrence on great defenses is irrelevant. As most of you probably know, defenses that are good at these two things are also good on other downs. So why is this interesting…
Offense
The story becomes very different when you look at offenses. The conventional wisdom that was supported for defenses is largely blown up on the offensive side of the ball. Sacks and interceptions may be indicators of great defenses, but they are not symptoms of bad offenses.
The slope of these two charts are about 20% of the gradient of the corresponding defensive charts and virtually flat. On offense, the amount of sacks and interceptions are largely independent of performance. There is obviously the immediate negative effect of the play, but giving up sacks or throwing interceptions show virtually no correlation to success or failure on other downs.
What it means?
For one of side of the ball it merely quantifies conventional wisdom. Good pass defenses get interceptions and sack the quarterback and teams that get interceptions and sack the quarterback are often good pass defenses, even on other plays. The value they create is roughly equal to value created by the big plays.
On offense, it’s a very different story. Interceptions and sacks will always be bad plays for an offense, but their rate of incidence is not strongly correlated to performance on other downs. In fact, if given the choice between a quarterback who threw a lot of picks the prior year but was generally successful otherwise and a quarterback who was very safe but not all that productive, my guess is you will be better of going for the quarterback with the picks.
Special thanks goes to Ty and The Lions in Winter who has been working on a similar line of reasoning for the Lions revamped defensive line.
Potential Future Diaries
Just some ideas I am kicking around or have half started. Let me know what you think about these or any other things you would like to see.
- A follow-up piece on fourth downs digging deeper into how the decision making changes based on the relative strengths of the offense and the opponents defense
- A broader look at “luck”, looking back over the last four years.
- When are extra yards not worth it. The secret dead zones of football.
- Probably not for several months, but a big season preview is in the works.
- Something Carr vs. Rodriguez, now that I have 2006-2007 seasons of data I have two years to compare the two more directly.
- How the best players of the last four years (TEBOW!!!) progressed over the years. Maybe a companion piece on Michigan defenders.
- Any other suggestions? An article a week means I need all the ideas I can get, I’m not afraid to beg!
He didn't just pull these numbers out of the air. These come from actual games. Trying to put the numbers into a single game wouldn't say anything because how many data points in one game will you have to compare it to? 2? 3, maybe?
The complaint isn't with the sample size. The number of games used is great, actually. The complaint is that the data shows little to no correlation. It implies that sacks and ints are completely unrelated to defense quality, and he's using the slope of the lines of best "correlation" to draw conclusions. It's kind of like plotting defense quality versus the length of my dog's most recent dook, finding out there's no relation, and then trying to draw conclusions about the "best" relation that statistics can come up with (except in the Mathlete's case, it's actually interesting that there's no relation between sacks/ints and defense).
or if it's more that there's just a lot of outliers. If you removed the 10 furthest outliers both above and below the line, it might look a little more uniform. Though you may have a point. Statisticians are notorious for Beautiful Mind Syndrome (seeing patterns and significance everywhere).
Mathlete,
How about doing a post where you're actually watching a game and try to incorporate some of the principles you've been discussing.
How is luck, per your findings, factoring in to the actual game in front of you. Or does any of this sack/int stuff come into play and how. I think you'll find some stuff that supports your work and others that make it murkier.
I dont know if this is making any sense and it probably is dumb because it involves the small sample size of just one game, but I dont know, I think there is value is focusing on a single game. Or do a series breaking down Michigan's year.
Or watch all of the northwestern games and give me something tangible regarding their luck.
Actually, since I wrote a post claiming they arent the residue of luck. maybe we can somehow watch some NW games together and do a post where we go back and forth. All in good fun.
Just some ides.
Those R-squared's are still pretty freaking low in terms of how much one explains the other. Right? I mean isn't .30 the typical benchmark? .07 is the highest any of these charts shows.
Granted that could be because 1,000 variables are at work here, so .07 really isn't that small, but I don't know exactly.
When academic-style work is presented to a non-academic audience. Freakonomics was so popular because a non-academic writer was brought in to "translate" academic work for a broader audience. To some extent, we need something like that here - perhaps a co-author for The Mathlete who understands the study and can write it in a more broadly accessible fashion ... or we need a separate board for those of us who want to consume and analyze material like this in a rigorous fashion.
I love this blog, but I have serious reservations about whether this is the right forum for pieces like this, in its current form. I think The Mathlete's work is *outstanding* - deserving of a ton of praise - but it would benefit from a robust analysis by those with the tools to do so (similar to peer reviews in academia).
Until this formula is proven incorrect:
The team that scores the most points wins the game
the rest is white noise.
Or, as John Madden once said, "the team that scores the most points will probably win the game."
There was a little bit of back-and-forth with the Mathlete and myself. I've been trying to test a hypothesis of Lions fans: that the improvement of the defensive line will result in a dramatically improved pass defense, despite a still-wretched secondary. I asked the Mathlete to check out my work so far, and he provided a lot of meaningful feedback and insight. I know he's done a lot of work you're not seeing here.
Unlike me, I think the Mathlete was just generally exploring the relationships between sacks and pass defense, and INTs and pass defense. Without a hypothesis to test, and with present-but-mild correlations, these charts doesn't have the WOW factor of the fourth-down analysis, where we see breathtaking, indisputable, dramatic conclusions that dovetail perfectly with what we might have suspected but are stunned to see in maize, blue, and white.
Peace
Ty
A theoretical question:
At what point do all these statistics temper a simple enjoyment of the game of football, as well as common sense in watching the game?
If Justin Turner intercepts a ball and puts the O back in the game at the opposition's 10-yard-line, I KNOW that play is way more important than a sack Richard Ash might get with 2 minutes left against Bowling Green when Michigan is up 70. Not according to this post, though.
I've always held the belief that there's no difference in quality of defense between 3-4, 4-3, 46, 335, etc. Rather, the performance of the defense is more due to the coach and having the players to run that system. It would be interesting to quantify this, but I realize it may not be feasible given the work load involved. Obviously a fair analysis would be to compare only college teams and then factor in the # of years under that system & DC.
I can't wait to see how many sacks we get out of the 3-3-5. It seems like one of those defenses that throws a lot of blitz packages out their and with the athletes we have it will be fun to watch.
I think this diary raised some very interesting issues and led to a good discussion. It's a shame The Mathlete didn't come out to play.
Comments