<Clap> This is pretty good, I like it. </Clap>
"...don't believe something because an "expert" is saying it. Believe it because of the evidence."
While everyone is busy breaking down the scrimmage film with a Jim Garrison-like passion, I thought I would sneak a little preseason preview of some concepts I have been thinking about for how to measure success on a down by down basis. If you want to avoid the nerdy details, skip down for some pretty charts.
Looking at down by down success is a tricky thing and right now there are only limited tools for how to evaluate how an offense is utilizing its most precious resource. The only mainstream tool is third down conversion percentage. This tool’s simplicity is both its weakness and a hidden strength.
Third down conversion rate does not take into consideration how hard your third downs are to convert. Two teams could have identical conversion percentages but if one team has a lot of third and shorts and the other doesn’t the team that doesn’t is accomplishing a much tougher job than the first team. That absence of context is also the hidden strength. Third down percentage isn’t a great predictor of how good your team performs on third down as much as it is an all-encompassing look at how good your team is at getting to manageable third downs and then converting them.
The newer stat that looks at all downs is the Success Rate metric, one I have been on record as not being a huge fan of. Success Rate is a more nuanced look at each down and assigns them a binary pass fail grade depending on whether they meet certain threshold criteria. A binary makes some sense on third down and more sense over the collection of downs, but there is too much opportunity for other value to come and go for the binary to be of major use.
A third way is an expected value (EV). How much value is each team adding or subtracting on given downs. This is a literal value look at ranking teams by what they are accomplishing on a given downs. I have traditionally used this metric but again, it lacks the detail of what is really going on behind the numbers. An EV look tends to lend a lot of value to big play teams and punish consistent gainers. There is evidence to support the rankings coming out that way, but again, I don’t think the numbers tell a good football story in one dimension.
As I began digging into this I pulled all kinds of numbers looking at each of the three downs separately before it dawned on me, first and second down are really a package deal. They are the offense’s opportunity to either do something big or maximize their chances of a third down conversion, first and second downs and typically on the offense’s terms. You can only create big plays so often and even being good at getting in great third downs all the time still means you are having a lot of plays with a chance for the defense to get off of the field. 3rd and 1’s are converted 72% of the time by the offense, so if you get in three of those situations the odds are nearly two to one that you get stuffed on one of them. Being good at avoiding third downs is a better skill for an offense than getting in manageable ones (although both are obviously preferred).
So to that end, I put together three key metrics for an offense for 1st and 2nd downs:
Early Conversion %: Percent of first downs that are created prior to third down. An average team will convert at about 50% with the best offenses closing in on 60%, like the 2011 Oregon offense.
Bonus Yards: This is a big play metric. For the plays that create a conversion, how many yards beyond the sticks does the average play go. Average teams are around 6.5. Mike Leach’s 2005 Texas Tech team was one of the best ever at 9 yards beyond the stick.
Average 3rd Down Distance: The first two metrics are about the successes, historically, most football coaches are more about minimizing the negative. This metric is for them. For the 50% of the time that the average team faces a third down, how many yards are they typically facing. The average team still has 6.5 yards to go on an average third down. Last year’s Air Force team that Michigan faced was the best of the last 10 years with an average distance faced of 4.0 yards for the season.
Now that early downs have hopefully been understood a little better, it’s time to look at third down and focus on a true measure of the down itself. One option that’s sometimes used is to break down the conversion rates into yardage buckets representing short yardage, medium, etc. This isn’t the worst way to go about it, but still isn’t great. Unless its over a large portion of time, sample size problems are likely and you still potentially have problems, although much smaller now, of where do the actuals trials fall into the buckets. Too many buckets and the splits become hard differentiate, too few and there is little continuity to what you are measuring.
To try and solve these issues, here is my suggested stat:
Adjusted 3rd Down Conversion Percentage: Each third down distance has an average conversion rate that looks like this:
1 yard to go converts at 72%, 10 yards to go at 28%. If an offense converts a third and 1, they get +28% for that play. Fail and it’s –72%. Average up all the third downs for a period and you are left with a single number to reflect how a team has done on third downs, that isn’t weighted by being better at first and second down. The other nice thing is that it is naturally anchored to zero. An average team is at +0%. 2011 Wisconsin with Russell Wilson and Montee Ball was the best Big Ten third down team at +16%. 2011 Alabama was the best third down defense at –15%.
Taking all the above analysis, I pulled the results for last season and put them together in a fancy new Tableau table (click to control the view [ed-S: we know; we're working on the links]).
Circle size represents average third down distance
So, Michigan was pretty good on a down by down basis, last year. Only Clemson and A&M where better at third downs when accounting for yards to go. Michigan was also one of the best teams at avoiding third downs altogether, converting on first or second down about 54% of the time.
The other big take away from this is that there are a lot of Big Ten teams at the left hand side of the chart. It’s a bit hard to tell from this view, but Big Ten teams are some of the best at managing third down distance but some of the worst at everything else. Fully half of the teams in the conference are in the lower left quadrant of teams that are bad at both. An offense whose goal is to get into manageable 3rd downs is an offense that is set up to fail.
Michigan lands pretty average across the Big Five conference landscape in both early downs and third downs on the defensive side. The strength of Michigan State’s defense really shows up here, as they only allowed teams to convert before 3rd down about 2 out of 5 times.
I am trying to put together a package of weekly reports and rankings that I can publish online. If anyone has any thoughts as to what you want to see that aren’t otherwise available, I am open for suggestions.
I think these charts do a good job of reflecting what’s happening on a down by down basis. What they don’t show are the impact of big plays and high leverage plays like turnovers and red zone plays.
<Clap> This is pretty good, I like it. </Clap>
Literally grading on a curve. Good explanation, too.
Speaking of Good Stuff (TM) when do we get more from you?
Would it also make sense to lump in 4th down conversions with 3rd down conversions? If you're trying to measure how effective an offense is at keeping drives alive, then 4th down conversions would also measure into this. Of course given the rarity of a 4th down attempt it would need to be weighed by the fraction of times an offense even goes for it on 4th down.
Rather than add in 4th downs, would it make sense to exclude any failed 3rd down conversions in which the offense attempted a conversion on 4th down? I'm able to keep up with the Mathlete's impressive efforts here but don't have the brain to consider whether that serves the same purpose as what you're getting at.
Also curious how downs resulting in a turnover impact the data, if at all. I look forward to future iterations on the topic.
Big plays happen in football when the defense is fooled. Either a player is mis-reading his keys, or the defensive scheme is "solved" by the play (every defense has a weakness). It would be good if you could incorporate into your metrics how "failed" plays set up big plays. The hard part is, sometimes they're set up from previous weeks' outcomes and game preparation, sometimes they're set up on previous series, sometimes they're set up on the previous set of downs, and sometimes they're set up on the previous play.
I would guess that trying to incorporate set-up plays as "successful" would be damn near impossible, and/or take a shit-ton of experienced film study. But if you can do it, it's definitely going to add value to your metrics.
First, this is really interesting and useful stuff. Thanks for putting this together, and all the other great things you do for this blog.
Now the question: Doesn't field position make a big difference in your third down performance? If you are trying to convert from inside your own 40, then (a) the offense is more likely to be conservative in play-calling, and (b) the defense is more likely to know that proclivity - further reducing the third down success rate. Score differential at the time of the play (and time remaining) would have some impact too, it seems. Hypothetically, if you're down by 4+ with little time remaining, your offense will be more likely to call a higher risk / higher gain play call.
Maybe these factors get washed out in a large sample size, but it seems like a team with poor special teams play may be in conservative mode much more often than a team playing with better average field position. I wonder if there's a reliable way to account for these factors in the analysis?
Is it just me or are others have problems enbiggening the second and third charts? I get the dreaded not found error.
Really cool analysis! So cool, I want MOAR:
I'd be interested in seeing a repeat of this analysis, but instead of color-coding by conference, color code by philosophies. That is, are there any trends to where Spread vs. Air Raid vs. Pro-Style teams end up? On defense, maybe it's more about how blitz-happy a team is - do they trend to the lower right? If you could measure average number of DBs on the field per play, do those teams trend to the upper left?
Also, clearly the upper right is best and the lower left is worst for both offense and defense. But if a particular style or philsophy takes you into the upper left vs. the lower right, which is 'better?' That is, which correlates with better PPG/wins/TO ratios? To take it a step further graphically, if you collected data on multiple years could you model a series of PPG curves on these graphs?
Mathlete, you have outdone yourself. Great research, charts, and content.
I had this exact same annoyance with down and distance success metrics out there. I forget exactly what it is but you have to get like 5 yards on first down to be considered successful and that seems completely silly. Glad someone better than me tried figuring out a better way to do things.
I've lurked this site for years and admire your stuff.
I'm an Oklahoma fan and would appreciate any weekly data analysis you provide. I comment at footballstudyhall and wonder if you look at that site.
First off, great stuff. Abosultely love your work.
I had a thought about using color differently. Jeff M has a couple interesting ideas above, but my initial reaction was it would be nice to visualize just how much the factors displayed in the chart (adj 3rd down conversion, early conversion %, and avg third down distance) impact the overall quality of an offense. Perhaps using your preferred method of rating an entire offense. (F +/-, your own metrics, whatever) to differentiate from best all the way to worst in a sliding scale ('Scarlet = Bad' through purple all the way up to 'Blue = Good'?).
Naturally, you would expect more blue in the top right, and scarlet in the bottom left, but it still might be interesting to see.
Awesome read, always enjoy your work. Interesting take on measuring down-and-distance. Here are a few thoughts I had from the post:
- Agree with Jeff M that looking at color coded based on type of offense / defense would be interesting.
- I kept wanting to pull-up the top 25 / Win-Loss and see if this stuff really helps tell a story to how to win football games.
- There is a definite trend in both charts. The better you are at early conversion, the more likely you are to convert your third-down distance. I guess some teams are just better at execution.
- I keep going back to the question of how does this help me coach a football team. What about these metrics help me understand how to strategize a game. Should I go for the homerun play all the time? Does just moving the chains work better? Or are these metrics more of a rating of offenses and defenses to help us judge the performance of players and coaches. Brings me back to Jeff M's idea.
- There also seems to be a trend from third down distance and conversion percentage. That is, teams that have a lower average third down distance are more likely to convert third downs at any distance.
- I'd like to see how the big play metric plays a role here.