Canning a Coordinator: Does It Work?

Yesterday I posted something that in retrospect was too much on the self-indulgent than on the value-add side of things. To make it up to the MGoBlog community, I wanted to do some data crunching. The problem is that there are many posters on the site that do the numbers thing very well, offering some great original analysis that deserves as wide an audience as possible. So I tried to find a niche.

One meme that seemed worth analyzing was "we just need a change in coordinator". This premise is not particular to Rich Rodriguez, as Lloyd Carr, Jim Herrmann, and Ron English could attest. It's of course not even particular to Michigan (see Texas vs. Greg Davis). Fans think a change of coordinator will bring success – but most of the evidence is anecdotal (e.g. Guz Malzahn at Auburn, Manny Diaz at Mississippi State). Of course the success or failure of coordinator changes depends on who is hired, not just on the fact that someone was hired. But I wanted to attempt a more systematic look at the effects of coordinator changes on the performance of that coordinator's unit.

Before you type tl;dr, here's a quick summary of findings: changing the defensive coordinator of a poorly performing defensive unit tends to lead to a modest improvement in the next year. Changing an offensive coordinator is much more of a crapshoot with no clear trends, even if we look at performance two years after the change.

Background I Feel Obligated to Place Here But Feel Free to Skip

The available data is a bit sparse. For performance metrics, I wanted to use the Fremeau Efficiency Index (FEI) from FootballOutsiders.com. That (publicly available) data only goes back to 2007. FEI is great because it eliminates a lot of noise in the performance data, especially strength of schedule and atypical end-of-half drives. For 2010 I'm using the FEI rankings as of the games of 11/27/10 – with so few games this weekend I didn't want to wait until mid-week to post this.

A bit more difficult was data on coordinator changes. In recent years Rivals.com has helpfully posted a "coordinator carousel" listing which coordinators left, where they went, and who replaced them. I took that data and coded the coordinators into four categories – stayed, promoted, fired/demoted, and unknown. A coordinator was classified as 'promoted' if he got a coordinator job at a 'better' school (arbitrarily determined by me, based mainly on the conference of the school) or if he got a head coaching job at any school. A coordinator was classified as fired if he didn't get a new job, took the same job at a 'worse' school, or took a position job at any school. The 'unknowns' are mainly coordinators who went on to take a position in the NFL or the same position at a similar school – in many cases it's hard to determine if that's a promotion or a demotion. Nearly all Michigan fans believe Jim Herrmann's trip to the NFL was 'encouraged', but for some coaches a job in the NFL could be their desired career path. I did some Googlestalking to try to parse out which was which, but if I could find no definitive sentiment I just grouped them into a separate category . This coding was a bit tedious and I would welcome anyone who wants to double-check or validate my coding.

An issue confounding the ability to test the effect of coordinator changes is that they often come with a head coaching change as well. If a head coach comes in with a whole new staff and the FEI metrics improve, is that because of the head coach or the coordinators? So I also separated out coordinators that came on board as part of a new coaching regime (e.g. Malzahn) and those that came on with an existing head coach (e.g. GERG).

Finally, we know that there's a ton of other factors that can affect unit performance besides a coordinator change, the primary one being the players on the field. Did the best players graduate, or did an inexperienced group get more seasoning? Was recruiting on the upswing or the downswing? Those factors are not addressed here (see, I told you I wasn't the Mathlete).

Actual Results: Aggregated

So, our dependent variable is the change in the team's FEI ranking for the unit from the last season under the old coordinator to the first season under the new coordinator. We have 359 records (120 FBS teams by 3 years, except since Western Kentucky in 2008, their first year as a FBS school). Here's a quick look at the aggregated results:

In the aggregate, across all types of coordinator changes, offenses tend to improve by 1.3 spots in FEI rank if their coordinator doesn't change, and decline by 2.3 ranking spots if the coordinator does change. For defenses there is almost no effect, and no differentiation between units with a coordinator change vs. those without a coordinator change. This is almost the essence of a null relationship.

Actual Results: Broken Out by Type of Coordinator Change

But coordinator changes should make a difference, right? The only way to tease that out is to break out those coordinator changes by whether the coordinator was part of an all new staff, whether he was pushed out, or whether he was promoted to a better job elsewhere. If we do that, we start to see some more sensible relationships:

We're not talking huge sample sizes, here (the smallest is 28 defenses where the old coordinator was promoted). But what we see makes some sense, at least for the defense. On average, if a defensive coordinator was fired, those team's units improve in FEI rank by 11.2 points the following year. In contrast, if the old defensive coordinator was promoted (because he was deemed to be good at his job), the defense declines by 4.3 points in FEI rank.

What about the offense? Here, it's the same pattern but not quite as extreme. If the offensive coordinator is promoted, the team declines in FEI rank by 8.0 spots. If he's fired, the team improves slightly by 2.1 spots. [I did a quick check to see if performance for the offense improves markedly in year 2 after a coordinator change, and couldn't find any compelling evidence that it does.] Note that all-new staffs seem to be more harmful to the defense than to the offense. And, as we might expect, teams have a hard time replacing coordinators who were good enough to get promoted to a better opportunity.

Actual Results: Broken Out by Previous Season's Rank

But we have to be conscious of regression to the mean. Coordinators tend to be fired from poor-performing units. The terrible performance of that unit may be due to the coordinator, but it's also likely due to some outside factors, including luck. Just as it's hard to be the #1 team year in and year out, it's hard to be the #120 team year in and year out. So a team that finishes terribly in one year is likely to improve its performance the next year, even if the coordinator remains (if that puts me on record as saying that if GERG stays Michigan will finish better than 104^th out of 120 teams in Defensive FEI in 2011, so be it).

The question is, once we control for the performance of the units, does a coordinator change seem as beneficial? In other words, if team A has a terrible offense and doesn't fire its coordinator, while team B has a terrible offense and does fire its coordinator, does team B tend to improve more than team A? With a fairly sparse data set we can't get too specific with our controls, so I simply sliced the data into thirds based on their previous year's rank for the offensive or defensive unit.

If we look at the top 40 teams in FEI, all teams decline from one year to the next. This is the essence of regression to the mean (and competitive parity). We have to be very careful here because sample sizes are small (not many coordinators change if the team is performing relatively well, though tell that to Georgia's Willie Martinez). Among top 40 teams in FEI, the best thing is continuity of staff – and even there, teams are likely to lose 18 or 11 points of rank.

Now we see some more intuitive results. Presumably coaches are less apt to tolerate a bad performance if they don't think their coordinator is the right one for the job. Among teams in the middle of FEI performance, firing a coordinator tends to lead to a slight improvement in FEI rank over and above what is seen if the staff remains the same.  Defenses are up 6.7 spots in FEI rank vs. a 0.3 point decline if the staff doesn't change, and offenses are up slightly (+3.0 vs. +1.3 for same staff). This does not mean that all teams would be better off firing their coordinators -- just that there is some juice to the conventional wisdom that a canned coordinator can be replaced with a better alternative -- at least for the defensive side of the ball.

Note that replacing a promoted defensive coordinator has slightly more benefit than replacing a fired coordinator – not sure what to make of that, except that again sample sizes are small (8 teams over 3 years fell into this category – the bar with the 10.9).

Finally, what if your unit stinks, and ranks in the bottom 40 in FEI?

We have really small sample sizes for the "old coordinator promoted" groups, as you'd expect (why hire a coordinator from a bad team?), so interpret those results with caution. The main areas to examine are the far right and far left of the chart. First, the offense. If an offense is terrible but keeps its coordinator, it tends to improve 23 spots in the FEI rankings. If a terrible offense replaces its coordinator, the FEI rank only goes up 9 points. Contrast that to the defense; keeping the DC of a terrible unit leads to a 12 spot increase in FYI rank, on average, while firing him leads to a 24 point gain. You might argue that it takes longer for changes in offensive coordinators to show a benefit, but in the few cases where a new offensive coordinator lasts to year 2, there's no clear evidence for that claim.

Fun fact – note that replacing a staff entirely leads to about the same change as keeping the same staff. Two possible interpretations – any staff would do better after a terrible year, or athletic director's are incredibly foresightful and know which underperforming staffs to fire and which to retain. In any case, here we're seeing the conventional wisdom hold true for defenses (a canned coordinator outperforms his predecessor), while not so much for the offense. You could argue that the offenses among canned coordinators would have done worse if the coordinator had stayed, but given the improvements for the other types of offensive coordinator changes it's not the likeliest explanation.

Double Bar Charts! What Do They All Mean?

The gist of these charts is that firing an offensive coordinator seems to have no clear positive effect for the team's performance, either in year 1 or year 2. In fact, teams seem to be better off if they keep their offensive coordinator. It appears that swapping out an offensive coordinator could be an indicator of something seriously wrong with the program that a mere offensive change can't fix (see Tommy Tuberville and Tony Franklin at Auburn).

On defense the story is different. Getting rid of an underperforming coordinator appears to pay clear dividends. These benefits aren't monumental – about 7-12 or so extra spots of FEI ranking over keeping the coordinator, but they're consistent.

What Does This Mean for Michigan?

Who knows? This analysis is aggregated, and mileage will seriously vary based on the team and the coaching staffs. On average, given Michigan's horrible defensive FEI in 2010, getting rid of GERG and replacing him with another coordinator would lead to a jump of about 24 points in FEI ranking, while leaving him as DC would lead to a jump of about 12 points in FEI ranking. Whatever happens at Michigan would vary from this weak 'prediction', but I put it out there in any case. The most optimistic scenario, as we'll see below, would be a jump of 75 spots in the FEI.

Bringing in a whole new coach is a different animal – and again shows the danger of looking at means instead of specific situations. But based on the averages we see, a new coach would lead to a 34 spot drop in offensive FEI rankings and a 14.5 bump in defensive FEI rankings, while keeping Rodriguez and canning GERG would lead to an 18 point drop in offensive FEI rankings and a 24 point bump in defensive FEI rank. Again, this is a very weak prediction based on aggregated data, and is not the main purpose of this diary. The purpose was to see if changing a coordinator is a cure all, a bandaid, or an empty act of desperation. It appears that for offenses, a coordinator change is at best a bandaid, while for defenses it may be more of a cure.

If you're curious, here are the top 3 best and worst coordinator cannings where the head coach DID NOT change, based on changes in performance year-over-year, from the  coaching offseasons after the 2007, 2008, and 2009 seasons:

For those curious, Michigan replacing Scott Shafer with GERG ranks as the 6^th-worst firing of a sitting DC, looking strictly at one-year changes in unit performance.

Final Notes

This analysis is far from perfect, and I welcome any and all feedback. I am somewhat concerned that the 'best' firings all seem to be from the 2009-2010 season. Not sure if that's a function of using FEI data from before the end of the season, or if head coaches are paying more attention to coordinator fit, if there are flaws with my coding, or if it's just a fluke.