CC: Ranking player development

Submitted by stubob on

Inspired by this Bill Connelly post at sbnation about Nebraska, I wanted to look at the upcoming coaching change in terms of developing talent, and exceeding or underperforming expectations. In the article Bill asserts that comparing performance ranking to recruiting ranking is a reasonable way of determing performance.

So I took that theory and applied it to Michigan's past coaches, some of our leading candidates, and our competition to see what I could see. I tried to use the most recent tenure as possible, to show comparisons relative to each other. One thing that stood out early was that almost every coach underperformed expectations, but with most of these coaches recruiting at top-10 level, it would be hard not to. Without further ado, here's what I found.

Michigan Men

Let's start at the very beginning, it's a very good place to start.

coach school years recruiting avg f+ avg +/- career recruiting average career f+ average career +/
Hoke Michigan 2011 26 9 17      
Hoke   2012 6 20 -14      
Hoke   2013 4 37 -33      
Hoke   2014 20 58 -38 14 31 -17
Rodriguez Michigan 2008 11 73 -62      
Rodriguez 2009 12 58 -46      
Rodriguez 2010 17 47 -30 13.33333 59.33333 -46
Carr Michigan 2004 6          
Carr   2005 5 12 -7      
Carr   2006 9 6 3      
Carr   2007 8 28 -20 7.333333 15.33333 -8

Not much we don't already know. Carr started tailing off, Rodriguez improved from a terrible start, and Hoke has fallen off after an outlier of a start. One interesting way to view this would be year-by-year, since obviously Carr's last year preceeds Rodriguez's first, etc.  Michigan's recruiting has been very good, except for Hoke's first class, which still almost ended up top-25. Performance has been great (but not exceptional) to, well, I don't need to remind you how bad 2008 was (worst of any analyzed here).

The Contenders

coach school years recruiting avg f+ avg +/- career recruiting average career f+ average career +/
Harbaugh Stanford 2007 44 66 -22      
Harbaugh   2008 45 58 -13      
Harbaugh   2009 18 31 -13      
Harbaugh   2010 24 6 18 32.75 40.25 -7.5
Mullen Miss St 2011 34 49 -15      
Mullen   2012 22 62 -40      
Mullen   2013 25 33 -8      
Mullen   2014 35 5 30 29 37.25 -8.25
Patterson TCU 2011 30 18 12      
Patterson 2012 29 31 -2      
Patterson 2013 35 44 -9      
Patterson 2014 42 4 38 34 24.25 9.75
Miles LSU 2011 8 2 6      
Miles   2012 14 10 4      
Miles   2013 6 17 -11      
Miles   2014 2 14 -12 7.5 10.75 -3.25
Pelini Nebraska 2011 16 28 -12      
Pelini   2012 30 19 11      
Pelini   2013 22 39 -17      
Pelini   2014 36 25 11 26 27.75 -1.75

Yep, Stanford went from underperforming to outperforming expectations, even as recruiting improved. Performance went from average to excellent, and recruiting went from above average to good. That's pretty good performance in both categories, and will be hard to beat for the candidates we've selected.

At this point, I couldn't decide whether to order by who's best or who's most likely to accept if offered. Since we're all dreaming at this poit, I decided to go with who's best.

Dan Mullen started off with above-average recruiting, and improved some, but not much of the last four years. But the performance improvement is amazing. His team went from average to elite in two years. How much regression would we expect next year? Well, their four year average is about 37, offset against an upward overall scoring trend.

Gary Patterson is pretty much the poster child for over-achieving. Recruiting in the mid-30's, with teams outperforming by almost 10 spots. It's easy to see why people are trippig over themselves to see what he'd do with a top-10 recruiting clss. Recruiting hasn't gotten much better, but he's been there for so long, and competing against Texas and everyone else for recruits, it's tough to imagine them rising much higher than they are. Performance had been sliding backwards until this year, but still inline with expectations overall.

Next up, somewhat surprisingly is Bo Pelini. I was somewhat surprised that Nebraska fired him, especially after winning their last game, but the writing has been on the wall since the open-mike comments last year. Pelini has performed about inline with expectations or better, which was a little surprising. I didn't realize how far down Nebraska was in the recruiting rankings, which does explain their performance a bit. Recruiting in the mid-20s, performance in the mid-20s, trending improvement. We could certainly do worse.

The Hat. Looks like the consensus on him is pretty accurate. Great recruiter, good on the field, trending downward. Now, part of that is due to their great class this year, which would indicate a potential bounce-back coming, but that's what the numbers say now. Still only -3 relative to expectations, which seems pretty good.

The Competition

coach school years recruiting avg f+ avg +/- career recruiting average career f+ average career +/
B Kelly ND 2011 9 13 -4      
B Kelly   2012 18 7 11      
B Kelly   2013 5 26 -21      
B Kelly   2014 11 34 -23 10.75 20 -9.25
Meyer OSU 2012 5 14 -9      
Meyer   2013 2 9 -7      
Meyer   2014 3 3 0 3.333333 8.666667 -5.33333
Saban Alabama 2011 1 1 0      
Saban   2012 1 1 0      
Saban   2013 1 2 -1      
Saban   2014 1 1 0 1 1.25 -0.25
Muschamp Florida 2011 12 34 -22      
Muschamp 2012 4 4 0      
Muschamp 2013 3 48 -45      
Muschamp 2014 9 39 -30 7 31.25 -24.25
Dantonio State 2011 32 11 21      
Dantonio   2012 33 15 18      
Dantonio   2013 37 6 31      
Dantonio   2014 25 15 10 31.75 11.75 20

Obviously Muschamp isn't competition any more, but I included him to compare with Hoke. Florida had better recruits, but has been unable to do much with them, 2012 excepted. As a result, he comes in with a score between Hoke and Rodriguez, and, not surprisingly looking for a new job.



Urban Meyer is improving a loaded OSU team, from underperforming to meeting expectations in just three years. They've had top-5 recruiting classes, and the performance is starting to match. No reason to expect otherwise next year, unless the coordinators get poached (please, everyone poach their coordinators).



Nick Saban, well, what is there to say? Number one recruiting classes all four years, number one performance all four years. Technically that's five years of recruiting, since so many recruits don't make it to campus, but those are the numbers I have. Newsflash: Alabama is good, has been good, and probably will be good.



Brian Kelly at ND was a bit of a surprise. I didn't realize how far down their performance was. Next year could be make-or-break for him, and if their performance continues to slide, I'm not sure where he would wind up next. I was actually considering him as a possible candidate, but after looking at the numbers, I'm not convinced. We know he can recruit, but seems to be having trouble putting it on the field



Finally, Mark Dantonio. Tip of the hat to you, you crazy nutball. Recruiting in the mid-30s, top 15 performances. That's pretty clear evidence of outperforming expectations.

Conclusion

So after looking at year-to-year performance, I charted the overall relative over/underperformance of each coach. Here's the result.



So, what does all this tell us? Well, it's easier to outperform expectations when you're lower in the recruiting rankings. If you have a top-10 class, there's not much room to outperform. It's really about meeting expectations at that point. Harbaugh is the only coach I profiled with consistent improvement, but as far as our best available candidates, can you believe it's Dantonio, Patterson, and Pelini?

Comments

stubob

December 5th, 2014 at 3:15 PM ^

http://www.footballoutsiders.com/stats/fplus

It's a combination of two other advanced metrics, FEI and S&P+.

FEI: A scoring rate analysis of the remaining possessions then determines the baseline possession efficiency expectations against which each team is measured. A team is rewarded for playing well against good teams, win or lose, and is punished more severely for playing poorly against bad teams than it is rewarded for playing well against bad teams.

S&P+ takes into account success rate, equivalent points per play, drive efficiency, and adjusts for opponent quality.

I chose it as a way to measure relative success/failure of a team, without using simpler stats like yardage, record, or points. All these advanced stats take those factors and create a ranking from them. I specifically chose F+ since it's a combination of two other advanced stats (like I used 247's aggregate recruiting rankings).

theintegral

December 5th, 2014 at 3:34 PM ^

Are you plotting class ratings from the class of 2011 with the F+ score of the same year?  Those recruits are likely not even playing or having very little playing time.  

Rating scores for teams are base on the total ratings of everyone recruited....more recruits, greater score.  Thus, 2012 and 2013 are quite good and 2014 is a 26.  The average recruit's score in 2014 is as good as the previous years.

Dantonio's is so good because you are comparing skills of recruits from 2014 to the skills players who were recruited in 2010 and 2011 and spent 4-5 years getting better.  MSU has the oldest, most experienced starters in the nation.

If you want to rate player development, you have to look at who is actually playing and the outcomes.  There are better metrics already in place presented on this site.

 

stubob

December 5th, 2014 at 5:22 PM ^

The coaching tenure of these guys is so short, almost nobody gets long enough to see it. I mean, this was Hoke's first year with All His Players. The trend should still be there, if you can truly "coach-up" your players, the performance improvement should be there from the start. That's the idea - you have a known quantity, in this case the recruiting rankings of the players, and can compare them between coaches.

I also figured that with a top-5 recruiting class, the odds would be in your favor of getting some talented freshmen on the field (like Peppers, except for his knee).

ShadowStorm33

December 5th, 2014 at 7:01 PM ^

I have to agree with this; I really don't think this analysis tells us much. I think you'd have to do some kind of weighted, blended recruiting number to capture who's on the team, and who's playing, and then compare that to the performance. For instance, in Harbaugh's first few years, the core of his players came from the 2004-06 recruiting classes, which ranked, on Rivals, #54 in 2006, #41 in 2005 and #106(!) in 2004. Factoring those in, instead of simply the incoming freshmen, the numbers get much closer to 0 than negative teens or twenties.

Plus, this also doesn't take into account the state of the players when a coach arrives. Again, looking at Harbaugh, the year before he arrived Stanford's F+ was #104. Even using your method, with say the Rival's ranking for 2006 (#54), they were -50 the year before Harbaugh arrived, and immediately jumped to -22 in his first year. That team was a steaming pile of shit when Harbaugh arrived, and you can't just waive a wand to make that go away. The huge jump in F+ (and therefore your +/-), without a significant improvement in recruiting, tells me a lot more about his year than simply saying the team's performance lagged quite a bit behind the incoming recruiting class.

Everyone Murders

December 5th, 2014 at 3:36 PM ^

That's amazing.  With his classes, there is nothing but potential downside (given his classes are always at or near No. 1).  Miles is impressive too, although with lower classes not as strikingly so.  Being good at both bringing in talent and meeting expectations over time to me equals good coaching.

I hate myself for acknowledging this, but kudos to Dantonio.  He has outperformed in part because of a great defensive system that relies on great DB play and consistent play elsewhere, but the formula is working.

Anyway, nice diary.  I hope you update this when Michigan names its head coach (or better yet, I hope you don't have to because of Harbaugh - although I'd take Mora in a second).

tolmichfan

December 5th, 2014 at 10:43 PM ^

I like the attempt of this analysis. Personally I hate advanced stats. I'm my opinion unless you are going to change the final scores due to tempo then I don't get why it matters. Yes they are helpful at looking backwards and what happened but people on this board put to much "stock" in them.

Second with the variance in schedule from year to year and team to team it's so hard to just use stats to compare coaching staffs. For example in Dantonios fourth year that everyone gushes over, his best wins were against a bad Michigan team and a bad Notre dame team, and Penn State was in its downward trend. Then they went and played a Bamma team that isn't close to as good as it is now, and they got crushed. That MSU team didn't even play OSU at all that year.

It's important to take the eye test when comparing stuff in college. Then you need to use numbers to back this up. Also Michigan under RR and even more under Hoke had to deal with so much attrition that neither coach could overcome it. It's not really fair to say this years senior class was rated as high as it is when a ton of those guys didn't make it to their senior years. Of the seven scholarship players that graduated this year ( not including kickers) only two of them were 4 star or higher players.

Their should be a way to go back and reevaluate the rankings of the class do to attrition and the scouting services just fucking up the rankings. After watching him play would any of us call Will Camble a 5 star player. I wouldn't, after watching him I would reevaluate him down to a 3/4 star type. He had the size he was just lazy.

jbibiza

December 6th, 2014 at 8:16 AM ^

Thanks for the effort. Of course it would be more accurate to compare the results of the third year against the recruits of the first year as pointed out by The Integral in #4 above. Since you are tracking player development, you could use the recruitng classes from the previous coach for the first 2 - 3 years.  Thus Harbaugh's onfield results for 2007 would be compared against his predecessor's 2004 recruiting class (as noted by ShadowStorm 33).

Also, and very significant, is that using class ratings is a distortion, whereas average stars is a much more accurate (albiet imperfect) measure.

alum96

December 7th, 2014 at 4:31 AM ^

I saw a post about 2 months ago where someone did all this work (didnt use FO data) - will try to find it via the google.  Basically plotted the teams with the most overperformance and underperformance vs rolling average of previous 5 classes or whatnot of recruiting.

Yep here it is, deadspin

It's actually a lot of work for the person who put it together.  As expected we did poorly.

http://regressing.deadspin.com/chart-which-ncaa-football-teams-outplay-…