The Mathlete

Last month I took a look at how the recruiting services stacked up when it came to predicting future college success. With the latest NFL Draft in the books I wanted to review it to see if there were major differences between services predicting the draft versus college success.

Even though NFL draft position doesn’t always equate to success at college, it is an indicator of talent and there are very few instances were great college players don’t ultimately end up in the NFL, mainly highly efficient noodle armed quarterbacks. Even rushing quarterbacks typically find their way onto draft boards where the Kellen Moore’s and Case Keenum’s of the world sign on as undrafted free agents.

Biggest Hits & Misses

Here are the players from the first three rounds that one service rated significantly different than the other two.

ESPN

ESPN had significantly more outliers than other two services. Luckily, the hits outweighed the misses. ESPN was the sole advocate for five first rounders and had more hits (9) than the other two services combined (7). The Worldwide Leader also racked up the misses at a record pace, missing on six top picks.

Hits

Morris Claiborne, Ryan Tannehill, Luke Kuechly, Kendal Wright, Whitney Mercilus, Courtney Upshaw, TJ Graham, Mohamed Sanu, Sean Spence

Misses

Matt Kalil, Mark Barron, Nick Perry, Mitchell Schwartz, Cordy Glenn, Isaiah Pead

Scout

Scout had only four players from the first three rounds that they significantly diverged from the other services on. Their divergence was centered on receivers as three of the four players they deviated on were wide outs.

Hits

Justin Blackmon, Stephen Hill, Olivier Vernon

Misses

Alshon Jeffery

Rivals

The players were different but Rivals hit and miss counts were virtually identical to Scout.

Hits

Quinton Coples, Brandon Taylor, Jamell Fleming, Jayron Hosley

Misses

AJ Jenkins

Round by Round Scorecard

How each round went, points given for whichever service had a draft pick rated highest (must be in the top 1000 ranked). Numbers listed as Rivals/Scout/ESPN.

Round 1: Push 8/9/9

Round 2: Rivals 11/4/6

Round 3: Rivals/ESPN 8/3/8

Round 4: Scout/ESPN 6/12/10

Round 5: Scout 5/9/3

Round 6: Rivals/ESPN 5/3/6

Round 7: Scout 8/12/6

Overall the wins were pretty evenly balanced across the services. Rivals taking at least a share of the first three rounds would probably give them the overall edge. They also had the lowest overall average ranking of draft picks.

Scorecard by Class

Class of 2007: Push 18/18/15

Class of 2008: Scout 22/29/26

Class of 2009: Rivals 10/6/5

Despite their overall strength, Rivals struggled in the main class of the draft, 2008.

The look at the NFL draft results versus the recruiting rankings mostly reinforce each other. Based on the data from both looks, here is how I would describe each service’s strengths and weaknesses.

Rivals: The best and most consistent performer. Year after year producing output at or above the level of any other service.

Scout: A very good second to Rivals. Similar consistency to Rivals and an overall level that is close but still a notch below. A quality second voice.

ESPN: The all or nothing service. Demonstrated that overallocation to the SEC that isn’t necessarily supported by the NFL draft. Blatant neglect of west coast recruits but willing to deviate from the other services, with mixed results.

[Ed-Ace: Brian (knee) is day-to-day, though he did prepare some content that will be posted this afternoon. Post-Burke-return hoops stuff and a Spring Game primer will appear later this week. In the meantime, enjoy some Mike Hart.]

In honor of Michigan’s all-time leading rusher’s birthday yesterday, a look at one of the unique careers in college football.

Since the 2011 season completed, I have been re-loading 9 seasons worth of games (6,063 to be exact) to update my database to include 2011’s new feature of Win Percent Added. In doing so, something immediately popped out at me. No running back added more wins to their team than Mike Hart did for Michigan.

Sometimes when you are looking at advanced stats you are surprised by how counter-intuitive results can be and sometimes you are surprised how well the data fits the existing narrative. Mike was the back who wouldn’t go down, always got the extra yard, killed the clock and never fumbled. Those are all the things that factor highly in Win Percent Added, especially the 4th quarter capabilities. Burning the clock in the fourth quarter is a key requirement of a successful running back. Especially a Michigan running back. No one did it better than Mike.

For his career, Mike Hart was responsible for 4.4 Wins running the ball. Reggie Bush edges him out if you count receiving WPA, as well, but those are tainted wins. It’s not just longevity and playing time that pushed him to the top. His per game average of 0.11 is fifth, behind two players with only a single season in the database and two more with two seasons at non-BCS level schools.

Freshman Season

At this point, writing about Mike Hart is a daunting task. What is left to write that hasn’t been written? He joined the team in the 2004 class as a 3 star recruit. He nearly set the national high school rushing record but wasn’t even the highest ranked running back in Michigan’s class. He would have been the fifth highest rated running back in Miami’s (YTM’s) recruiting class. He saw his first quality action in his second game of his career against Notre Dame in the second week of the season. By week three he was over 100 yards and posting a +5 EV+ and a crucial .36 WPA as Michigan held on for a 24-21 win over San Diego St.

Hart would go on to string together three straight 200 yards games in Big Ten play, including a 0.26 WPA in the Braylon Edwards game. His EV+ was always strong for a running back but where his EV+ was strong, his WPA was Herculean. Mike Hart made all the plays to win the game but none of them to lose them. By the end of the 2004 season true freshman Mike Hart had gone from anonymous three star to posting a per game WPA of 0.15, still my best recorded number in the Big Ten.

Sophomore Season

Injuries killed a large portion of the 2005 season. Kevin Grady, Max Martin, Antonio Bass and Jerome Jackson all took carries but none could come close to the production from Mike Hart. Kevin Grady was the only one to surpass a +1 EV+ in his absence, and that was mostly unnecessary against Indiana. Jerome Jackson did have a solid 0.14 WPA on 11 carries in an overtime win against Iowa, but that was limit of the success when Hart was out. In five full games of action Hart averaged 0.23 WPA which if replicated across an entire season would have given him the second highest (Reggie Bush, 2005) WPA average in a season for any running back since 2003.

It’s hard to think about what could have been with a healthy Mike Hart. Three carries in a seven point loss to Notre Dame, a DNP in a three point loss to Wisconsin eight ineffective carries in a four point loss to Ohio. There’s a very real chance he swings those three games and Michigan shares a Big Ten title with Penn State and spends its holiday taking on Florida State in the Orange Bowl rather than getting screwed over by the refs in the Alamo Bowl.

Junior Season

With fewer games coming down to key fourth quarter possessions in 2006, Michigan didn’t need the fourth quarter machine Mike Hart. He finished the season with a profile almost exactly like Chris Perry’s 2003 season. With not much in the way of close games, he didn’t have any massive, WPA pushing games like he had in his first two years, but 10 of 13 games would finish at .07 or better. For the year Hart ended at .09 WPA/game, his third top 20 Big Ten WPA year in as many tries. John Clay is the only player to have even 2 top 20 finishes.

Senior Season

For the second time in his career, injuries would derail an outstanding Mike Hart season. After surviving The Horror and somehow managing a strong WPA in the follow-up beating by Oregon, Hart was on track for a season to along side his junior year. An ankle injury in mid-season cost him a couple games of action and a couple more of effectiveness. 2007 would be his lowest rated season but still crack the Big Ten top 50. He would finish the year with enough quality carries to become Michigan’s all-time leading rusher and set the then non-existent WPA record.

Uniqueness

When I talk to people about how much more valuable quarterbacks are than running backs they usually point to running out the clock in the fourth as the unquantifiable equalizer between the two. When I first developed the Win Percent Added I was anxious to see how true it was. If you properly value the ability for a running back to keep the clock running and close out a game, what happens to the value relationship between quarterback and running back. After I crunched the numbers I found that the fourth quarter benefit was largely overstated. Until I looked at Mike Hart. There are very few running backs whose value is truly magnified by the little things like the narrative claims.

Mike Hart is the narrative.

Appendix

Mike Hart, Seasons

Mike Hart, Games

As Michigan enters year #3 with Denard starting the season at the helm and year #2 under OC Al Borges, I did a deep dive into teams that have made the leap into offensive greatness. A handy guide to going from a good offense to an elite offense in one year.

In making predictions you are always safer predicting things to regress towards the mean. I wanted to look at the teams that have gone from good but not great offensive teams to truly elite teams in one season. There were some surprisingly strong correlations within this group.

1. A quarterback with experience returning.

2. The same offensive coordinator as the previous year

3. Continuity in your receiving core

These three three aren’t sufficient conditions for making the leap, but they are necessary conditions.

From 2004 to 2011 there were 20 teams, including 2010 Michigan, that increased their offensive EV+ rating by at least 4 points per game and ended the year above 10. I wanted to take a look at these 11 teams to figure out what was necessary to make a jump like this, what might seem necessary but wasn’t and how many teams fit this same criteria but couldn’t make the leap.

Returning Quarterbacks

It’s not rocket science to say that having a returner at quarterback is a good thing, but the exceptions to this rule prove it as much as anything. Of the 20 teams to make the leap, five didn’t technically have a returning starting quarterback. Two, Auburn 2010 and Stanford 2009, did it with future first overall draft picks. Wisconsin 2011 technically counts but Russell Wilson was a three year starter for NC St before arriving in Madison. Tim Tebow got quality field time on a national championship team despite Chris Leak’s position as starter. The final exception was Oregon 2010 when Darron Thomas stepped in to great success.

The returners themselves take all forms. There are runners like Josh Nesbitt from Georgia Tech in 2009, Pat White at West Virginia in 2006. There are pass-first athletes like Robert Griffin on Baylor 2011 and Vince Young on Texas 2005. There are future first round picks like Sam Bradford at Oklahoma in 2008 and Matt Leinart at USC in 2005.

Three times it was done with multiple quarterbacks seeing action. Michigan in 2010 saw returning starter Tate Forcier make regular appearances during the Denard Robinson injury hour every Saturday. Cincinnati didn’t miss a beat in 2009 when Zach Collaros stepped in for injured starter Tony Pike. Arizona State had the same story in 2005 when Rudy Carpenter replaced the injured Sam Keller.

How many years at the helm wasn’t a significant issue. There were as many teams who made the leap with second year starters as third and fourth starters combined. The only team to do it with a non-transfer four year starter was Northwestern in 2005 under Brett Basenez. Zac Robinson did it in his second year at Oklahoma St in 2007, Chase Daniel did it at Missouri in his third year of four running the offense at Missouri.

Not all the QB’s developed within the program, either. Cam Newton, Russell Wilson, Aaron Rodgers and Ryan Mallett all accomplished the task after transferring from a previous school or Junior College.

Offensive Coordinators

On average, about 35% of BCS programs turn over their offensive coordinators in a given year. For teams making the offensive leap it drops to 10%. Charlie Weis found his famed “Decided Schematic Advantage” in 2005 (only to lose it thereafter) with Brady Quinn. Arkansas and Ryan Mallett where the only others to accomplish the feat, but the presence of head coach and offensive guru Bobby Petrino indicates the continuity was likely high.

Returning Receivers

Teams making the leap returned 75% of their value from their receiving core, nearly fifty percent higher than the average team (53%). Only four teams returned less than 60% of their receiving value from the prior, and those teams all managed to make the leap due to ultra efficient passers, Tim Tebow, Russell Wilson, Aaron Rodgers and Vince Young.

Missing The Cut

Returning Running backs

My crusade to bury the position of the running back continues. Teams making the leap where all over the map on returning RBs. Texas and Northwestern both made the leap in 2005 with virtually no non-QB carries returning from the 2004 season. Baylor, Michigan, Florida and Cal all did with only marginally used carriers returning. On the flip side USC returned Reggie Bush and LenDale White in 2005. Georgia Tech, Oregon, West Virginia and Missouri all returned over 80% of the previous years carries.

It’s not that having a key ball carriers returning is a bad thing, it’s just not necessary to make the leap.

Recruiting Rankings

Just like the running back question, the recruiting rankings are mixed bag. Cincinnati, West Virginia, Cal, Northwestern and Baylor all made the leap without the benefit of a roster full of 4 and 5 star recruits. USC, Auburn, Florida, Texas and Oklahoma always enter a season stacked. Better recruits = better results is true but Better Results = Better Recruits is often incorrect.

Odds of Making It

Of teams that meet criteria 1-3 above (with 3 being defined at 60%) and are within 1 standard deviation of +10 EV+ (4.4 or higher) 25% make the leap to +10 or higher. 40% of this group improves but not to an elite level. That leaves 35% of teams to regress under this environment. The worst offender is Michigan State in 2006. Poised to become a potentially great offense under Drew Stanton, Sparty fell apart and dropped from a +7.2 in 2005 to a –2.6 in 2006.

Can Michigan be that team

With the loss of Junior Hemingway, Kevin Koger and Martavious Odoms Michigan falls below the 60% threshold (38%). All of the other criteria fit nicely for Michigan and when it has been done without the aid of returning receivers, its come on the back of a Heisman level quarterback performance. If year 2 of Al Borges Denard Fusion Cuisine comes together, that doesn’t seem out of the realm of possibility. I won’t be on record predicting a leap like year for the 2012 Michigan offense, but they are one of a handful of schools that I would even consider.

Evaluating coaches is a tricky thing. Ultimately it comes down to wins and losses but even comparing one situation to another in the unbalanced world of college football is a tricky proposition. Mike Shula has a higher career winning percentage as a head coach than Brady Hoke. However Hoke has spent all but the last year at non-BCS schools where Shula was at Alabama. School prestige, resources and recruiting all play major roles in team success along with coaching. Many of them often go hand in hand but I think I am finding some ways to parse out different pieces of the puzzle independently. This is my first of hopefully many off-season looks at coaches, and who at excels at what parts of coaching.

The Data

To evaluate how coaches develop and evaluate talent I needed a way to separate out better inputs (recruits) from the output (team success and draft placement). Team success is a viable way to look at it and at some point I would like to circle back to compare PAN and recruiting for a comparison, but for today’s exercise I am going to look at recruiting ranking to draft position.

Time Period

The main challenge with this method is that draft placement is such a lagging indicator from recruiting. Since only some of the 2007 recruits and most from 2008 on have yet to be drafted, I am only looking at recruiting classes from 2002-2006.

Evaluating Recruits

I have now been able to add all four recruiting services to my database. Since we are only looking at classes up until 2006, that means just Scout and Rivals for all years except 2006 when ESPN came on board, as well. Recruits are given a number value based on national rank, position rank and stars. Each year has 25,000 points assigned across all players so the early years with fewer players have their individual ceilings a bit higher. Consensus 5 star players are typically 50-60 pts. Generic three stars are in the low teens and below. Anyone without a position rank or less than 3 stars is zero points.

Here is Michigan’s 2012 class for reference.

Evaluating Draft Picks

Because of the much higher value to higher draft picks, the draft pick evaluations are fitted using an exponential formula.

500*e^(-Pick#/50)

This works out to about 500 for the first pick and then each round is half of the same pick in the previous round (1st pick in second round about 250, 1st pick in the third about 125, etc.). This puts the total points for a 255 player draft at 24,600, almost identical to the total for a year’s worth of recruits.

Coaches

Players are counted towards the coach that recruited them. This will only be somewhat an evaluation of player development since the coach gets “credit” for the player they recruited even if they leave the next year. I have also restricted the search to coaches with at least 1,000 total recruiting points over the five year period. This is about equal to two top 15 classes or five top 50 classes. This gives us 43 qualifying coaches to review.

The Show

First thing I did was look at each coach and how many recruiting points they accumulated versus how many draft points they had.

The first thing that jumped out at me was that there seemed to be a strong correlation between total recruit points and total draft points. This is going to be true to some extent, but it seemed that ability for the top schools to load up wasn’t properly accounted for. So I plotted the two versus each other and found a very strong correlation was present.

Since we are looking for more on talent evaluators and developers than MOAR 5 stars, I used the correlation between the two to adjust recruiting points to give a more fair comparison between the lower end and the top end. This allows for a more common evaluation tool between elite programs/recruiters and the rest.

Now we have something to talk about.

One thing that jumped out at me was that NFL guys did seem to have a bit more success. Maybe their buddies were just doing them favors, but there are a lot more guys with NFL experience at the top than the bottom. Oh, except for the big guy coming in last at #43. Weis’s monster class of 2006 (934 team points, my #7 class of the last 11 years) yielded two 6th round draft picks. His first class which was much less regarded still only yielded a single fourth round draft pick. In the words of our fearless leader, #MissYouBigGuyXOXO.

Lloyd Carr comes in just below average on the adjusted scale. Barry Alvarez checks in at #1 among Big Ten coaches and #2 overall. Wisconsin’s lineman machine is real. The evil genius Nick Saban is #3 based on his last three classes at LSU. Ohio coaches new and old round out the top ten.

Of the nine elite recruiters (3,000 or more adjusted recruiting points) Pete Carroll and Jim Tressell come out on top, with Phillip Fulmer close behind. The bottom three are all southern coaches: Bobby Bowden, Larry Coker and Mark Richt.  Bob Stoops, Mack Brown and Lloyd Carr make up the middle third.

Ted Roof takes home the prize for most recruiting prize without a single draft pick with 515 points and nothing to show for it. Top performers who missed the cutoff included Dan Hawkins, Bret Beliema’s first class, Ed Orgeron, Mike Stoops and Greg Schiano.

Other Notes

Many thanks to all who have helped populate the recruit database. We are 25% of the way done.

Still have lots of ideas for future posts including the final post on how to use game theory to maximize success based on the overvalued running back and success rates. If there is interest, I would like to do a retrospective on previous seasons through the eyes of advanced analytics and throw up some of the best WPA graphs of the season. Hopefully I can start with 2003 in the next month. I am open to any ideas you have out there, as well.

If you are on the twitters follow me at @the_mathlete. I am trying to post little snippets that aren’t quite column worthy there. Recently I have tweets about which state’s recruits stay in-state the most (Utah and Arkansas) and least (NY/NJ and Hawaii) and used my recruiting points ranking to list the top 4 Michigan high schools in producing 3* or better talent (Cass Tech, OLSM, Detroit Renaissance & FHH), correctly guessed by @Joshua_Block.

Thanks to all that helped build the coaching database. Now it's time to move on to recruits. I have uploaded all available recruiting sites databases back to 2002 in an effort to connect them to team rosters. Of 16,865 recruits, I have connected most of them to players in the databse. However there are about 2500 players still unconnected. Some of them were academic or legal casualties, some of them were transfers. Most of them are offensive lineman that never showed up on the play by play in the first place. 

For those who consider themselves recruiting and or Google ninjas, I can use your help. I have listed the players, school they signed with and year they signed for all the missing entries. Whatever info you can help fill in would be a great help. There are more instructions in the spreadsheet and feel free and contact me with any questions you might have. My email is in the instructions. Thanks again for everyone's help.

Recruit Database

Previously: Predicting team success based on recruit rankings

Before signing day I took a look at how team recruiting rankings were predictive of future success. I found that good defenses almost always come with good recruits, but on offense great offense often comes without being fully stocked, although it doesn’t hurt.

This week I wanted to look more at the individual level by comparing recruiting rankings to draft success. For most positions college success is going to translate well into future draft status. Michigan might have the biggest exception to that rule in Denard Robinson (although some think he might be a top WR pick). For almost everywhere on the field but rushing quarterback, college success and production are highly correlated to NFL stock. It’s not perfect but it’s a great place to start.

The debate on do recruit rankings matter rages on. Dr. Saturday, may he blog in peace, annually refreshed his look to affirm their accuracy. Rarely do you find anything resembling an analytical take down but from even the best writers on college football can come the anecdotal dismissal. Hopefully those of us who prefer to use data have already won you over and this can be a nice look at some of the ups and downs within the overall success of recruiting rankings. If you’re there yet, hopefully you are after you read this.

The Data Sets

On the recruit side, the pool of players will be the recruiting classes of 2002-2006. All but 2-3 of those players have had their shot to be drafted between the 2005 and the 2011 drafts. I will only be looking at the players who were ranked for their position, as well. This means I have all 4 & 5 stars and the best of the 3 stars. I excluded fullbacks and specialists because the numbers are pretty low and they are mostly all 3 stars or less.

It’s All in How You Word It

There are two key arguments against recruiting rankings. The first is the one used by Bruce Feldman in his recent article on Stanford linked above. It’s the yeah but what about…argument. Ignore recruiting rankings because Stanford is good. Ignore recruiting rankings because JJ Watt is good. There of course exceptions. There are plenty of flameouts and come from nowhere success stories but this is a volume game and the exceptions don’t disprove the rule.

The second argument is the famed failure to divide. Here are two true statements:

If you are drafted, you are more likely to be a three star or less recruit than four or five star.

The more stars you have the more likely you are to be drafted.

The first statement is used by opponents of rankings but isn’t really a relevant statement. The second is the key point. If every single five star was drafted, there would still be six times more three stars and below drafted than five stars. Because four stars and above are so selective they can’t win the quantity game but they dominate the likelihood game. The NFL is full of unheralded recruits but for every five start there are literally hundreds of unheralded recruits playing college football. The pool just starts much bigger.

Tell Me Something I Don’t Know

So at this point we can all agree that recruiting rankings matter, right? If you’ve made it this far you’ve earned a chart.

Percent of Recruits Drafted

*Position based on recruited position, not drafted position

Across all positions, each additional star more than doubles your likelihood of being drafted. It’s not only true in the aggregate but at the position level, as well. There isn’t a single position where a 3 star recruit is more likely to be drafted than a four star. And this is a self-selected group of 3 stars and not the entire pool. In almost every case, a fifth star is another large bump from 4 stars. OLB, OT and WDE are virtually equivalent between 4 and 5 stars. Even a largely college specific position like Dual-Threat QB (RQB) and undefined positions like Athlete show the same trend.

The top positions for 5 star success are Athlete, DT, ILB and Safety at over 60% and the tight end position which was a perfect 4/4 in getting 5 stars drafted.

But getting drafted is only half the story, the other is draft position.

Average Pick For Drafted Players

At the position level, the draft spot doesn’t hold up quite as well as the previous chart, but overall there is a strong trend favoring the higher starred players. On average, a drafted five star player will be picked in the middle of the third round, nearly a full round ahead of the average four star player and another 17 picks ahead of ranked three star players.

On twitter on Friday I teased a question about which position did five stars underperform four star counterparts. There is actually a position on each side of the ball. On defense it’s outside linebackers that don’t follow the trend and on offense it’s the tackles.

I think it’s interesting that Rivals has struggled to match top high school talent at position like tackle, outside linebacker and defensive end at the rate they have at other positions. Despite the weakness at these positions, similar positions like guard, inside linebacker and defensive tackle have had their rankings hold up quite well.

The Takeaway

Don’t get too hung up on the magic of the fourth or fifth star. They are a nice aggregation but there isn’t going to be much difference between the last five start and the first four star. The bottom line is the higher ranked a recruit is the better they are likely to be, with plenty of exceptions. Positions like tackle, weakside d-end and outside linebacker the difference between a four star and a five is almost negligible. And there are no guarantees. Loading up on top talent gives you the highest likelihood of having team success and successful individuals, but when you get down to the specific player level it becomes a crapshoot. More 5 stars players never hear their names called than ones who do. For four stars it’s still a nearly 4:1 chance against getting drafted.