Recruiting Bias and Accuracy

[ED: Moved to the diaries. This obviously took some work beyond the level of a standard post. ZL]

Brian recently commented on the awesome post by UpUpDownDown over at BHGP that analyzed the teams and conferences that are best at developing their recruits into NFL players. 

Part of UUDD’s argument is that player development (and, in particular, playstyle) is a driving factor behind the Big Ten outperforming (and the Big 12 underperforming) expectations with respect to defensive players and offensive lineman.  Brian had an alternative/additional explanation: a combination of recruiting service bias and difficulty in evaluating high school lineman.

I think there may another element at work: scouting services overrating certain sections of the country and underrating others, particularly the Midwest. Rivals (the source of the rankings used) doesn't even have a Midwest analyst. Meanwhile, OL rankings are particularly inaccurate since many high school kids need to put on 50 pounds before they can play in college. The flipside—skill position players more easily projectable—sees a much, much lower spread amongst conferences. The worst-performing conference is the ACC at 94% of expectation; the best is the Big East at 108%. That's a much lower spread than you see in the D and OL numbers, one that looks like an even distribution distorted by a little randomness.

If there was a regional bias in recruiting rankings, hard-to-evaluate OL would be the place it would show up most prominently. I think there is. Your ratings are just wrong when Wisconsin has two four-star linemen in the last five years, as they do on Rivals. They are not evaluating linemen correctly. I'm not sure what Big 12's hole of suck on defense represents but I'd be more convinced it was a playstyle thing if they were running 3-3-5s or something. Going up against Blaine Gabbert and a bunch of other passing spreads doesn't make much difference to anyone but a few linebackers, it seems.

Not content to let our fearless MGoLeader’s assertions hang out there without poking around the data a little bit, I asked Mr. UUDD for his dataset* and set to work determining (1) whether Midwestern recruits are underrated by the recruiting services, and (2) whether offensive lineman are comparatively more difficult to evaluate. 

Specifically, I looked at (1) whether non-5 star Midwestern recruits outperform the “percent drafted” expectations for their star ranking,** suggesting that Midwestern recruits are underrated, and (2) whether the spread is smaller among the “percent drafted” numbers for offensive line recruits relative to all recruits, suggesting that the rankings are relatively less accurate.

Midwestern Recruits Slightly Outperform Expectations

The first piece is that there is a bias by the recruiting services against Midwestern recruits because the services spend relatively less time and resources tracking the Midwest. That bias translates into lower recruiting rankings for Midwest recruits, resulting in underrating of those recruits. Chart:

Midwestern recruits of the 2-4 star variety slightly outperform draft expectations relative to their peers from other parts of the country.  However, the sample sizes here are way too small to reveal whether or not this difference is significant.

Of course, the chart doesn't disprove my mildly paranoid belief that Midwesterners are consistently being slighted by the jerks on the coasts, so let's call this a win. 

Note that the Midwestern 5 star recruits underperform the mean. This has no impact on the claim (5 star recruits can't be underrated), but it's interesting nonetheless. Really small samples for 5 stars is all the explanation I need. 

Stars Matter Less for Offensive Line Recruits

The second piece is that the big boys are harder to evaluate because they are less prepared for college football than their smaller brethren.  Offensive lineman in particular often need a redshirt and a whole lot of S&C before they can show potential. Thus, recruiting rankings for offensive lineman are less accurate because the evaluation essentially comes down to "he's big and does not apparently soil himself."

Once again, the data is consistent with the claim, but not at statistically significant levels. The spread between the chances of being drafted as a 2 star offensive lineman and a 5 star offensive lineman is much smaller than the spread for all positions. In other words, stars may matter less for the big guys, but we need more recruiting cycles to know for sure.

* Huge, huge thanks to UpUpDownDown for sharing his work. As I found out very quickly trying to replicate the dataset, the data is extremely difficult to cross reference because a lot of recruits have the same name or slightly modified their name during their college career. 

** Note one small wrinkle in the dataset: players that are eligible to declare for the draft, but haven’t, are counted as undrafted.  Thus, a number of players from the recruiting classes of 2008 and 2007 that will eventually be drafted are nonetheless included in the denominator, but not the numerator, in the percent drafted numbers.

Edit: More Fun

In response to comments, the following charts reflect the overall percent drafted for only the 2002-2006 recruiting classes, and the N values for each set. I agree that including '07 and '08 players that haven't declared isn't ideal, but I wanted to be able to compare apples to apples with UUDD's analysis.

N Values