2 and 3 Star Athletes: Drafted at Same Rate When Placed in Same Environment

3*s is more than 2*s. in fact, it's ONE whole star more than 2*s.

I have a problem with your speculations here and elsewhere about the quality of coaching being the only critical element in differentiating draft rates at high and low tier schools.

Differences in the ambiguity of ratings, not the quality of coaching, could account for the different draft success rates of lowly rated candidates at high and low tier schools.

That is, suppose that a two star that goes to a top school is accepted in the top school because his rating is ambiguous: little reliable information was available from recruiting services; so he may look like a boom or bust candidate. Whereas, a reliably rated two star about whom much is known may be accepted at a low tier school even though he clearly is a two star and has little upside.

Here is an extreme illustration of the point.

Suppose that a player will not be drafted unless he is ranked at the end of his college career as the equivalent of either a four or five star, based on his performance in college.

Suppose that at the low tier school, out of ten recruits, the number of two star recruits being ranked at the end of their career at a given star level has a pretty tight distribution, reflecting the reliability of the initial recruiting rankings:

Rank of ten typical two star recruits, revised according to performance, by the senior year:

1 star, 2 players
2 star, 6 players,
3 star, 2
4 star, 0
5 star, 0

The average senior rank = 2.0, the same as the predicted 2 star rank when entering school. However, the result is that no one achieves 4-5 stars by is senior year, so no one is drafted.

But at the high tier school, the distribution of senior performance ranks of two star players is more spread out, reflecting the initial ambiguity of the rankings:

Rank of two star recruits, revised according to performance, by the senior year (note, there are a lot more busts, but also more high performers):

1 star, 6 players
2 star, 1 player
3 star, 1
4 star, 1
5 star, 1

Here, again, the average senior performance rank = 2.0, the same as the predicted 2 star rank when entering school, also the same as that achieved at the low tier school. However the result differs: 2 players or 20% of the two stars are drafted.

Conclusion:

The average performance of senior two star players could be no higher at the top tier than at the low tier schools—as if there were no net effect of coaching in helping players get drafted---yet the high tier school has more draftees. The reason for the difference, in this example, requires only that the the two star recruits at the high tier schools had more ambiguous ratings than the two stars at the low tier schools.

Thanks for taking the time to do it.

no one needs to "resist the temptation" do they? Correct me if I'm wrong, or not understanding your point here, but all--not some, all, of the respectable data points, published here and referred to by Brian many times, show that in fact that recruiting ratings are in fact, correlated nicely to team success and likelihood of being drafted. They cause and predict nothing, but in fact, the higher you are rated the higher your chance of being an excellent player, and your team an excellent team.

P.S. I do want to commend you for all the work, at the same time that I can't believe that an analysis of seven teams can in any way be definitive or statistically significant. Correct me if I'm wrong there as well.

P.P.S. I do agree with your assertion that we should stop ragging on our 2 stars, just not for the reasons you state.

and 3 vs. 4 star splits for as many teams as you can get.

Three points:

1. Mike Hart was a three star, so...

(kidding)

2. You (NOLA Blue) are wrong that the percentage of a population doth a good sample size make. It's all about the absolute size, baby. That's why good polling outfits sample 500-2000 people both from states and from the U.S. as a whole.

3. Nevertheless, 521 is pretty damn good.

4. Okay, four points. It seems to me like the strength of your argument is that highly-ranked players go to the schools with the best facilities/coaching/hostesses, so it is impossible to tell whether that same player would have done as well at a more poorly ranked school (Donald Rubin's fundamental problem of causal inference).

In that respect, it seems like you'd want to stratify the BCS schools into quality quartiles or something, sample a few from each, and run the analysis. I didn't get the logic behind the choice of schools.

Alternatively, you could, say, take all the three- or four-stars, and regress their probability of being drafted (or a fancier Tobit model where the DV is draft position, censored if the player wasn't drafted) on their rankings, plus a variable capturing the quality of the school, and see which one wins out.

5. There is no fifth point. Except that I love this shit. Keep up the good work.

if we're looking to figure out how good a recruit is, our best bet is to infer via offers and commitment? In that case, we could generate our own star rankings if reported offers are accurate enough. I think someone may have already done prestige star rankings here, actually. But if the recruiting services are giving us this information, they should have incorporated those facts into their rankings already, no? Even if they don't bring any scouting prowess of their own to the table, they should be able to deduce which recruits the coaches--whose capacity to scout are to some degree known--like best and rank accordingly. Perhaps it could be done better mathematically, but I assume they're close. And if not...that speaks pretty badly of them.

Before I start, I appreciate that this must have been a labor of love and taken a great deal of time to compile.

I don't fault your methods, and the findings speak for itself. What I don't understand is how at some point between demonstrating that there's very minimal difference between 2 and 3 star athletes as far as NFL draftability and the end of your dissertation, you come to equate being drafted with being a contributor.

You conclude:

"please stop dismissing our 2-stars. It is obvious that they are just as likely to be contributors as our 3-stars."

That's a nice sentiment, but it's not what you showed.

It's a long leap over an enormous gray area to get from "be drafted" to "be a contributor."

So much goes into being an NFL draft choice that doesn't relate to contributions on the field. Careers and millions of dollars are made based on tenths of a second in the 40 yard dash and single inches of height.

Just off the top of my head and with a quick scan of rivals I found that, in your time frame, 2 star recruits Jared Zabransky (QB) and Ian Johnson (RB) went undrafted, and thus wouldn't have registered for your study. Nobody familiar with BSU would ever claim that those two weren't contributors for Boise, maybe two of the biggest.

I love your data and your effort, but what seems really important is how the guys in each group who aren't absolutely elite pan out against each other. At what rate do the remaining ~90% of 2 stars and ~90% of 3 stars become undrafted contributors? Of course, what defines "contributor" is going to be highly subjective and require a whole 'nother level of data mining.

But, frankly, as Michigan fans, I think what we all care about is the rate at which the coaches are turning our lower star players into celebrated (or at least solid) contributors (ala Zabransky/Johnson), not how many of them are defying the odds to make The League (however nice it is for them :P).

NFL draft status is an easy and practical measure for collegiate success but not a particularly thorough one.

NOLA, thanks for all of the work. It is a very interesting analysis.

One idea, which already be covered, is when a 2-star player goes to a BCS program, is he inherently a better player than a 2-star that goes to a non-BCS program BECAUSE the coaches at the BCS programs are better evaluators of talent than the recruiting services.

This is a awesome post, I think our three stars are going to surprise a lot of people.

This post is amazing. I love posts that question established wisdom through statistics.

I have one concern, though. By using Utah, Boise, and TCU as examples, you're committing to an analysis of players that, by definition, performed surprisingly well. To make the jump from a collection of (mostly) 2-stars to a top-10 program, a team needs to have quite a few players that exceeded all rational recruiting expectations and performed like blue chips.

These "surprise" players are, of course, the ones who get drafted. So already, you're selecting a group of surprisingly good 2-stars and 3-stars to evaluate, because in looking at Utah, Boise, and TCU, you're probably looking at the three best collections of 2-star and 3-star recruits in the country. It seems possible (or even probable) that when you take such an abnormal population of lightly-recruited players, the "normal" recruiting correlations might not apply.

Of course, the rebuttal to this is that these guys are surprising players, but more 3-stars than 2-stars should be surprising if ratings mean anything. I guess the response to that would be to look at the raw numbers of 3+ star athletes at these schools. For instance, Boise only had 10 3+ star players over this time period. To do as well as they have, they would need more than 10 good players... so by selecting them, you've already selected a team with abnormally good 2-star recruits.

I don't think this flaw is fatal to your analysis. But I'm really eager to see if this finding holds up when you evaluate some more average teams.

This is a great analysis, thanks! My one worry that the coaching stability requirement may distort the results. All the teams studied except for Cal had significantly better records than their recruits' rankings would suggest likely.

To me, being drafted is too large of a bucket for this comparison. There is a pretty big difference between the first and last round. I suggest weighting their draft status by round (or multiple round buckets). If three stars are being drafted two rounds ahead of two stars, then they aren't really equal. Your sample size is probably not large enough at this point for this kind of resolution.

You also take away a lot of resolution by averaging the percentages drafted. At the very least, give us the standard deviation. I'm forgetting the specific test you would want to use, but you can show whether or not the two percentages are statistically different. You might have done this and I just missed it while skimming the article.