2 and 3 Star Athletes: Drafted at Same Rate When Placed in Same Environment

Submitted by NOLA Blue on
Normal 0 false false false EN-US X-NONE X-NONE /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin;}

I’ll post the results first, since I understand some people have jobs and spouses… reasons, methodology, discussion are all below.

In short, I analyzed 4 recruiting classes (2002-5) and the corresponding 5 NFL drafts (2005-9).

Likelihood of being drafted from Boise St, TCU and Utah: 

3-stars 8.00%;  2-stars 9.28%

Likelihood of being drafted from Cal, Iowa, Oregon, Va Tech: 

3-stars 12.99%;  2-stars 12.20%

The rationale for the use of these schools and their separate grouping is given below; grouping involves facilities and initial selection involves coaching stability.  Also, the BCS schools (Cal, Iowa, etc.) will be used again in the soon to be posted analysis of 4-stars vs. 3-stars and the effect of 5-stars on their fellow teammates.

As you can see, there is no difference between a 2-star and 3-star athlete, as far as talent and potential can be measured by draft status.  When placed in the hands of a capable coaching staff (all of the above universities) and a similar environment (BCS with high level of facilities, or BCS crasher with decent facilities) there is no appreciable difference in draft status.  I now beg you all to stop bagging on our 2-star recruits, it not only shows a lack of respect but from this point forward will have to be considered a lack of basic comprehension.  An analysis of 521 athletes ranked with 2- or 3-stars, and the subsequent 57 draftees to come from this pool, has proven the system flawed for 2- and 3-star athletes.  Please feel free to refer anyone who remains ignorant to this fact here, so that they may either argue with the methodology or inform their thought process.

 

Now, to begin…

A few weeks back, I wrote two diaries (here and here) regarding the feasibility of recruiting services being able to dependably differentiate between athletes from the top 0.23% of high-school football players.  (If you don’t need a refresher of the findings, please skip to the next paragraph.)  The first post looked at the fact that the talent pool (number of total high-school football players) has grown each year consecutively since at least 1988, while the total number of active scholarships for football at NCAA Division 1-FBS schools grew at one-third the rate (fluctuating between 9,095 and 10,115 scholarships.)  Scholarships available basically grew by 11%, while the pool of high-school football players grew by 31.6% over the same period.  Thus, we see a shrinking percentile from which schools are drawing their talent (parity, anyone?)  The second post was an analysis of the 2009 NFL draft, looking at the impact of “Rivals’ Top 100” athletes from the 2005 recruiting class on total draftee production.  The results of the 2009 draft showed that teams could be broken down into 4 categories:  those which produced at least 4 draftees, those with 2-3 draftees, and those with less than 2.  Thus, when considering that a random spread of draftees across all Div 1 college teams would result in 2.15 draftees per team, it is easy to see that there are “over-performers,” “average-performers” and “under-performers” as far as draftee-production is considered.  The other factor considered was teams’ acquisition of “Top 100” (Rivals) recruits in 2005:  twenty-three of the nation’s college teams accounted for the signing of 86% of these athletes.  Thus, there was a clear delineation between recruiting rankings:  again “over-performers” and “under-performers.”  The result of comparing these groups:  there was no correlation between being a highly successful recruiter of “Top 100” talent in 2005 and producing NFL draftees in 2009.  This was admittedly a very narrow slice to analyze… and yes, I had admitted that upfront.

So, after reading these diary posts there were many great comments and ideas put forth by the MGoBlog community.  One idea in particular caught my attention:  both “brax” and “4roses” suggested analyzing the correlation between stars and draft-status from among players placed in the same environment.  This would be the only way to extract a correlation that accounted for, and negated, the recruiting services’ currently common practice of bumping up athletes’ star ratings after they commit to big-time BCS schools.  The goal would be to isolate the effect of differences in coaching and training facilities to get a true bearing on recruits’ talent and potential, and whether or not “stars” actually peg innate talent/potential differences.

Right now the central argument of “star” supporters is “hey, look at the rate of drafting for 5 stars, 4 stars, 3 stars and so on…  True, there is a clear regression at work there.  But, it does not necessarily speak to the star system predicting talent and performance; rather it is very likely that the stars are predicting which schools recruits go to.  There is an obvious difference in facilities among schools, and there is clearly a shuffle of coaches that generally flows in one direction (towards U of M, Florida, Texas, etc.)  This creates a potential gap in player development, and should heavily affect draft-status.  If the administration of “stars” were to be biased toward awarding extra stars to recruits heading to big-time schools, then it would lose its validity as a measure of actual player talent and potential.

So, the analysis…

Idea:  Cross reference players’ star-rating at time of signing day with their ultimate rate of selection to the NFL.  Compare the rates for each star-level within a small sample of schools deemed to be similar.

Time Period:  I retrospectively evaluated the star-ratings of players from four consecutive recruiting classes (2002-5.)  Their draft-status was referenced to five consecutive NFL drafts (2005-9.)

Parameters:  To be included in the 2-star vs. 3-star evaluation, a school has to have had a high-number of both 2 and 3 star recruits through the four recruiting classes; as well as a significant number of draftees over the five year period (in this case, Boise set the low bar with only 7 draftees over that period.)  Also, in considering the pool of 2- and 3-stars, “kickers” were not included in the count.

Sample selection:

The first group is a group of three schools which consistently compete at a high level, without the benefit of BCS caliber facilities, and consistently place players into the NFL.  They also have had stable coaching situations.  For this pool of players, I examined Boise St, TCU, and Utah.  I acknowledge Utah had the least stable coaching situation, considering Urban Meyer’s departure.

The second group comes from the larger pool of BCS schools with high draft-success.  Thus, for teams taken from the BCS level, I had the luxury of being able to use a limiting-requirement of coaching stability; only schools with no head-coaching change between 2002 and 2009 were used.  This reduced the available pool:  Cal, Florida St, Georgia, Iowa, Ohio St, Oklahoma, Oregon, Penn St, Texas, USC, and Va Tech.  For today’s comparison between 2- and 3-star athletes, the list is further pared down to the schools with a high number of 2- and 3-star athletes:  Cal, Iowa, Oregon and Va Tech.  This group is important because they have a high number of 4-star recruits as well; meaning they will be a bridge between the analyses of 2- vs 3- star recruits and 3- vs. 4-star recruits.

Data:

Boise

Tot 4

Tot 3

Tot 2

1

9

76

4 Draft

0

3 Draft

1

2 Draft

5

%

0%

11%

7%

 

TCU

Tot 4

Tot 3

Tot 2

5

21

53

4 Draft

1

3 Draft

2

2 Draft

7

%

20%

10%

13%

 

Utah

Tot 4

Tot 3

Tot 2

2

20

65

4 Draft

1

3 Draft

1

2 Draft

6

%

50%

5%

9%

 

Cal

Tot 4

Tot 3

Tot 2

21

36

27

4-Draft

5

3-Draft

4

2-Draft

3

%

24%

11%

11%

 

Iowa

Tot 4

Tot 3

Tot 2

13

37

34

4-Draft

0

3-Draft

3

2-Draft

7

%

0%

8%

21%

 

Oregon

Tot 4

Tot 3

Tot 2

16

42

34

4-Draft

1

3-Draft

4

2-Draft

4

%

6%

10%

12%

 

Va Tech

Tot 4

Tot 3

Tot 2

17

39

28

4-Draft

3

3-Draft

9

2-Draft

1

%

18%

23%

4%

 

Results:

BCS Crashers…

Tot 4

Tot 3

Tot 2

8

50

194

Draft 4

2

Draft 3

4

Draft 2

18

Tot %

25.00%

8.00%

9.28%

 

4-, 3- and 2-star Analysis of Cal, Iowa, Oregon and Va Tech…

Tot 4

Tot 3

Tot 2

67

154

123

4 Draft

9

3 Draft

20

2 Draft

15

Tot %

13.43%

12.99%

12.20%

 

Discussion…

First, resist the temptation of looking at the drafting of 4-stars from the BCS crashers as legitimizing the star-system.  Eight athletes from a four-year recruiting cycle does not bring nearly enough statistical power to make any claims.  Therefore, do as I have done:  ignore them.  In the comparison of 3- and 4-star athletes we will look at a much larger sample size from schools which have enough 4-star recruits and NFL draftees to power a comparison.  So, save those comments for that analysis.

Second, for those of you still reading this, I have included the 4-star data for the BCS teams; so you get an advanced view of the 3- and 4-star analysis.  Pretty crappy reward for your reading efforts?  Yes… but I appreciate those efforts nonetheless.  :^)

Third, a bit of repetition… but with more definition.  As you can see, there is no difference between a 2-star and 3-star athlete, as far as talent and potential can be measured by draft status.  When placed in the hands of a capable coaching staff and a similar environment, there is no appreciable difference in draft status.  We can see that while each school has a fairly large sample of 2- and 3-star recruits to analyze (among the schools considered, 81 per BCS-crasher and 69 per BCS) the number of draftees from an individual school would not allow enough power.  Hence, the need to group the schools.  I left the schools in two separate groupings because of two factors:  one, it allowed a consideration of the effect of different facility levels on player development; secondly, the BCS group will be used as a bridge to the 3- vs. 4-star analysis. 

We can see that the rate of drafting from the middle-echelons of the BCS for 2- and 3-star athletes is approximately 50% higher than the rate of drafting from the top echelon of non-BCS schools (12%-ish vs. 8%-ish.)  I generally relate the BCS crashers’ success to the high-quality of coaching staffs they have assembled and been able to keep.  It would be interesting to see if there is a large difference in facilities at these schools when compared to Cal, Iowa, Oregon and Va Tech.  If the gap is minimal, then the difference in drafting above could be considered to have more to do with media exposure and proven play against higher-performing competition.  It is an interesting question, and probably pretty difficult to investigate.

So, an analysis of 521 athletes ranked with 2- or 3-stars, and the subsequent 57 draftees to come from this pool, has proven the system flawed for 2- and 3-star athletes.  I already made my feelings known at the start of the post, but I will say it again… please stop dismissing our 2-stars.  It is obvious that they are just as likely to be contributors as our 3-stars, and as Michigan Men they deserve our support regardless of contribution.

Go Blue!

Comments

michelin

February 21st, 2010 at 11:39 AM ^

I have a problem with your speculations here and elsewhere about the quality of coaching being the only critical element in differentiating draft rates at high and low tier schools.

Differences in the ambiguity of ratings, not the quality of coaching, could account for the different draft success rates of lowly rated candidates at high and low tier schools.

That is, suppose that a two star that goes to a top school is accepted in the top school because his rating is ambiguous: little reliable information was available from recruiting services; so he may look like a boom or bust candidate. Whereas, a reliably rated two star about whom much is known may be accepted at a low tier school even though he clearly is a two star and has little upside.

Here is an extreme illustration of the point.

Suppose that a player will not be drafted unless he is ranked at the end of his college career as the equivalent of either a four or five star, based on his performance in college.

Suppose that at the low tier school, out of ten recruits, the number of two star recruits being ranked at the end of their career at a given star level has a pretty tight distribution, reflecting the reliability of the initial recruiting rankings:

Rank of ten typical two star recruits, revised according to performance, by the senior year:

1 star, 2 players
2 star, 6 players,
3 star, 2
4 star, 0
5 star, 0

The average senior rank = 2.0, the same as the predicted 2 star rank when entering school. However, the result is that no one achieves 4-5 stars by is senior year, so no one is drafted.

But at the high tier school, the distribution of senior performance ranks of two star players is more spread out, reflecting the initial ambiguity of the rankings:

Rank of two star recruits, revised according to performance, by the senior year (note, there are a lot more busts, but also more high performers):

1 star, 6 players
2 star, 1 player
3 star, 1
4 star, 1
5 star, 1

Here, again, the average senior performance rank = 2.0, the same as the predicted 2 star rank when entering school, also the same as that achieved at the low tier school. However the result differs: 2 players or 20% of the two stars are drafted.

Conclusion:

The average performance of senior two star players could be no higher at the top tier than at the low tier schools—as if there were no net effect of coaching in helping players get drafted---yet the high tier school has more draftees. The reason for the difference, in this example, requires only that the the two star recruits at the high tier schools had more ambiguous ratings than the two stars at the low tier schools.

NOLA Blue

February 21st, 2010 at 4:24 PM ^

I am very clear in expressing that coaching is only one of many critical elements in differentiating draft rates at high and low tier schools...

In fact, it was quite implicit that I considered the coaching staffs of Boise, TCU and Utah to be on par with coaches of the BCS. To quote from the 14th line down:

"When placed in the hands of a capable coaching staff (all of the above universities)"

And in the discussion:

"I generally relate the BCS crashers’ success to the high-quality of coaching staffs they have assembled and been able to keep. It would be interesting to see if there is a large difference in facilities at these schools when compared to Cal, Iowa, Oregon and Va Tech. If the gap is minimal, then the difference in drafting above could be considered to have more to do with media exposure and proven play against higher-performing competition."

So... I point to relative equality in their coaching staffs, and suggest a consideration of facilities, media exposure, and level of competition as reasons for the difference in recruitment rate. That would be 3 items I consider to impact players' draftability, outside of coaching.

Finally... differences in ambiguity? Your fantasy-distribution comes up with no 2-stars being recruited from non-BCS and 20% of 2-stars being recruited from BCS... I really have no idea what your point is, outside of it being speculation for speculation's sake. Re-read the results. I think you may have missed the entire point of the study: according to the data, there is no difference in eventual drafting of a 2-star and 3-star athlete if the two athletes are from the same team.

michelin

March 1st, 2010 at 2:59 PM ^

My reaction to coaching quality being a main factor in deciding the draft fate of recruits came from my recollection of your first diary on this topic, which did examine performance at different schools and conjectured about coaching being a big determinant of whether or not the athlete gets drafted. I had skimmed this very long article, and I do now see that you also acknowledge other factors, with your clarification.

Yet, you may want to reread my post to see that there is another factor that I point out, which may play a significant role in determining draft success (though perhaps you could argue it would cancel itself out if you have enough data to reliably compare only 2 and 3 star recruits at the same school).

You don't have to like merely illustrative numbers I gave, but my point remains. A two star in rivals that you know a lot about is not the same as a two star in rivals that has not been observed much. There may be a greater upside in the latter, and when it comes to producing people for the draft, only the upper crust make the grade. So, you should have more of the latter getting drafted. Also, the good schools are more likely to recruit such players, compared to the well know two stars. Hence, the good schools will get more of these players to get drafted--and this will have nothing to do with the other factors you note. I would not disagree with the factors you note as being possibly important. But we don't really know for sure.

NOLA Blue

February 21st, 2010 at 4:04 PM ^

I am sorry, but a sample of 521 athletes is a full 5.2% of the current NCAA Division 1-FBS scholarship football athlete population... all of the 2- and 3-star athletes from 5.9% of the 119 schools over a 4 year period. This would be the equivalent of a political poll for which 15,400,000 people were called and interviewed. Have you ever seen a poll for which more than 5% of the population was sampled? This is actually an absurdly large sample size.

Secondly... four years is a short sampling period? That is 40% of a decade my friend. It also happens to be about as long of a period as is possible if we are to assess schools with no change of the head coach between the time of a recruit's arrival and the time when they can be drafted... hence the span from 2002 to 2009.

I am not sure what you are basing your statements on... clarifying what you would believe to be an acceptable sample size and large enough time-period (with some sort of rationale) would be more beneficial than making two very bizarre statements.

Magnum P.I.

February 21st, 2010 at 4:46 PM ^

A minimum sample size of 40 is the rule of thumb with most inferential analyses in the social sciences.

You could actually do some cool stuff with "multilevel" stats using players and teams as separate units in the some model. You'd need at least 20 teams to make that work, though.

NOLA Blue

February 21st, 2010 at 5:11 PM ^

Thanks for the input.

If you believe that both the team and its members simultaneously influence and are influenced by the team's membership, then you are going to love the 3-star vs. 4-star analysis... really interesting phenomenon seems to occur with increasing concentrations of 5-star athletes on teams.

wolverine1987

February 21st, 2010 at 4:58 PM ^

no one needs to "resist the temptation" do they? Correct me if I'm wrong, or not understanding your point here, but all--not some, all, of the respectable data points, published here and referred to by Brian many times, show that in fact that recruiting ratings are in fact, correlated nicely to team success and likelihood of being drafted. They cause and predict nothing, but in fact, the higher you are rated the higher your chance of being an excellent player, and your team an excellent team.

P.S. I do want to commend you for all the work, at the same time that I can't believe that an analysis of seven teams can in any way be definitive or statistically significant. Correct me if I'm wrong there as well.

P.P.S. I do agree with your assertion that we should stop ragging on our 2 stars, just not for the reasons you state.

NOLA Blue

February 21st, 2010 at 5:41 PM ^

Yes...

You are not wrong, the recruiting rankings do correlate with being drafted. But if you are happy purely with correlations, then you might be interested to know that the population of storks in Europe has declined in correlation with the decline of Europe's fertility rate... so storks really do bring babies? In science, and life, correlation is pretty much worthless if you don't have a defined question you are trying to answer; or if you transpose a correlation to answer a question to which it is not related.

The validation of the recruiting rankings, on this blog and elsewhere, has been to say "a recruit heading to Florida has a greater chance of being drafted than a recruit heading to Northern Illinois." I don't need a recruiting service to tell me this; and if that is the purpose behind following recruiting rankings, then paint me perplexed as to how these sites actually build a following.

The real reason we follow recruiting rankings is because we hope that they will tell us how good a player is (or may become.) That way, when we sign a 4-star, we can all sleep better "knowing" that he has a better chance of becoming NFL-caliber than a 2-star. But, the question we have to test the recruiting rankings against is "does the 4-star player going to Iowa have a better chance of becoming an NFL player than the 2-star player going to Iowa." To check the validity of recruiting rankings in this case requires comparing players at Iowa to players at Iowa, cross referencing their recruiting stars with NFL draft success. This tells us whether or not 4-star players are ACTUALLY better than 2-star players. Today's analysis was looking at 2- and 3-star players, and found no difference. So, there is no more reason to be anxious about a 2-star recruit coming to Michigan than a 3-star recruit; because when placed in the same system, they both have the same odds of becoming an NFL caliber player.

Yes...

7 teams is more than sufficient to represent 119 teams... especially given that not all 119 teams heavily recruit 2-star athletes. But even against the total number of FBS teams, that is 5.9% of the total (and 5.2% of all scholarship football players.) Below I pointed out that this is the equivalent of a political poll interviewing 15.4 million citizens. 5.9% is a massive sample size.

And yes...

We should stop the worry, angst, bad-mouthing of 2-stars. Thanks for the agreement! :^)

Blue_Bull_Run

February 21st, 2010 at 7:23 PM ^

I'm kind of confused though why we're using only 7 teams. And within those 7 teams, the sample size at each school tends to be quite small, too - like Utah's 1/6 split. I have to imagine that could have very easily been a 2/5 or whatever, which would have quickly changed the percentage.

Also, if you look at the "BCS crashers total," you'll notice there are only 50 samples for the 3*, compares to 194 samples for the 2*. If one or two of those 3*s had somehow managed to squeeze into the draft, then the percentages favor the 3*s again.

I understand that data collection is a bitch, but shouldn't we sample some of the upper echelon programs, too? Out of the three "better" programs, I'd say the results for Iowa and Oregon are surprising, but VaTech's data clearly favors 3*s.

bjk

February 21st, 2010 at 9:58 PM ^

NOLA was interested in whether the number of stars or the team a player came from was more important in predicting draft percentage, and so he needed teams with a statistically usable high number of both two- and three-star players. That is what constrained his selection. That and coaching stability.

Magnum P.I.

February 22nd, 2010 at 12:00 AM ^

He easily has enough power with his sample to achieve statistical significance. What your point of only having seven teams gets at is the "external validity" of the results, or rather how far you can extend the findings. Is the sample representative of all div IA teams? Or is there something unique about the seven he used that makes it a bad idea to generalize the findings to all teams?

El Jeffe

February 21st, 2010 at 7:59 PM ^

Three points:

1. Mike Hart was a three star, so...

(kidding)

2. You (NOLA Blue) are wrong that the percentage of a population doth a good sample size make. It's all about the absolute size, baby. That's why good polling outfits sample 500-2000 people both from states and from the U.S. as a whole.

3. Nevertheless, 521 is pretty damn good.

4. Okay, four points. It seems to me like the strength of your argument is that highly-ranked players go to the schools with the best facilities/coaching/hostesses, so it is impossible to tell whether that same player would have done as well at a more poorly ranked school (Donald Rubin's fundamental problem of causal inference).

In that respect, it seems like you'd want to stratify the BCS schools into quality quartiles or something, sample a few from each, and run the analysis. I didn't get the logic behind the choice of schools.

Alternatively, you could, say, take all the three- or four-stars, and regress their probability of being drafted (or a fancier Tobit model where the DV is draft position, censored if the player wasn't drafted) on their rankings, plus a variable capturing the quality of the school, and see which one wins out.

5. There is no fifth point. Except that I love this shit. Keep up the good work.

NOLA Blue

February 22nd, 2010 at 1:08 AM ^

True on percentages not dictating a good sample size... just trying to be as visual as possible in my responses.

As for the quartiles... when I have the time to write up the "3- vs 4-star" evaluation, you will see that the data actually is divided into four quartiles. Two of the quartiles are able to be used to evaluate 2 vs 3; Three of the quartiles are able to be used to evaluate 3 vs 4; and as you can see, one of the quartiles is able to be used in both analyses.

Why not use all four quartiles in both analyses? Well, because the question is based on a player's comparative potential within the same unit (in this case 3-4 schools from a comparable strata.) This means that a unit without sufficient data regarding one type of player (say, 2-star recruits) can't be used to analyze the success of said-type within that unit.

Really, it's almost as though the data is split into quintiles... with one quintile being the tier below Boise, TCU and Utah. But that quintile was not examined due to its lack of NFL production, and therefore inability to be measured using draft-status.

colin

February 21st, 2010 at 8:38 PM ^

if we're looking to figure out how good a recruit is, our best bet is to infer via offers and commitment? In that case, we could generate our own star rankings if reported offers are accurate enough. I think someone may have already done prestige star rankings here, actually. But if the recruiting services are giving us this information, they should have incorporated those facts into their rankings already, no? Even if they don't bring any scouting prowess of their own to the table, they should be able to deduce which recruits the coaches--whose capacity to scout are to some degree known--like best and rank accordingly. Perhaps it could be done better mathematically, but I assume they're close. And if not...that speaks pretty badly of them.

Taps

February 21st, 2010 at 10:20 PM ^

Before I start, I appreciate that this must have been a labor of love and taken a great deal of time to compile.

I don't fault your methods, and the findings speak for itself. What I don't understand is how at some point between demonstrating that there's very minimal difference between 2 and 3 star athletes as far as NFL draftability and the end of your dissertation, you come to equate being drafted with being a contributor.

You conclude:

"please stop dismissing our 2-stars. It is obvious that they are just as likely to be contributors as our 3-stars."

That's a nice sentiment, but it's not what you showed.

It's a long leap over an enormous gray area to get from "be drafted" to "be a contributor."

So much goes into being an NFL draft choice that doesn't relate to contributions on the field. Careers and millions of dollars are made based on tenths of a second in the 40 yard dash and single inches of height.

Just off the top of my head and with a quick scan of rivals I found that, in your time frame, 2 star recruits Jared Zabransky (QB) and Ian Johnson (RB) went undrafted, and thus wouldn't have registered for your study. Nobody familiar with BSU would ever claim that those two weren't contributors for Boise, maybe two of the biggest.

I love your data and your effort, but what seems really important is how the guys in each group who aren't absolutely elite pan out against each other. At what rate do the remaining ~90% of 2 stars and ~90% of 3 stars become undrafted contributors? Of course, what defines "contributor" is going to be highly subjective and require a whole 'nother level of data mining.

But, frankly, as Michigan fans, I think what we all care about is the rate at which the coaches are turning our lower star players into celebrated (or at least solid) contributors (ala Zabransky/Johnson), not how many of them are defying the odds to make The League (however nice it is for them :P).

NFL draft status is an easy and practical measure for collegiate success but not a particularly thorough one.

NOLA Blue

February 22nd, 2010 at 12:50 AM ^

There are definitely contributors that do not move on to the NFL. I used the term loosely, and really should have defined it with a qualifier, such as "NFL-quality contributors."

And wow, defining what happens to the other 90% would be interesting; albeit far too much work for me to fit in...

Interesting thoughts.

Realus

February 22nd, 2010 at 8:47 AM ^

NOLA, thanks for all of the work. It is a very interesting analysis.

One idea, which already be covered, is when a 2-star player goes to a BCS program, is he inherently a better player than a 2-star that goes to a non-BCS program BECAUSE the coaches at the BCS programs are better evaluators of talent than the recruiting services.

NOLA Blue

February 23rd, 2010 at 11:13 AM ^

This is an excellent question, that would really be worth examining as a potential source of the differences in success at different levels of Div 1 football.

My study was not actually designed to address the differences between 2-stars at two different schools. I was really interested in knowing whether a 3-star who comes to Michigan is any more likely to excel -at Michigan- than a 2-star placed in the same environment.

I just posted more data that will better evaluate the question of Michigan's athletes (as the schools in this analysis were not from the same stratus...)

http://mgoblog.com/diaries/new-value-5-star-players-behave-social-catal…-

Ali G Bomaye

February 22nd, 2010 at 9:14 PM ^

This post is amazing. I love posts that question established wisdom through statistics.

I have one concern, though. By using Utah, Boise, and TCU as examples, you're committing to an analysis of players that, by definition, performed surprisingly well. To make the jump from a collection of (mostly) 2-stars to a top-10 program, a team needs to have quite a few players that exceeded all rational recruiting expectations and performed like blue chips.

These "surprise" players are, of course, the ones who get drafted. So already, you're selecting a group of surprisingly good 2-stars and 3-stars to evaluate, because in looking at Utah, Boise, and TCU, you're probably looking at the three best collections of 2-star and 3-star recruits in the country. It seems possible (or even probable) that when you take such an abnormal population of lightly-recruited players, the "normal" recruiting correlations might not apply.

Of course, the rebuttal to this is that these guys are surprising players, but more 3-stars than 2-stars should be surprising if ratings mean anything. I guess the response to that would be to look at the raw numbers of 3+ star athletes at these schools. For instance, Boise only had 10 3+ star players over this time period. To do as well as they have, they would need more than 10 good players... so by selecting them, you've already selected a team with abnormally good 2-star recruits.

I don't think this flaw is fatal to your analysis. But I'm really eager to see if this finding holds up when you evaluate some more average teams.

NOLA Blue

February 23rd, 2010 at 11:05 AM ^

...more average teams do not produce a significant number of NFL draftees. So, yes, the measure of NFL draft status is a selection bias of its own, as I am required to consider teams with high numbers of draftees.

I just posted the results from 7 other teams; and when taken in conjunction with the two subsets on this diary post, I have examined 4 small groups that represent different sub-sets of the top 60 or so programs in Div-1 FBS. We will not be able to relate the results to Akron, etc.

Three of the sub-sets have enough 2-star athletes; it is apparent that their draft-rate increases to just above 12% when going into a successful BCS program instead of the 9% seen at BCS crashers... the BCS crashers do not actually have an extraordinary group of 2-stars...

Thank you for the comments, hopefully you enjoy the new data just as much!

Ali G Bomaye

February 23rd, 2010 at 11:20 AM ^

I just took at the new data. My first thought is that you seem to have even more free time than I do - congrats! My second thought is that it seems to answer my concern - with the exception of Virginia Tech, for some reason, 2-star and 3-star recruits seem to be drafted at about the same rates for most other schools. I think the strongest evidence of this is the Cal-Iowa-Oregon grouping - they have approximately equal numbers of each, but also pull in a few 4-5 star players.

Once again, great work - kudos.

mejunglechop

February 23rd, 2010 at 8:10 AM ^

This is a great analysis, thanks! My one worry that the coaching stability requirement may distort the results. All the teams studied except for Cal had significantly better records than their recruits' rankings would suggest likely.

NOLA Blue

February 23rd, 2010 at 10:57 AM ^

Restricting to high coaching stability will inherently select better coaches. However, since the data is comparing 2-stars to 3-stars on the same teams... both are benefited by the elevated coaching. I just posted the data from 7 more BCS teams; the new post might clarify this a bit better.

mgobaum

February 23rd, 2010 at 11:35 AM ^

To me, being drafted is too large of a bucket for this comparison. There is a pretty big difference between the first and last round. I suggest weighting their draft status by round (or multiple round buckets). If three stars are being drafted two rounds ahead of two stars, then they aren't really equal. Your sample size is probably not large enough at this point for this kind of resolution.

You also take away a lot of resolution by averaging the percentages drafted. At the very least, give us the standard deviation. I'm forgetting the specific test you would want to use, but you can show whether or not the two percentages are statistically different. You might have done this and I just missed it while skimming the article.