(Feel free to skip down to the "discussion" for 7 points that can be taken from this data... but, please make sure to first read the research-question the study was designed to answer.  Also, for those of you interested, the study-design is very clearly detailed; so please, only comment on design issues such as sample size if you have read the rationale... of course, if you disagree with the rationale or believe something to be missing, fire-away.)

Sometimes the benefit of writing in two parts is the inherent advantage of finding out what readers’ thoughts are.  After reviewing the comments on yesterday’s post (re: 2- vs. 3-star) I am left with the clear impression that I did not do a good job of explaining the research-question I was testing, nor the size of the sample and what populations it might represent.  I think the addition of today’s data will clear some of that up, as a few questions pertained to “why did you stop there… why not upper echelon teams?”  Today’s data is from the upper echelon.  But, I will take a moment to clear up the central issues of vagueness from yesterday.

First, the question this research answers… 

The most important part of using research to answer questions is assuring that the research was designed to answer the question being asked.  I was motivated to perform this analysis by the abundant number of times, on MGoBlog and elsewhere, that correlation of stars to draft status, or stars to team performance, has been confused with stars answering to the abilities and potential of individual players.  Among yesterday’s responses, I replied to wolverine1987 regarding this issue.

One question we might seek to answer is this:  “Does a recruit heading to Florida have a greater chance of being drafted than a recruit heading to Northern Illinois?”  There are two ways to answer this question: one is using your knowledge of team performance, which makes it quite obvious that the Gators are more likely to send a player to the NFL; the second is to look at the average number of players drafted from each school over the last 4 years, and assign a probability to the recruit’s chances.  Neither of these require recruiting rankings to be solved; but, yes, recruiting rankings do correlate to both.  However, I do not believe that the majority of football fans are curious as to whether recruits going to one school have a higher chance of being drafted than recruits going to another school...

My belief is that we would each like to know “if Michigan signs that 4-star DT out of Texas, will he be more likely to become an NFL-level contributor to our team than that 3-star DT out of Illinois?”  This question requires the star system to assess which players are actually better; it requires that their assessment of Joe X. as the #8 DT, vs. John Y. who they place as the #36 DT, to actually be a reliable measure of difference between the two, stating players’ talent and potential with measurable validity.  As you may have already figured out, correlating the probability of being drafted to all draftees’ recruiting stars, or correlating stars to teams’ performances does not isolate whether the stars actually correlated to individual players’ talents/potential. 

The probability of being drafted is impacted by talent/potential, coaching, team members, facilities, successfully proving oneself against high-level competition, and potentially even media exposure.  Therefore, to actually test whether stars correlate with draft success for individual players (which would answer the question of whether or not they accurately predict individual players’ abilities,) we have to isolate the probabilities of success within systems in which the levels of coaching, facilities, competition, and media exposure are as similar as possible (identical would be the impossible ideal.)  Otherwise, we are just answering the obvious:  Florida, who has a better coaching staff, better facilities, better competition, and more media exposure sends more players to the NFL on average than N. Illinois.

So, this brings us to sampling…

Obviously, a perfect sample would be finding one school that correlates perfectly to a well-defined group of schools.  The requirements for this perfect situation: 

1. A large enough number of 2-, 3-, 4- and 5-star recruits at this single school to assure that a random distribution would be allowed to approach normal in regards to the success of recruits at this school developing into NFL-caliber players.
2. A large enough number of NFL draftees to assure the same process of normalization is allowed (should there actually be normalization.)  If draft-status occurs at the same rate among 2-, 3-, 4- and 5-stars from this school, then we could say that a random assignment of stars to the same athletes would have produced the same result… this would mean the stars were not predictors of success.
3. A group of schools to compare to, which themselves exactly share the “ideal school’s” traits of coaching, facilities, level of competition, and media exposure. 

Next, for the perfect assessment across all strata of collegiate football, we would have to hope that there are some perfectly segmented groups (4, 5…6?) These groups would be different from each other, but the schools in each group would be perfectly matched to each other.  Then, we could take a random member from each group and assess the behavior of its recruits’ success rates, projecting the results to all players within that segment of schools.

This is not the world we live in.  Hence, the inevitable arguments over sample selection and sample size.  No matter how well designed a study is, there will always be uncertainty that it achieved a fully representative sample.  If you are looking for answers of 100% accuracy, then do not look to science… it will always be chasing perfection, and never reach it.  But, looking at the world from a level of random chance vs. predicted order is useful for evaluating a system which is proclaimed to having found order among chaos (i.e. the star-system makes just such a claim, and must therefore be assessed versus randomness to see if recruiters are actually able to differentiate among the top 0.23% of high-school athletes.)

My sample selection…

For this study, which I broke into two analyses, I wanted to isolate players’ abilities from the impacts of coaching, facilities, etc.  In other words, I wanted to see if players’ development within a homogenous environment responded differently based on stars (as administered by Rivals.)  Hopefully we all agree that a study looking at all 119 teams would not isolate players in a homogenous environment.  Therefore, I am interested in assessing players within semi-homogenous sub-groups that, while not representative of 119 schools (or even 40 for that matter,) will offer a look at player development from within multiple strata of Div 1-FBS football.  Players will be compared to players within the same stratus.  

Because no single recruiting class from any single school produces enough NFL-draftees to allow for the possibility of normalization (should it actually occur,) I was forced to track multiple recruiting classes.  Considering that teams are also dynamic over the course of a decade, and that one of the major influences on a player is the players around him, I could not stretch this tracking too far (players selected in 2002 affect players in 2003 while players in 1996 do not affect them… of course, this is a good thing, because the star-system was not around in 1996.)  So I only allowed it to stretch to 4-years, basically representing a full team cycle (yes, I am aware that 5^th year seniors blur this a bit… so 95% of a team-cycle.)  I also needed to be able to see the draft results for all four years of recruits… thus 2002-5 was the latest 4-year cycle I could consider such that I could cross-reference the 2005-9 drafts; and that also lines up with the first available Rivals data-base.  So, four recruiting classes of 2002-5 compared to the five drafts of 2005-9.  Again, 5^th year seniors have not yet had the chance to be drafted; hopefully I will have time to revisit this data after the 2010 draft results are in.  I do not expect there to be much impact, but there is always the possibility.  (**Edit** I did not include kickers/punters in the counts of either recruits or draftees.)

Most teams did not have enough draftees over this four year period to allow strong normalization (USC the notable exception, with 30 draftees…)  So, it was obvious I would have to analyze small groups of schools…

Next, the first step to accomplishing isolation:  coaching.  As “bjk” so aptly pointed out yesterday, my study was constrained by coaching.  In fact, this was the strongest constraining factor on selection.    One of my concerns regarding the use of 4 recruiting classes, was the issue of environmental stability.  The factor which can most quickly impact the environment of player development is coaching… in other words, facilities take years to build and can’t be wholly replaced simultaneously while a coach takes a day to fire and a month to replace.  So, in order to assure stable environments to compare players within, I added a limitation to selection:  there could be no head-coaching change between 2002 and 2009.  Thus, all players assessed at the school received strongly similar tutelage (not accounting for position-coach changes.)  The one exception I made to this rule was Utah; I would not have felt comfortable that Boise and TCU were a large enough sample, with only 15 draftees… adding Utah’s 8 draftees makes me much more comfortable.  Off the top of my head, this gave me:  Boise, Cal, Florida St, Georgia, Iowa, tUOS (aka Bastards from Columbus,) Oklahoma, Oregon, Penn St, TCU, Texas, (Utah,) USC, and Va Tech.  Could I have spent hours researching every team in the nation?  Yes… but I was happy with a list of 14 schools.

The second step to accomplishing isolation of talent/potential:  facilities.  The 119 teams from Div 1-FBS may be broken into groups according to a perceived difference in facilities.  For example: U of M, Texas, USC, LSU, the University of Ohio State, Oklahoma, Florida State and Georgia all have Mount Olympus-level facilities.  Are they all exactly identical?  Of course not.  But they are similar enough that I feel confident that if I were to lengthen this list to the top 20-24 programs in terms of facilities, and then select 3+ of them to analyze, I would be able to say that the results from those analyses were representative of the total sub-group.  The remaining 99 teams could probably be divided into three more sub-groups… those with exceptional, strong or simply functional facilities.  Of course, the teams at the gray areas are arbitrary (i.e. is there really a difference between #46 and #52 even though they may fall into different sub-groups?)  For my purposes, I split the teams with high coaching-stability into groups that I felt were very similar to each other in terms of facilities…

Mount Olympus:  Florida St, Georgia, Oklahoma, (Oregon,) tUOS, Penn St, Texas, and USC

Exceptional:  Cal, Iowa, Va Tech

Strong:  Boise, TCU, Utah

Functional:  None evaluated.  Doesn’t actually matter, because these schools don’t produce draftees on a regular enough basis to power a study without randomly analyzing 20-30 teams; at which point it would become impossible to suggest all the athletes were in an environment approaching identity.

Remember the mentioning of players influencing players?  Well, that bears out that the supposed strength of recruiting must be considered; and I gave heaviest consideration to the two extremes (5- and 2-stars.)  As we all know, 5-stars are actually a dependable source of high-performers, and before the data came in I had to assume that 2-stars would underperform in comparison to 3- and 4-stars.   With that in mind, we end up with 3 tiers of BCS schools to pair with the BCS crashers:

From the “We collect 5-stars like they’re Gold Teeth” category:  (sorry, all I can imagine is Pete Carrol smiling with a mouth full of bling…)

Florida St, Oklahoma, Texas and USC:  This group signed 40 five-star athletes over four years.  That’s an average of 2.5 top-flight athletes per school… per year!  They also had too few of 2-stars to count.

From the “Hey, we collect 5-stars too” category:

Georgia, Ohio St, and Penn St:  This group could have been highly consistent in terms of recruits, except Joe Pa gives recruiting analysts the big middle finger…  Each school recruited exactly 4 five-stars apiece… for an average of 1 top-flight athlete per year.  Then Joe Pa flipped his $h%t and decided to skip over the 4-stars in favor of an army of 3-stars; and swiftly failed miserably at turning them into NFLers.  Joe Pa was certainly an outlier among my sample of 14 schools… probably correlating to either his status of the inventor of college football or his deteriorating sanity.  God bless him.  But, 2-stars were still far and few between in this group.  (Georgia had 5, tUOS had 6, and Joe Pa took 13… Joe Pa is effectively a recruiting ‘tweener.  His thirteen 2-stars borders on having enough to evaluate; but considering the next group of schools averaged over 30 two-stars each, his collection of one 5-star each year made his pool closer to those of Georgia and tUOS.)

From the “We’d take a 5-star, but normally we have to give their spot to 2-stars” category:

Cal, Iowa, (Oregon) and Va Tech:  This group signed only 9 five-stars total… for an average of just over ½ of a five-star per year for each school.  But, what they lacked in prodigies, they made up for in presumed projects… 154 x 3-stars and 123 x 2-stars.

And finally, the “We’ll see your clutch of 5-stars, raise you none, and then promptly kick you’re a$$ with our army of 2-stars” category.

Our beloved BCS crashers… the destroyers of logic.  Couldn’t they just play along with the rules of our dominance?  These guys, as you have seen, signed less 4-stars than Cal, Iowa, Oregon and Va Tech signed 5-stars.  Nearly their entire squads were made up of 2- and 3-star prospects; with two-stars making up more than 75% of the scholarships.

Why is Oregon in parentheses?  Because they are the only one for whom a decision must be made… by facilities they are elite.  By recruiting composition they fit squarely with Iowa, Cal and Va Tech.  In terms of facilities, I cannot give a quantified answer as to the difference between their facilities and Cal’s or Iowa’s or Va Tech’s.  When faced with a decision for which I have numbers (recruiting composition) vs. the inherent vagueness of determining placement on a facilities list, I have to go with the numbers.  Therefore, Oregon was placed with Cal, Iowa and Va Tech.  Also, you can see that the teams with facilities presumably worthy of Mount Olympus are split into two categories based on 5-star recruiting efficacy.

So, the list of groupings:

#1.  Florida St, Oklahoma, Texas and USC

#2.  Georgia, tUOS and Penn St

#3.  Cal, Iowa, Oregon and Va Tech

#4.  Boise, TCU and Utah

At this point the first question is:  are these groupings homogenous within themselves?  Could I move Cal to any other group and feel like it is a better fit with a balanced consideration of facilities, recruiting, media exposure, etc.?  That process is repeated for each school.  The only school that stands out to me is Florida St; but that is because they have underperformed so badly on the field as of late.  As far as recruiting through 2002-4 they are definitely on par with the top group, facilities match, and so does media exposure... competition?  Well, they don’t really fit any better with Georgia or the Big Tens in that regard.

The second question is:  whom do these groups represent?  Each of these groups are probably representative of only 5-20 teams each.

Group #1 – These schools absorb such a disproportionate amount of 5-star talent that they can only really represent themselves… maybe Florida and Alabama are represented by this list?

Group #2 – These schools probably represent what we would expect to find at the top-tier of the Big Ten, Big 12 and SEC (not including the mega-recruiters from the Big 12 and SEC.)  Perhaps with a renewed parity in the Pac-10, this group would represent the top-tier of the Pac-10 as well (perhaps USC going forward will be better represented by this group… all depends on whether or not they keep up the 5-star recruiting.)  As it stood for the last decade, the top-tier of the PAC-10 pretty much consisted of USC, with an immediate drop-off to a middle tier.

Group #3 – These schools probably represent the top-tiers of the ACC and Big East, and mid-tiers of the Big 10, Big 12,Pac 10 and SEC.

Group #4 – Real tough to say.  On the field these three have proven themselves to be able to take on the top tier of the SEC and Big 12.  Likewise, the #2s and 3s from their conferences have proven conquerors of the mid-tier of the Pac-10 in recent years.  But in terms of facilities and recruiting composition, I think they are more representative of the mid-tier of the ACC and Big East, and lower tiers of the Big 10, Big 12, Pac-10 and SEC.  Perhaps we should just take the data of their players to be an isolated analysis.

For me, I only really care about the University of Michigan… and in that regard, I would consider it most homogenous with the Georgia, tUOS and Penn St grouping.  So, I would expect our 2-stars, 3-stars and 4-stars to behave in a similar fashion. 

Unfortunately, as stated before, it is not feasible to analyze the bottom tier of NCAA football due to a low rate of NFL draftees.

Drum-roll for the Data…  (Boise, TCU and Utah not included today, but I’ll repost their aggregated results under the results section.  Their individual data can be found in yesterday’s post re: 2- vs. 3-stars.)  Data in alphabetical order.  Percentages given with decimals for teams which will be discussed in isolation.

Aggregated Results:

BCS Crashers

Cal, Iowa, Oregon and Va Tech

Georgia, tUOS and Penn St

Florida St, Oklahoma, Texas and USC

Discussion

What I take from the data and results…

1.       Five stars can go to any BCS school consistently in the top 50 without fear of missing out on the NFL due to choosing the wrong university.  Although only one of the sub-groups examined had enough 5-stars to be able to draw a reliable probability, the conglomerate 45% draft rate seen at Florida St., Oklahoma, Texas and USC is at least theoretically corroborated by the similar rates seen within the two other groupings of BCS schools (44.4% and 50.0%.)  Therefore, if the recruiting gurus at Rivals believe an athlete to be worthy of five-stars, then that athlete has approximately a 45% chance of eventually being drafted as long as he goes to schools on at least the same level of Cal, Iowa, Oregon, and Va Tech (the “lowest” level I could measure.)  This data cannot be used to assign a probability to teams below this level due to a lack of historical data re: 5-stars at these schools.

2.       On teams with no, or few, 5-stars athletes… there is no difference between the draft rates of 2-, 3- or 4-star athletes in terms of likelihood of developing into NFL-caliber players.  We see this for 2- and 3-stars among BCS crashers (their 4-stars can’t be assessed due to insufficient sample size.)  We also see this for 2-, 3- and 4-stars at Cal, Iowa, Oregon and Va Tech.  Such athletes have a 12.2%, 13.0% and 13.4% chance of being drafted, respectively.

3.       2-star players at BCS schools have just over a 12% chance of eventually being drafted... Important for consideration of walk-ons.  The schools on the level with Cal and Iowa turned in the only sample size of 2-stars large enough to really let normalization play out.  At these schools, the 2-stars were drafted at a rate of 12.20%.  It is interesting to note that at the big-timers (Georgia, tUOS and Penn St) the rate was the same for their smaller sample (12.5% of 24 players.)  The potential of 2-stars at this tier of schools may be even more impressive considering that the 2-star sample in this group was dominated by Joe Pa’s group of 13.  Look at Joe Pa’s track record with 2- and 3-star athletes.  He is getting less out of this group than the coaches at all the other schools measured (exceptions: 3-stars at Oklahoma and 2-stars at Va Tech.)  His rates (7.69% and 7.69%) are even lower than the conglomerate rates of the drafting of 2-stars and 3-stars from the BCS-busters, who have a much lower draft-rate than BCS schools.  This, in addition to his having coached at one school for more than 60 years, is what makes Joe Pa an outlier.  Georgia and Ohio State’s tiny sample sizes of 2-stars means we can only assign hope that 2-stars at a school like Michigan has a 16-20% chance of being drafted.  This brings a renewed sense of importance to the emphasis on a walk-on program… if we could bring in ten 2-star walk-ons each year, there should be 1-2 NFL athletes among them given a span of 12-20% probability.

4.       Five stars behave as a catalyst to their teammates’ development.  As the concentration of 5-stars among a team’s composition is increased, the team’s draft rates are improved for 3- and 4-stars.  The data groupings express a consistent progression of average five-star signings each year: 0, 0.5, 1 or 2.5 per team in the grouping.  Recruiting between 0 and 0.5 five-star athletes each year seems to have no effect on the lower-rated players (2- and 3-stars behaved the same in the environment represented by Boise, TCU and Utah; and 2-, 3- and 4-stars behaved the same in the environment represented by Cal, Iowa, Oregon and Va Tech.)  The lack of effect seen after  recruiting 0.5 of a five-star each year suggests there is a threshold that must be reached before a catalytic effect.  This threshold lies somewhere between 0.5 and 1, given the large bump 4-stars see in their draft rate on the campuses which acquire at least one 5-star per year (bump from 13% to 24%.)  Going from 1 to 2.5 five-stars per year equates to a smaller bump for the 4-stars:  from 24% up to 29%... probably something to do with the “catalysts” beginning to take up too many spots on the starting roster, which diminishes playing time for everyone else even as their play is further improved. 

4-stars are most prone to elevating their play.  This either means there is actually a difference between 4- and 3-stars’ abilities; or it is indicative of coaches following a biased hierarchy in which it is much more difficult for a 3-star to get a whiff of the field.  I don’t know, I have never been on a football team, but my experiences as a collegiate swimmer suggest that there are some interesting psychologies at work as far as how coaches stratify their athletes.

5.       Georgia might be evidence of one style of coaching…  although Georgia is not a sufficiently large population in and of itself to be statistically relevant, it is interesting to note that there is no apparent correlation between stars and NFL-drafting on Mark Richt’s team.  5-stars were drafted at a 25% rate, with 2-stars (!) coming in at the next highest rate of 20%.  (4-stars 15.2% and 3-stars 17.1%.)  So, after 5-stars (who had an unusually low success rate) the correlation between stars and drafting was the exact opposite of what we would expect.  This might be indicative of a coach who gives all comers equal opportunity at promotion and demotion… sound familiar?

6.       As NFL draft-rates for 5-stars increase, the draft-rates for 2-stars plummets.  An interesting case study is looking at Georgia, tUOS and Penn St side by side.  It cannot be taken with more than a grain of salt, because these teams have neither enough 5-stars nor 2-stars to be statistically relevant.  But, for the love of patterns I’m going to charge ahead anyway and present that data here:

Perhaps this is further evidence of a hierarchy, in which programs with multiple starting positions filled by 5-stars are inherently going to have fewer opportunities for a significant proportion of their 2-stars to try-out in a real-game situation.

7.       USC Case Study.  As I have already mentioned, USC had enough recruits AND draftees to be able to compare recruits within the ranks of 3-, 4- and 5-stars.  This is highly valuable, because it is the only piece of data which is both statistically relevant AND from a completely homogenous setting.  This is where we find out if there is a true difference in the odds of recruits going pro based on their star ratings…

Ummm… so yeah.  4-stars and 5-stars are equivalent.  Dag-nab…  this analysis was supposed to settle the debate.  I can only explain this by the fact that apparently there is a second critical threshold of obtaining 5-star recruits (USC grabbed 3.2 per year between 2002-5) beyond which their presence produces a sense of normalcy to which everyone else on the team believes they are part of.  How scary is it facing off against 1 Brandon Graham in practice?  Pretty damn scary… But how scary is it if there were 10 Brandon Grahams?  The only way to deal with the absolute terror would be for players to reset to a new sense of normal and elevate their expectations to what they perceive as a new normal.

In conclusion….

Hooray for stepping up emphasis on the walk-on program!  The potential of developing a couple of NFL-caliber athletes each year would be a huge gain from something which is basically scholarship neutral.

And, from now on I will be highly interested in our pursuit of 5-stars.  It is quite evident that their value extends well beyond individual preparation for the NFL; their presence adds to the quality of practice and motivation for everyone around them, and can apparently give people a new sense of normalcy where they go all Neo on the recruiting services and leave the reality of the Matrix.