I did a simple analysis in order to answer the question of how Scout and Rivals recruiting rankings compare to players’ actual levels of success in college. I only included players who made the first or second all-big ten teams from 2005 – 2008 so I can’t make any conclusion whatsoever about which site is more likely to overrate a player but I can conclude which site is less likely to miss on identifying top tier talent. This can be used as a decent proxy for which site does a better job overall.
A few assumptions:
If a player made the first team all-big ten team their “true rating” was 5 stars. If they made the second team their “true rating” was 4 stars. The one exception is QB – both first team and second team QBs were given a 5 star “true rating”. I went back and forth on this – the second best QB in the big ten probably is a 5 star player but I don’t thing you can really say the same for the 4th best guard.
Both the coaches’ team and the media teams were included.
I couldn’t find recruiting rankings for players from the class of 2001 so this only includes the classes of 2002 – 2008.
There were 159 players in the sample. Rivals and Scout gave the same rating to 60.4% (96 players) and differed on the remaining 39.6% (63 players). Of the 63 differences, Rivals was closer to the mark on 73% (46 players) and Scout was closer on 27% (17 players).
Position specific nuggets:
- Rivals was much better at predicting successful defensive lineman – of the 12 DLs where Rivals and Scout differed, Rivals was better 11 times.
- Rivals also was stronger with DBs (7/10), LBs (4/5), and OLs (11/14)
- There was no position where Scout performed better in total
General recruiting conclusions:
By looking at the average deviation from the “true rating” by position, we see that both recruiting services struggle with the same positions: TE (avg. dev. 2.1 stars), QB (1.9), DB (1.9) whereas other positions are easier to predict: RB (1.4), LB (1.4), WR (1.5), and OL (1.5).
- Rivals does a better job than Scout at predicting all-conference talent and therefore may be a slightly more reliable judge of talent overall.
- It’s easier to predict collegiate success at some positions (RB, LB, WR, OL) than others (TE, QB, DB).
Thoughts/suggestions? I would probably be interested in beefing this up but this what the first thing I came up with off the top of my head.