Rivals vs. Scout
I did a simple analysis in order to answer the question of how Scout and Rivals recruiting rankings compare to players’ actual levels of success in college. I only included players who made the first or second all-big ten teams from 2005 – 2008 so I can’t make any conclusion whatsoever about which site is more likely to overrate a player but I can conclude which site is less likely to miss on identifying top tier talent. This can be used as a decent proxy for which site does a better job overall.
A few assumptions:
If a player made the first team all-big ten team their “true rating” was 5 stars. If they made the second team their “true rating” was 4 stars. The one exception is QB – both first team and second team QBs were given a 5 star “true rating”. I went back and forth on this – the second best QB in the big ten probably is a 5 star player but I don’t thing you can really say the same for the 4th best guard.
Both the coaches’ team and the media teams were included.
I couldn’t find recruiting rankings for players from the class of 2001 so this only includes the classes of 2002 – 2008.
Results:
There were 159 players in the sample. Rivals and Scout gave the same rating to 60.4% (96 players) and differed on the remaining 39.6% (63 players). Of the 63 differences, Rivals was closer to the mark on 73% (46 players) and Scout was closer on 27% (17 players).
Position specific nuggets:
- Rivals was much better at predicting successful defensive lineman – of the 12 DLs where Rivals and Scout differed, Rivals was better 11 times.
- Rivals also was stronger with DBs (7/10), LBs (4/5), and OLs (11/14)
- There was no position where Scout performed better in total
General recruiting conclusions:
By looking at the average deviation from the “true rating” by position, we see that both recruiting services struggle with the same positions: TE (avg. dev. 2.1 stars), QB (1.9), DB (1.9) whereas other positions are easier to predict: RB (1.4), LB (1.4), WR (1.5), and OL (1.5).
Overall conclusions:
- Rivals does a better job than Scout at predicting all-conference talent and therefore may be a slightly more reliable judge of talent overall.
- It’s easier to predict collegiate success at some positions (RB, LB, WR, OL) than others (TE, QB, DB).
Thoughts/suggestions? I would probably be interested in beefing this up but this what the first thing I came up with off the top of my head.
November 4th, 2009 at 4:45 PM ^
u shuld by rivals cuz they're sight is more prettyier lol
Seriously though, I know you mentioned the "true ratin" issue but that makes it hard to draw meaningful conclusions from this. Still, it was a good read.
November 4th, 2009 at 4:50 PM ^
Doesn't any evaluation of which site does a better job have to assign some sort of "correct" values to players in order to come up with accuracy values? I'm just assuming that if a guy makes first team all big ten then his "correct" rating should have been 5 stars and if he made the second team than his "correct" rating should have been 4 stars.
November 4th, 2009 at 6:56 PM ^
...you can invest in one of those t-shirts from the MGoStore and enjoy free recruiting analysis here
November 4th, 2009 at 4:49 PM ^
What if they didn't make first or second team? Does that mean 3* or did you judge star rating below that point by number of starts? Like Kevin Grady for example was a 5* guy so what would his "true rating" be? I think you could call him a 4* full back because he can block pretty well, has the speed to get to any LB and is a powerful runner.
November 4th, 2009 at 4:54 PM ^
1. Most FB do not get above 3*
2. He was a 5* coming out of high school as a RB not a FB
3. He doesn't even start over Moundros.
Therefore I'd say that his true value is 3*
November 4th, 2009 at 4:54 PM ^
Good question - in this case the analysis is very limited in that it excludes any players who didn't make first or second team all big ten. So I can't make any conclusion about which site overrates bad players more often but rather which misses really good players less often. Obviously it's not ideal but to extend beyond that we would have to come up with a rating methodology for the "true rating" of players that didn't make the all big ten team. That could be something like multiple year starter = three star "true rating" but there are problems with that as well.
November 4th, 2009 at 4:51 PM ^
ESPN owns Scout and ESPN always has an agenda, even in recruiting. Rivals reports what it sees and hears, but ESPN manipulates its rankings to make the season ending recruiting races more dramatic. It's the same as all the meaningless preseason rankings talk they shove down our throats for 3 months before a game has even been played.
November 4th, 2009 at 5:02 PM ^
I dont think ESPN owns Scout......Scout is part of FSN, IIRC.
You may be confusing Scout with Scout Inc.
Scout Inc is part of ESPN. I believe it was a Todd McShay creation or something like that back in the day.
So, its 2 entities.
Scout
Scout Inc (ESPN)
But, keep the tin foil hat on! j/k
November 4th, 2009 at 5:03 PM ^
I'm pretty sure that Fox owns Scout - hence why their CFB rankings are always just reprints of what you find at cfn.
http://www.newscorp.com/news/news_253.html
November 4th, 2009 at 5:19 PM ^
Scouts Inc. with Scout
November 4th, 2009 at 4:55 PM ^
I had no idea that ESPN owns Scout. It would be interesting to see what type of players typically are scored differently by ESPN and Scout . . .
November 4th, 2009 at 5:05 PM ^
I know Scout tends to hand out a LOT more fifth stars than Rivals. Could the relatively high frequency of 5* players on scout versus rivals have anything to do with the discrepancy? The two sites obviously have different standards for 5* players.
Currently, rivals has 21 5* players and scout has 50.
I like your analysis, but I think we would get a better feel for which recruiting site is more accurate buy lumping together all the 4* and 5* and figuring out which site is more likely to overrate a player. This would require some kind of metric to define a player as a bust. Now I'm just rambling. Anyways, just my $.02
EDIT: Maybe you could judge a player's "true rating" by their draft position in the NFL?
November 4th, 2009 at 5:14 PM ^
Good idea on the overrating - could look at how many 5 stars from each site are clearly busts e.g. non-starters.
I'll give the draft position idea some thought. One concern would be undervaluing players who were drafted late/undrafted who then go on to be fantastic players. Similarly, we wouldn't want to give Ryan Leaf a true value of anything more than negative 8 stars.
November 4th, 2009 at 7:31 PM ^
Are you looking for which site is better at predicting NFL talent or college talent? Going by NFL draft position seems like a bad idea as many all conference players that do not get drafted in the first round may come as busts. As an example, Pat White was an absolute stud and had to be multiple all-conference, but was only drafted in the 2nd round and gets played sparingly.
Ryan Leaf, though a bust on the NFL level, had what I'm sure would be considered a 5* worthy career at Washington St. How does a player like him fit in? High ranking, Great career, high draft position, big bust..
November 5th, 2009 at 8:08 AM ^
Good point - because I am just trying to evaluate the ratings services' predictors of successful college players it would be a bad idea to include any consideration of NFL draft position. If we're talking college, Ryan Leaf is without a doubt a 5 star player.
November 5th, 2009 at 8:10 AM ^
Because I looked only at first/second team all big ten players the fact that Scout gives out more 5* ratings actually goes further in supporting the conclusion that Rivals is better. Scout misses the top tier talent more frequently than Rivals even though they hand out top tier ratings more often.
November 4th, 2009 at 5:40 PM ^
I think busts are a better way to determine the success of players; there's a lot of factors that go into whether someone is first or second team all-conference, but being neither is usually a bad sign.
Also, a scout five star is less talented on average than a Rivals five star because there are more of them, but that doesn't mean the site is worse, just that it defines rankings differently.
November 4th, 2009 at 9:26 PM ^
Did you look at whether the difference in "real" ranking was above or below the original ranking? For example, did Scout more generally give a fifth star to a 2nd-team All Big Ten player (5 star for Scout vs. 4 star "real"), or did they more often give that player 3 stars? Since Scout gives more 5-stars, you would expect their rankings to be slightly over-valued compared to your definition of the player's "real" ranking as compared to Rivals. If Scout missed in the other direction (handing out rankings that were too low vs. the "real" ranking, when their 5-stars already mean less by virtue of handing out more), then that indicates a bigger difference in the two sites' rating competency than if they miss to the high side. Did I explain that clearly? It's all so clear bouncing around in my head, anyway.
November 5th, 2009 at 8:17 AM ^
Believe it or not, neither Scout nor Rivals ever gave a 5* rating to a player with a "true rating" of 4* (again, only looking at players who made second team all big ten). You're dead on with your second point - Scout was more likely to under-value players even though Scout gave out more 5* ratings.
November 4th, 2009 at 7:34 PM ^
neither are worth the money. You get everything you need here.
The holidays are coming up and we need to share a little holiday bling with Brian and the guys. Don't know what I would do if I could not visit this site.
November 4th, 2009 at 8:49 PM ^
The best one is whoever rates UofM players the highest
November 4th, 2009 at 8:56 PM ^
It's worse on Scout than on here. And the sad thing is people pay $9.95 to circle jerk when you can do it for free on here! :P
November 4th, 2009 at 11:06 PM ^
I bought the one month sub, just to try it out. It was sorely lacking. I dropped the next month, considering myself pathetic for purchasing a rumor mill about high school boys making a decision to go to college.