Recruiting Rankings, Corporate Sponsors, And Potential Bias
Rivals ranks Five-Star Challenge camper Steven Parker 123 spots higher than any other service.
Rivals released their updated 2014 Rivals100 today, and this would've gone without more than a passing mention if not for this tweet that accompanied the release:
#FunFact about new rankings: 75 of the updated #Rivals100 Members were either at a #RivalsCamp and/or #RivalsChallenge event
— Rivals.com (@Rivals) August 19, 2013
Fans of recruiting often throw out unsubstantiated claims about bias in the player rankings; this "fun fact" from Rivals, though, is just begging for some investigation into potential issues with their rankings. Does Rivals favor recruits who show up to their camps?*
I decided to take a look at the players in the Rivals100 who are listed as participants in the Rivals Five-Star Challenge; they represent 51 of the top 100 prospects on Rivals. My rather unscientific method of looking for potential bias was to look at each Five-Star Challenge participant's ranking on Rivals and compare it to their highest ranking on any of the other three services; if there's consistent bias in the rankings, Rivals should be the high outlier for this specific set of prospects. You can pore over the full (chart?) chart here; below is a summary of what I found:
- 26 of the 51 Five-Star Challenge participants (50.9%) were ranked higher on Rivals than any of the other three recruiting services.
- 11 players from the group were ranked at least 20 spots higher on Rivals than elsewhere, compared to eight whose highest ranking was 20 or more spots above their placement on Rivals.
- Five FSC participants in the Rivals100 were ranked 50+ spots above their next-highest ranking, including significant outliers OK S Steven Parker (#46, 123 spots higher than Scout) and TX OL Demetriux Knox (#35, 92 higher than Scout). Only two such players — AZ OL Casey Tucker (#79 on Rivals, #27 on Scout) and FL LB Kain Daub (#86 Rivals, #24 Scout) — fit the opposite criteria.
- On average, the 51 FSC participants were ranked 5.3 spots higher on Rivals than they were anywhere else; that number would obviously be even higher if we were looking at the industry average instead of the next-highest rank.
Where the numbers get really interesting, however, is when we look at the relationship between Rivals, ESPN, and Under Armour. As of this year, Rivals's recruiting rankings are "presented by Under Armour." Meanwhile, ESPN and Under Armour are still partnered for the Under Armour All-American Game. When I mentioned this potential conflict on Twitter earlier today, our friend TomVH noted that both Rivals and ESPN have input into the Under Armour AA selections:
@AceAnbender @DGDestroys @Mengus22 Our guys still do most of the UA selection. From what I understand Rivals has a small part in that
— Tom VanHaaren (@TomVH) August 19, 2013
Since Rivals and ESPN both have potential conflicts of interest regarding Under Armour All-American prospects, I revisited my chart and looked for prospects whose highest non-Rivals ranking came from ESPN. Of those prospects, six are committed to the UA game. These are those six players:
| Name | Rivals100 Rank | ESPN Rank | Delta | Highest Non-ESPN Rank | Non-ESPN Delta |
|---|---|---|---|---|---|
| Alex Bars | 36 | 52 | 16 | 172 | 136 |
| Travis Rudolph | 77 | 65 | -12 | 152 | 75 |
| Deshaun Watson | 23 | 16 | -7 | 67 | 44 |
| Bo Scarbrough | 31 | 20 | -11 | 52 | 21 |
| Jalen Tabor | 20 | 17 | -3 | 30 | 10 |
| Da'Shawn Hand | 1 | 4 | 3 | 6 | 5 |
This is, to be sure, a limited sample, but I can't say I'm surprised to see that each player's highest ranking drops — significantly, in the case of the top three players on the chart — when the two services with ties to the Under Armour Game are removed. When running the numbers for the full set of 51 Five-Star Challenge participants and using the highest non-ESPN rank for Under Armour All-Americans, the gap between Rivals and the other services widens significantly — the prospects are ranked an average of 11.1 spots higher on Rivals than the other services.
Rivals received some immediate backlash when they published the tweet at the top of this post; they repeatedly replied to commenters with this explanation when pressed about potential bias in their rankings:
@travatrave It shows how many of the kids we were able to evaluate first hand and in person, no bias at all.
— Rivals.com (@Rivals) August 19, 2013
This isn't an illegitimate explanation; Rivals got the chance to see a large group of top prospects in a setting that no other recruiting outlet was allowed to attend, and that should rightfully lead to some disparity in player rankings — both to the positive and negative. The fact that the Five-Star Challenge participants skew to the positive, however, along with the trend of major outliers among Under Armour All-Americans, suggests that some bias is present when it comes to recruits who participate in a Rivals-sponsored event.
Recruiting rankings, as we well know, are by no means an exact science, and my methodology here is far from ideal. That said, the role of sponsors in recruiting rankings is worth watching with a critical eye.
----------------
*Notably, the Rivals Five-Star Challenge is only open to Rivals reporters. No other outlet is allowed to cover it.
August 19th, 2013 at 5:05 PM ^
August 19th, 2013 at 6:38 PM ^
I'll answer that question for you Nope for Drake and possibly for Malik. Drake has managed to drop every rerank due to "other kids being evaluated" I read that as he wasn't at our camps therefore he must drop.
While I'm not saying he faces the best comp but 96 receptions over 2000 yds receiving and 25 TD's and you drop is beyond me.
I guess damn him for wanting to graduate early and get to Michigan as soon as possible. Rivals apparently isn't a big fan of that.
August 19th, 2013 at 10:10 PM ^
August 20th, 2013 at 9:52 AM ^
McDowell went to the Rivals 5-star Challenge. Harris was supposed to go to The Opening but stayed home so he could work on graduating in December.
August 20th, 2013 at 10:19 AM ^
Go Blue!
August 20th, 2013 at 12:45 PM ^
Because the evidence is in: the ratings guaranttee nothing, but have been shown, year over year, by multiple looks at them, to be generally a good barometer of talent. There really isn't any doubt of that empirically any longer.
Havins said that, I agree that I trust these coaches too, and given a conflict between ratings and their judgement, will side with the coaches. But it is not a councidence at all that the coaches and ratings often agree.
August 19th, 2013 at 5:09 PM ^
pretty stupid business model. as every year passes, we have additional data points to compare recruiting rankings to college performance and NFL draft rate. if they're biased, it will be obvious in 5-6 years.
but if every service is biased... moot point. need one honest service that forgoes hype today for reputation a few years down the line.
August 19th, 2013 at 6:31 PM ^
Has anyone ever done a historical analysis of the "accuracy" of the recruiting services over time . . . their star ratings for recruits versus how those recruits actually turned out - All American, All Conference, productive starter, etc.
I think I saw something like that recently, but I could be wrong. if there is unjustified bias, it should tell on itself over time.
August 19th, 2013 at 6:58 PM ^
yea, i've seen these things done before. i think maybe mathlete did one. scout or rivals is best (cant remember which). espn is very bad. 24/7 - too early to tell, but they did steal all the midwest staff from scout or rivals, so they are expected to be more accurate in the midwest.
August 20th, 2013 at 9:56 AM ^
Rivals is best, then Scout, then ESPN.
August 20th, 2013 at 8:48 AM ^
The issue is that they aren't held accountable for their poor rankings. All they have to do is play the "well, sometimes they just don't pan out" card.
August 20th, 2013 at 9:59 AM ^
They're held as accountable as any business - people are free to cancel their subscriptions, not subscribe, or not visit their site.
If a book publisher publishes crappy books, the idiots are the ones who keep buying them. If a hair dresser mangles everyone's haircut, the idiots are the ones who keep paying them.
August 20th, 2013 at 10:44 AM ^
But, how many consumers will look at the careers of a significant sample of college athletes, then consider how those careers panned out based on expectations from rankings they read 5-years years prior, then compare those 5-6 year-old rankings across different service providers, then make a rational decision based on said comparison, to switch to the service provider that may have had better accuracy, on average?
On average, consumers do not perform this many steps in decision making. Purchases tend to be on impulse. Even if purchasing decisions were rationally-based, if you're talking a 5-year delay before revenue takes a hit, then that organization would be considered successful for those 5 years. Quarter-to-quarter and year-end earnings are the driving force behind most corporate decision-making. How many investors will wait 5 years before they see returns? Again -- impulse decision-making. Yada yada yada.
August 19th, 2013 at 5:11 PM ^
This is interesting, Ace. I think there's a potential source of bias in your analysis, though.
Let's say that Rivals invites the prospects that it believes are the best in the country, even if other sites disagree about who's best. For example, maybe Rivals invites Travis Rudolph to its event because Rivals thinks that Travis Rudolph is really good even though the other sites disagree. If they invite him, he comes, and then he's ranked higher on Rivals than on the other sites, that's not really because of Rivals' bias. They just always liked the kid better.
If you had infinite time, it'd be interesting to see this comparison run with the first rankings released for this cycle, too. I think this case would be more persuasive if the gaps showed up now (as they do) but didn't show up in the initial rankings. Then Rivals wouldn't have an argument that they liked these kids better all along.
Does that make sense?
August 19th, 2013 at 5:32 PM ^
Agreed. Isn't it just as likely, or more likely, that the recruit was invited because the service ranked him highly than it is that he was ranked highly because he attended their camp?
August 20th, 2013 at 2:53 AM ^
August 20th, 2013 at 10:02 AM ^
I was about to say the same thing.
August 19th, 2013 at 5:16 PM ^
August 19th, 2013 at 5:18 PM ^
August 19th, 2013 at 5:19 PM ^
I love you, Ace.
What? I said I love your work Ace. This piece made me happy.
August 19th, 2013 at 5:20 PM ^
Also, since you mentioned fan speculation of bias...
Our two most insufferable rival fan bases (OSU & MSU) both have their primary message boards on the 24/7 site. Something tells me it's not a coincidence that 24/7 is much higher on each program's recruits than the other sites.
MSU, for example, has no composite 4-star recruits in its 2014 class. Yet it has six(!) 4-star recruits according to 24/7. The "no composite 4-star recruits" is true despite the likely 24/7 inflation. Also notable: four of those six 4-stars snuck in with the lowest possible rating for a 4-star recruit (a 90). No MSU recruits have an 88 or 89. They're playing to the MSU fan base.
I'm sure we benefit from this somewhere, too, but all of these rankings are suspicious to me (though better than nothing on average).
August 19th, 2013 at 5:24 PM ^
Double post. Sorry. OSU sucks. Go Blue.
August 19th, 2013 at 5:43 PM ^
this thread now contains probably my three favorite posts of yours since your memorable exchange with Herm over avatars and such. that was highLARious.
August 19th, 2013 at 5:21 PM ^
But, "major outlier" needs a bit more definition. There are over 1,000,000 high school football players nationwide. Approximately 2500 will be selected by coaches/staff to play D1-FBS football, meaning these are somewhere between the top 0.25% to 1% of highschool football players. A Top 100 means the services are trying to differentiate a top 0.01% from the rest of the top 0.25%. Arguing that a player is #100 vs #300 (a whopping 200 spot difference!) Is simply suggesting that player resides in the top 0.01% vs the top 0.03%. A statistician would probably point out that the most major noise at the top (or bottom) 0.03% of a population is going to be due to biases and experimental mis-design. Obviously recruiting services are biased. But we are the chumps who buy into there being a palpable difference among the the kids rated in the top 300 in the nation.
August 19th, 2013 at 5:36 PM ^
most football players aren't even on the radar of these recruiting services. That is to say, I'm not sure that it makes sense to think of ALL high school football players as the population that the recruiting services are measuring. They are actually spending an inordinate amount of time on the top 300 recruits. How much time does Rivals spend evaluating a 2-star from Utah compared to a 5-star from California? They spend more time interviewing that 5-star player on camera than they do thinking about the 2-star from Utah.
August 20th, 2013 at 10:51 AM ^
August 20th, 2013 at 5:19 PM ^
Choosing to ignore a large swath of a population does not increase perceptible differences in the 0.03% one chooses to focus on. It is also why we get the delight of players like Kovacs, Ryan, and Hart blowing up 5-stars on the field after being glossed over by a few scouts.
Each of your responses remind me of the pleasure we all witnessed in 2006 when a kid named Dan Bazuin rocked our offense on numerous plays for CMU. He was a kid from McBain MI who travelled to the metropolis of Lake City (kind of a suburb of Cadillac) to play football. He may have come from an unscouted high-school, but he more than proved he belonged on the field lined up against Jake Long, Chad Benne and Mike Hart. Thankfully, Michigan handled their business against CMU that day, proving that a team full of highly scouted players is always better than a team of under-scouted players (oh... until the next season opener?)
Essentially you have both pointed to scouting services' selection bias. You have not made an argument against the difficulty of defining differences among the top 0.03% of a 1,000,000 plus population.
August 19th, 2013 at 5:36 PM ^
If I find out that recruiting rankings have sullied the good name of college sports recruiting then I'm gonna stamp my feet and make a sound like a cartoon tea kettle.
August 19th, 2013 at 5:58 PM ^
I'll just leave this here.
August 19th, 2013 at 7:32 PM ^
Mother of god...
August 19th, 2013 at 8:47 PM ^
The size difference makes it look like the kid he did that to is only in middle school.
August 19th, 2013 at 5:58 PM ^
FSU ran into the problem of relying on scouting services and Bowden admits it.........GO BLUE from New England!!!!!
August 19th, 2013 at 6:09 PM ^
I'm not sure if I'm reading it wrong or a number(s) is wrong, but I don't think the gap between Scout (27) and Rivals (72) rankings for AZ OL Casey Tucker quite meets your 50+ criteria.
August 19th, 2013 at 6:30 PM ^
August 19th, 2013 at 6:30 PM ^
August 19th, 2013 at 7:58 PM ^
Considering that Nike sponsors 66+% of FBS schools that statement doesn't mean much. Additionally, only 14 schools from the B1G, SEC, PAC12, and BXII are sponsored by shoe co's other than Nike.
Also, also.. the teams not sponsored by Nike aren't, in general, those who are very good at recruiting. Texas, Oklahoma, Alabama, Florida, Gerogia, LSU, OSU, etc. are all Nike.
August 19th, 2013 at 6:38 PM ^
There is bias in all reporting. The question is, will Rivals sacrifice their legitimacy for profit? If they are like every other news service in the internet age, the answer is a resounding "YES."
That said, are they more biased than the other guys? We'll see who ends-up being the most accurate. There is no doubt that more and more people are paying attention, and the sites/services that produce the best results will be rewarded down the line.
But ESPN is obviously the best. Right now.
August 19th, 2013 at 8:09 PM ^
should all never be the same because it is subjective evaluations by different people. A prospect's scouting report from one person may be different from another person.
If the rankings are all the same due to groupthink(all are guilty of it) because if you have a lot of outliners, average fans would think you're retarded for ranking prospects differently.
This is true of NFL draft process. I have had the pleasure of seeing the final team draft board and they are completely different than the one you see from the websites because of the information they have gathered from gametapes and background checks as well as coaches/system fit.
August 19th, 2013 at 9:40 PM ^
All that to say truth will out in the long term and we already know that rankings are biased (shit, , if coach would have played me you wouldn't know who Tom Brady is!). So why make a big deal about it?
Recruiting rankings are interesting fodder for the offseason but please don't put too much faith an entire cottage industry driven by has beens and never weres....
August 20th, 2013 at 6:55 AM ^
Are they, or are they not getting a little ahead of themselves here? I pulled this off of elevenwarriors blog post.
The Last Frontier
The scene is set.
Pasadena awaits the Rose Bowl every January, and every four years during the BCS era, the Arroyo Seco welcomes the national championship game. The 2013 season is one of those years.
Less than two weeks remain until the start of college football. But already No. 1 Alabama and No. 2 Ohio State seem destined to face off in a dream matchup. Alabama-Ohio State, programs that ooze tradition. Nick Saban vs. Urban Meyer, a battle of the sport’s top two coaches. Tuscaloosa and Columbus, college football towns to the core.
All that’s left to do is win 13 games – or sneak in the side door at 12-1.
Expectations at Ohio State merge at undefeated and unrealistic. However, in 2013, national title talk in mid-August is justified. The Buckeyes are an amalgamation of hype and substance. Rock star coach, Heisman candidate quarterback and a potent cocktail of returning players and captivating newcomers.
It’s clear after a 24-month span that included scandal, lackluster on-field results, the hiring of a new coach and the rebirth of one of college football’s bedrock programs, Ohio State once again has the nation’s attention and respect.
But despite owning the longest winning streak in the country and receiving preseason praise, the Buckeyes are content on ignoring all the buildup swirling around them.
“I don’t think it matters too much because you can be a preseason All-American and not do anything,” said junior defensive lineman Michael Bennett, when asked about the lofty preseason rankings. “You’ve just got to come out and play.”
“We didn’t get to do all the things we wanted to do. We’re still real hungry.” - Noah SPENCE
Preseason polls and watch lists exist partly to generate conversation. It works every year. The pitchforks come out and criticism rains down from Tallahassee to Ames. Inside locker rooms, rankings aren’t even a talking point. For most players and coaches, they’re irrelevant.
“Everyone else is talking about that. We’re not,” said sophomore linebacker Josh Perry. “We’re just trying to win a Big Ten championship. We’ve got other goals, we’re just trying to take it one game at a time.”
Games against Ohio State are never diminished. When Big Ten opponents knock off the Buckeyes, it can make a year or sometimes save a coach’s job. But during 2011, there was an emptiness that accompanied a victory over the evil empire of Ohio State.
There were no losses last year. The Buckeyes did regain their status as the hunted, though. And few times in the past has it been as palpable. Every coach and player was asked about the football factory in Columbus at Big Ten media days.
“Everybody in the Big Ten is going to be coming after us, which is what we expect,” Perry said. “We’ve got to expect their best shot. People are going to be turning on their TVs all around America. Whether they like the Buckeyes or not, they’re probably going to be watching. There’s definitely that target and that spotlight on us. We’ve got to be ready to go.”
Teams thrive on banding together and proving doubters wrong. Whether it’s college football or cricket, negative prognostications are used as fuel. One year ago, Ohio State had to answer questions about motivation. Many pundits thought there was nothing to play for. That simply wasn’t true, the Buckeyes said. And they proved it wasn’t difficult to get excited by rattling off 12 consecutive wins.
Every possible reward is available to Ohio State in 2013 – Big Ten championship, national title, etc. It hasn’t tasted defeat since the Gator Bowl on Jan. 2, 2012, but the Buckeyes remain starved for success.
“We didn’t get to do all the things we wanted to do,” said sophomore defensive end Noah Spence, about the postseason-less 2012 season. “We’re still real hungry.”
Said Bennett: “The country’s gunning for us. We’re not just a small team that goes out there and expects to go 5-7 or something like that, especially now after last year. We’re expected to be the best. We’ve always had that expectation. But it’s good now to have more people pushing you for it.”
So 12 days from the season opener, Ohio State waits anxiously to begin its 124th season. It’s been more than a decade now since the program’s last national championship. Listening to Meyer on Saturday makes one think another notch will be carved into the Big Ten’s futility meter.
“We're not ready to play a game yet,” he said. “The good thing is we don't have to be.”
Ranked No. 2, the only task for the Buckeyes is to win. On Aug. 31, the first of 13 tests will be administered.
August 20th, 2013 at 8:12 AM ^
TL:DR
1. Just link it for discussion
2. Totally jacked the entire article which is a no-no.
3. Fuck Ohio
August 20th, 2013 at 10:43 AM ^
August 20th, 2013 at 7:57 AM ^
I think it is probably just as possible that rivals is ranking the kids higher because they have had a chance to actually closely evaluate these kids. Now I'm not saying there is no bias at all, because that would be silly, but I don't think the fact that the rank kids that they have had an opportunity to closely evaluate confirms that point.
I mean, think about our coaching staff. I have read here that they don't offer guys until they get a chance to evaluate them in person. They realise that there is a lot that can be missed from just watching highlight videos and looking at 40 times etc. Sometimes it happens that they get a chance to see a guy up close and realise he is much better than they originally thought (think Stribling).
Now if other sites havven't had the same opportunity to closely evaluate these same recruits that attended the 5 star challenge that would likely be a very plausible reason for the difference in ranking. That, coupled with the fact that sometimes people just disagree are just as likely (IMHE) to cause the rating fluctuations as bias is.
Again, I'm not saying I don't believe that there isn't some bias in player rankings but if there is I don't think this is necessarily proving that point.
August 20th, 2013 at 10:33 AM ^
It stands to reason that an objective evaluation would result in a more precise appraisal, which goes both ways -- positive or negative. The fact that participants tend to get ranked higher than elsewhere after participating indicates that this is not an objective evaluation.
The inherent conflict of interest sheds further doubt on the process, as Rivals has reason to inflate the rankings of participants: Players getting higher rankings due to attending the Rivals camp gives players an incentive to attend said camp. This generates immediate revenue for the organization as well as bolstering its prestige by making their camps in greater demand among the most serious prospects.
Thus, plenty of room for skepticism as to the legitimacy of their process. Howeva, Rivals is a for-profit organization, not the high-school athletics equivalent of Consumer Reports, so this behavior is to be expected. They exist to generate revenue -- to the extent that accurate information genereates the most revenue, they will be accurate, but if skewing information generates the most revenue, then they will skew.
August 20th, 2013 at 9:19 AM ^
Is implied without actually being said.
But it makes sense that they would rank kids higher that they've actually seen up close and personal, would Stribling have gotten offered if he didn't camp at Michigan?
Comments