Jimmystats: When the Sites Disagree

Submitted by Seth on July 7th, 2016 at 3:30 PM

When Ace was scouting Kai-Leon Herbert, the offensive tackle prospect who announced his commitment to Michigan this week, we took note of the big disagreement in his rankings:

Scout Rivals ESPN 247 247 Comp

4*, #22 OT,
#183 Ovr

4*, #10 OT,
#61 Ovr

4*, 83, #15 OT,
#106 Ovr

3*, 86, #72 OT,
#683 Ovr

4*, #22 OT,
#166 Ovr

Some variation for a project recruit with big upside isn't that weird, but one site having him threatening the top 50 while another has him barely among the top 700 is some serious disparity. BiSB even wondered aloud if big disagreements like that portend anything for a guy. And well, I have a database. Let's see.

Methodology

I'm going back to my STARs ratings, which are a composite of the four main recruiting sites' scoring systems/stars/rankings normalized to a sliding scale of five stars. I cut out specialists, then used only players for whom we have at least three rankings to go from, and ran a standard deviation.

So What Happens When They Agree?

This wasn't very useful because most of the guys with high agreement were very well scouted (duh) and a few were like the toppomost of the tippytop. There were 12 guys who sparked almost total agreement (ordered by rating):

Player Class Rivals Scout ESPN 247 STDev
Rashan Gary 2016 5.0 5.0 5.0 5.0 0.00
Jabrill Peppers 2014 5.0 5.0 5.0 5.0 0.00
Ben Bredeson 2016 4.6 4.6 4.6 4.6 0.00
Justin Boren 2006 4.6 4.6 4.6   0.00
Chris Fox 2013 4.2 4.2 4.2 4.2 0.00
Josh Ross 2017 4.0 4.0 4.0 4.0 0.00
M. Witherspoon 2008 4.0 4.0 4.0   0.00
Ben Braden 2012 3.6 3.6 3.6 3.6 0.00
Christian Pace 2010 3.6 3.6 3.6   0.00
Teric Jones 2009 3.6 3.6 3.6   0.00
Frank Clark 2011 3.4 3.4 3.4 3.4 0.00
J.T. Floyd 2008 3.2 3.2 3.2   0.00

Teric Jones, Christian Pace, and Chris Fox lost their careers to injury, but after that only Marcus Witherspoon (for off-field reasons) didn't end up a regular starter, pending the careers of Bredeson (who may be the first tackle in this year) and Rashan Gary (who may be the greatest tackle in years).

So What Happens If They Disagree?

Here's the 15 biggest disparities. I've highlighted the biggest outliers.

# Player Class Rivals Scout ESPN 247 STDev
1 John Ferrara 2006 3.4 3.2 1.8   0.90
2 Jason Kates 2006 4.0 2.3 3.2   0.88
3 Austin White 2010 3.6 4.0 3.4 2.0 0.87
4 Kevin Koger 2008 4.2 4.2 3.0   0.69
5 Conelius Jones 2010 3.4 2.8 3.4 2.0 0.67
6 William Campbell 2009 4.8 4.6 3.6   0.64
7 Junior Hemingway 2007 3.6 3.8 4.8   0.64
8 Brandon Smith 2008 4.2 4.4 3.2   0.64
9 Chase Lasater 2017 3.4 2.8 3.2 2.0* 0.62
10 Brandon Moore 2008 4.0 3.2 4.4   0.61
11 Isaiah Bell 2009 3.6 3.2 4.4   0.61
12 Mark Huyge 2007 3.4 2.5 2.3   0.60
13 Rocko Khoury 2008 3.6 3.4 2.5   0.59
14 Boubacar Cissoko 2008 4.6 4.6 3.6   0.58
15 D.J. Williamson 2010 3.2 2.5 3.6   0.56

*247 hasn't ranked Lasater yet

Big winners on there are Koger, Hemingway, and Huyge, though BWC turned out okay once Hoke got his hands on him. FWIW the one guy Scout and Rivals really disagreed on before ESPN entered the ring was Alex Mitchell. Meanwhile I had to go back to the blogspot page to find Brian's take on the huge disparity over Junior Hemingway:

So, yeah... those numbers above disagree fiercely. Hemingway is either in the top 10, 20, 30, or 40 receivers in the country, depending on who you listen to. Rivals went so far as to downgrade him to a three-star after season's end for reasons unknown (read: plain old provincialism on the part of that particular region's rankings guru). Meanwhile, ESPN is freakin' out over here. Scout and Creepy Tom Lemming split the difference.

Ironically he turned out exactly as advertised:

Leaving aside his exact proportions of shirtlessness for the moment, Hemingway is a leaper capable of ridiculous grabs. His overall athleticism has been questioned by those skeptical of his talent, but no one debates his body control, leaping ability, and hands.

The most noticeable thing other than how many of those guys didn't pan out was that ESPN was usually the oddball.

How Do the Sites Compare?

Was ESPN always so odd? They ranked Koger as a DE (Brian did posit at the time that a move to DE was likely, since Michigan had few), while Rivals and Scout had him the #4 or #6 tight end. But it came up enough I had to look at them versus the average to see if that was normal:

pasted_image_at_2016_07_05_02_43_pm_1024

(click makes big)

Mathematically (by deviation of squares) they were by far the most likely to disagree with their peers:

ESPN: 39.73
Scout: 31.74
Rivals: 24.09
247: 13.80

If they were highly accurate that would be interesting, but as you see by the outliers, only one of the dudes they seemed super-way-excited about even started (though Metellus has time).

It was also interesting to see which players each site was most panting/skeptical about. I'll highlight if they got it right:

WHEN THE SITES ARE BEARISH:

Rivals Scout ESPN 247Sports
Nolan Ulizio (-0.5) Jason Kates (-0.9) John Ferrara (-1) Austin White (-1.3)
Bryan Mone (-0.5) Brandon Moore (-0.7) Kevin Koger (-0.8) Conelius Jones (-0.9)
Jr Hemingway (-0.5) D.J. Williamson (-0.6) Brandon Smith (-0.7) Jake Butt (-0.5)
Chris Wormley (-0.5) Davion Rogers (-0.6) Will Campbell (-0.7) Jourdan Lewis (-0.4)
Patrick Omameh (-0.4) Reuben Jones (-0.5) Rocko Khoury (-0.7) Devin Asiasi (-0.4)

Some of those guys it's too early to tell. But I might be a bit more leery of Rivals skepticism and hoping Reuben Jones proves Scout can be wrong.

Meanwhile in high expectations, here are the guys certain sites thought would outperform the consensus of their peers:

WHEN THE SITES ARE BULLISH:

Rivals Scout ESPN 247
Jason Kates (+0.9) Austin White (+0.9) Jr Hemingway (+0.7) Chris Wormley (+0.6)
Mark Huyge (+0.7) P.Omameh (+0.6) Isaiah Bell (+0.7) Nate Johnson (+0.5)
John Ferrara (+0.6) Josh Furman (+0.5) Brandon Moore (+0.5) Dennis Norfleet (+0.4)
Greg Mathews (+0.5) Sam McGuffie (+0.5) Q.Washington (+0.5) Erik Magnuson (+0.4)
Conelius Jones (+0.5) (tie* +0.5) Conelius Jones (+0.5) Mason Cole (+0.4)

* Marrell Evans, Brandon Smith, Tom Strobel, De'Veon Smith, and John Ferrara.

Some of the guys I didn't highlight were fine but only insomuch as they met their recruiting expectations. At least Rivals knew before everyone else that Huyge was unkillable but otherwise woooooof. Meanwhile Scout got burned by some major athletes (Furman and McGuffie at least wound up starting elsewhere), but the only real diamond they pointed out was Omameh; the five-way tie varied from slightly too positive (D.Smith, Ferrara) to vastly overrating (Evans, Strobel, B.Smith).

Of course these are just small sample sizes—useful for gauging extreme outliers but little else. So I used scatter charts to see if there was a major difference in the aggregate, tracking all their recruiting ratings by deviation from the mean and their starts/eligible seasons. The best scouting site would have the most bubbles very high and to the right, and fewest bubbles high and to the left (guys they were skeptical about who got a lot of starts).

imageimageimageimage

A few major outliers got cut out but a picture has emerged. When Scout says a guy is good you should probably pay attention. Rivals has a low batting average but will connect as often as they whiff. ESPN appears to lose track of guys who aren't ranked at the very top, so their outliers may be more cautionary than anything. 247 plays it mostly safe but once in awhile takes a calculated risk that usually pays off.

Or that they're huge Norfleet fans. One understands.

What does that mean for Herbert and the OL this year?

image

We haven't seen this kind of distribution before, honestly. These rankings could change so much before February however that I wouldn't put much stock in them anyway. The Herbert disagreement doesn't look so bad in the STARs. With nothing else to go on, I'd say keep an eye on 247's rating to see if that jumps after the Opening, and otherwise trust that Scout has him pegged.

Comments

turtleboy

July 7th, 2016 at 5:19 PM ^

It's funny them all agreeing on Ben Braden because i remember they all agreed he was about 2-3 inches shorter and 50-80 lbs lighter than he actually was by the end of his senior year. They all agreed that his junior height and weight numbers were the last word and never updated them despite a massive growth spurt he had.

tasnyder01

July 7th, 2016 at 5:19 PM ^

Now you've got me scouring through the old webs. I could have sworn you offered one of your old databases for people planning on doing diaries.

Would love to play around with this stuff before the season starts.

TrueBlue2003

July 7th, 2016 at 5:51 PM ^

but it looks like ESPN was pretty spot on with their outliers.  Looks like 6 of 8 were correct and the miss with Koger was somewhat understandable given the position they ranked him for.  The only whiff was Bell.  Their skepticism of BWC, Booby, Ferrara, etc was spot on as well as their love for Hemingway.

TheBorg

July 7th, 2016 at 8:08 PM ^

my impressions over the past 15 years is that nationally, Rivals has the more temperate staff compared to Scout.  But thanks to the fabulously uber job by Seth, the data sets the record straight.  In the case of Kai-Leon Herbert, there appears to be no reason for the lower rating by 247; previous articles and that which 20 minutes could afford brought forth an athlete that should be in their top 247.  Like you say Seth, we'll see if 247 changes his ranking after the Opening, assuming he performs well, as he's done at camps thus far in 2016.  Worst case scenario: he's an outlier early on, but will likely end up ranked somewhere between 50 and 150 on all four recruiting services rankings.

Seth

July 8th, 2016 at 9:49 AM ^

Depends who you follow. I'm mostly interested in Michigan recruiting and a few players that go to rival schools who are so good we'll have to face them more than once. I would personally prefer a site that gets an A in Michigan recruiting and averages a C across the country than get B Michigan coverage from a nationally B average site.

Wolvermarine

July 8th, 2016 at 9:26 AM ^

FWIW I would expect Herbert to drop quite a bit on Rivals next update, though still in the top 200 or so. Mike Farrell alluded to the fact that he was pretty bad at the recent 5 star challenge (the video out there on him was not impressive). I also expect 247 to move him up a couple hundred spots so that will likely even out. He's obviously a bit raw now but he has the kind of tools that could make him an All-American under this coaching staff.

lesgoblue

July 9th, 2016 at 7:00 PM ^

Love the analysis Seth, and the way you tried to find who to believe when there's a discrepancy  across services is especially helpful.  I wonder if one service is more accurate than the others on a region by region basis.  In other words, instead of just concluding to go with Scout when they're higher on a player than the others, I wonder if it can be concluded to go with Service X if the recruit is from Florida, but go with Service Y if the recruit is from the Midwest.

JenniferTPeterson

July 10th, 2016 at 4:30 AM ^

I am making $89/hour working from home. I never thought that it was legitimate but my best friend is earning $10 thousand a month by working online, that was really surprising for me, she recommended me to try it. just try it out on the following website. ►►► www.NetNote70.com