On star gazing

Submitted by MGoCali on June 26th, 2018 at 1:16 AM

The common theme in the past year or so worth of "hello" posts on the front page is the lackluster state of recruiting services. First there was scout/247 consolidation, and then it seemed rivals only scouted rivals camps, and then ESPN fired everyone except Sports Center anchors and Stephen A. Smith. Given all of that, it seems that stargazing is less informative than it was before. I'll quantifying this below.

While it's true that the top talent is still scouted and ranked highly by everyone (as in the top 50 or so), the recruits in the 3.5-4 star range are sometimes barely even scouted by all services, and thus the 247 composite is a lot less informative. You get a sqrt(n) improvement in measurement error for n times more measurements, so if you only have one legitimate scouting measurement of a given recruit instead of four in years past, the predictive power of the composite decreases by a factor of two. That is, a recruit with a composite rating of the 150th best recruit may truly be between 100-200 in the past with four services, but now there is no way to tell if he's between 50-250 if only a single site gave a thorough eval. If two sites have legit evaluations, then the predictive power of the composite is sqrt(4/2) ~ 1.4 times  or 40% less informative. Finally, even if all three have legit evals it's still sqrt(4/3) ~ 1.15 or 15% less informative.

It's clear that stargazing is less informative than it used to be. Even a consensus #1 is 15% less certain than it was in the past due to an additional missing independent measurement. It'll be nice to see this in the data in the next few years, but put me on the record predicting starz to correlate with college success and NFL draft stock less than it used to. 


big john lives on 67

June 26th, 2018 at 1:48 AM ^

Excellent observation. I have felt for some time the inexact science of rating recruits has been on a steady decline. To me, this places an even higher value on and trust in Jim Harbaugh’s ability to evaluate football players. 

Blue in Paradise

June 26th, 2018 at 1:52 AM ^

ESPN won’t have its own ratings in 2 years.  There are some smaller independent agencies out there, not sure if they provide national or regional coverage.

if I were 247, I would look into incorporating some new firms into the composite. If not, Jim S. has his own database- that includes virtually every prospect for the past 50 years.


June 26th, 2018 at 2:20 AM ^

This is one of those industries where the natural progression of capitalistic consolidation is not good for service quality. Wait, I guess that's most industries... Anyway, regional sites are fine, but they will introduce a lot of bias. I'm afraid the glory days are behind us. Unless, we start paying for the service somehow, like actually subscribing to paid recruiting sites, things won't improve. I'm not willing to do that, and the average fan is not either, I would guess. 

Blue in Paradise

June 26th, 2018 at 8:21 AM ^

Speaking of capitalism, here is something I don’t get:  Why is it rare to see recruiting analysts get picked up for jobs on the staffs of P5 teams?  

Supposedly, Urban Meyer puts a lot of faith is star ratings (which also creates a universe destroying circular equation since an OSU offer is a major factor in said ranking) and he has a large budget for football staffers.  So why is Urbz not out there hiring the Josh Helmholtz’s of the world as analysts or recruiting coordinators?  

Wouldn’t hiring the top 5 of these recruiting give a coach the leg up on the other star gazing coaches?  Certainly Josh and the others would prefer making $100k and the possible upward trajectory of working for OSU / Bama / Georgia/ etc... than working for Rivals.


June 26th, 2018 at 10:39 AM ^

Damn. Ive long held the same sentiments as Blue in Paradise, but that makes a lot of sense. I know it shouldn't at this point but it still continues to amaze me how crooked this game can be, and also makes me respect the hell out of Michigan for playing it straight. Id rather lose out on a few of these recruits than sacrifice our integrity. 

Blue in Paradise

June 26th, 2018 at 11:30 AM ^

We are generally on the same page with our comments but that doesn't make any sense to me.  I don't think Urbs / OSU need help from these guys.

The ratings bump for picking OSU / Bama / Clemson (I would say that Michigan generally gets this same bump for the lower rated guys) actually makes sense.  If the coaches who have proven their chops want a 3* kid - the recruiting analysts should reconsider the ranking.


June 26th, 2018 at 11:13 AM ^

Well, first you have to believe these guys really know what they are doing. Second it's easier to look at a player and say he's good at certain things and give him a 4 star rating. That does not always correlate to being a good fit in a coaches system.

Also coaches are offering kids they see in person, meet and many times work with them at a camp so they know what it's like to coach them and even get a feel for their personality.  All things guys sitting in front of a computer and looking at highlight tapes don't get a feel for.

Long story short trust a coach to find the players they need to fit in their system.

UgLi Eric

June 26th, 2018 at 1:52 AM ^

Nice take MGoCali. Logically it makes total sense. I was just thinking the same thing, especially after the Velazquez commitment. It's strange to see someone who is a perfect fit in our system be ranked in the 900s, but when you look at his Hello post, it's clear that scouting from a good staff should matter a lot more than what's publicly available. Our staff have resources and connections all over the coubtry reporting back to them, while the state of the recruitment sites seems like if the player didn't show up or play well during their camp, they won't give them much more than a cursory glance. 

I'm also less inclined to believe those who argue about past NFL success, moderators included, as indication of future success. 

I will concede that when a prospect trends up or down aggressively, it probably means more than their overall ranking, within 100 or so places, as you mentioned. 

But I can't wait to hear what others, who are very much experts, like Magnus, have to say about this topic. 


June 26th, 2018 at 2:13 AM ^

The uncertainty in my post was invented, so don't read into the 100 or so places that I quoted. I literally made that up. It's the factor that matters (i.e. two times less certain with 1 vs 4, 1.4 times less certain with 2 vs 4, and so on). It's impossible to know the intrinsic ranking of a recruit and also impossible to know the uncertainty in the estimate of that ranking without many many more recruiting sites. Only then could you start to bootstrap (technical term) the rankings and set up legitimate models of all of this. 

It further complicates things that their ranking (which I'm assuming to be a random variable in this post) is decidedly not random. By that I mean that regional biases (southern recruits tend to be over-rated), large school bias (small town recruits get overlooked), and a host of others play into this if you were really going to try to model this statistically. It would honestly need a full-on study, and it would be worth some research papers in peer-reviewed journals. I could imagine a statistics PhD on this topic, if anyone is looking to go back to school. 


UgLi Eric

June 26th, 2018 at 3:39 AM ^

This would be an interesting study. Obviously there's interest and money. It might not even need to be a PhD, could be an un or underemployed member of this here board, if ThaiStreets is still available, crowd funded by our own community to look into all questions and theories thus far. 

I'd also be curious to see what confirmation bias, FOMO, and even the inevitable post commitment ratings boost to a major program does to a ranking. I then wonder if actual performance changes based upon ratings and recent feedback. I know as a child I was a wide receiver and played lights out after taking a protein bar. I received good feedback and thought I could only perform to that enegre level with a protein bar. It was a confidence induced placebo. 


Btw this board is fun this time of day. 


June 26th, 2018 at 7:18 AM ^

Part of the issue on Velazquez is that he’s only attended one “camp.” 

He’s gone to individual camps at the universities, but ranking scouts don’t attend those for the most part. So his rankings reflect that. 

Another issue is the fact that he’s a tweener. He’s stuck between linebacker and safety, which is PERFECT when you’re being recruited for a position like VIPER, but not many teams use that position. 

So a team like OSU that likes the kid every time he camps there, has an issue pinning down exactly where he would play in their defense. 


June 26th, 2018 at 2:05 AM ^

I should note, that as Bluey has been hammering incessantly, the data have shown quite definitively that rankings matter. I'm just pointing out that they now matter less than they did when four sites were doing a good job.

Even in this environment, they matter, just quantifiably less.

Pulled P

June 26th, 2018 at 2:46 AM ^

I agree that rankings matter. But they aren't gospel, never was.

It's an inexact science to begin with, and now because of the reason the OP has aptly stated, they are just projections of a handful of people at 247 and Rivals. Also what schools do (offers, evals, boards) affect the rankings and vice versa. In the past I guess you could say the rankings had more objectivity because it had more input, but again, now less so.

And lastly what the hand-wringers should keep in mind is that rankings constantly change. When the staff spotted Karlaftis early when he was outside the 300 and pursued him, was the 3-star criticism warranted then? Should we, as fans, apologize to the staff now that he's a five-star, #12 by 247?

All this is to say rankings matter, but they're just a datapoint among many. The problem with us fans is that oftentimes it's the only datapoint we have. Sure, we can question the decisions, express concern, and that's part of following recruiting. But what I find hilarious is the few who are so adamant in their belief in the composite rankings that they are so sure 3-star offers are the result of gross incompetence. I'd say they should take the rankings as something less than gospel.




June 26th, 2018 at 9:54 AM ^

Few people would argue that, in broad strokes, rankings don't matter.  But what Bluey and others hammer on is the difference between the very broad, very inexact bucketing of "starz", wherein a guy ranked 340th in the country is a 4* and the guy ranked 350th is a 3* and so, by the law of starz, the 4* guy is better than the 3* one.  And as you noted, it's made even worse by the fact a lot of services barely even evaluate a lot of these guys outside of, what, 2 dozen reps over a summer at a camp?

It's why I do think people should pay more attention to offers, especially those that come after guys camp places, early on.  It doesn't mean that a guy with an FSU offer and a Michigan offer who is ranked 957th or something is really a 4* in the rough, but it does show that people who would actually use that player on their team see value in him, whereas a lot of the ranking services seem to focus more on how good a player might be on a hypothetical team created in a vacuum.



June 26th, 2018 at 11:05 AM ^

"Offers are nothing more than the start of a conversation and preliminary indicator of interest."

247 National Scouting Director Steve Wiltfong to Allen Trieu last week on their podcast.

Nowadays most kids won't even talk to you unless they have an offer in hand. UM has over 300 "offers" out. You really think they are all committable? Hell no.


June 26th, 2018 at 10:29 PM ^

But not everybody immediately commits to a program, and oftentimes they are being looked at elsewhere.  There are physical limitations to people being able to signing to only one place, so it makes sense to make multiple offers to players because statistically you miss out on most of them for any number of reasons.  Hell, half the time schools offer guys who are already committed elsewhere.  But generally speaking, it's probably safe to assume that if a guy gets an offer from a school he visited recently and/or camped at, it's likely committable or damn close.

But my larger point above is that your particular approach to determine what offers are "committable" and which aren't is seemingly random.  One kid has a half-dozen offers from major P5 programs and you consider them virtually all unusable.  Another kid, with similar rankings and pedigree, has another collection of offers and you'll say 1 is committable and the others aren't, or some are now but might not be in the near future.  And yet, you don't have any insider knowledge AFAIK.  It just feels like you went to a message board somewhere and read someone say it wasn't committable, or looked at existing offers from other clubs and extrapolated from there, about the state of a player's offer list.  And so I call BS on that because it's just your opinion you pass off as fact, and then aggressively argue with anyone who call you on it.


June 26th, 2018 at 3:39 AM ^

Even with four services the scouting outside the top 50 or so players was very uneven and unreliable. As compared to the scouting for the top programs in the nation the people that evaluate for the services, with some notable exceptions, are glorified amateurs. If they were truly among the best at what they do they wouldnt be working for pennies on the dollar.

With the state the services are now in they are mostly to be ignored. They are one insignificant data point among many. You cant truly scout a player just from film or in a camp setting or in 7v7. You have to see a player play the game to get a comprehensive scouting report and the services have rarely if ever done that. And even if they have its unlikely that any one particular scout even knows what hes looking at outside the obvious. Again with some notable exceptions. The correlation between stars and NFL potential is skewed because everyone and their brother knows who the top 50 kids in the country are. They are the guys who have been looked at since 7th grade and the services can hardly miss there.


June 26th, 2018 at 3:44 AM ^

One other major point is the uneven level of competition. Guys in the NE are lightly scouted and always have been due to the lower level of competition.Guys from California, Florida, Georgia, Texas, Ohio and Pennsylvania get ratings bumps just because of the location they play. And even in those states if your at a small school or one that historically hasnt been very good you may be somewhat overlooked.


June 26th, 2018 at 4:18 AM ^

All good points, and I agree with them.

My statements in the OP take nothing additional into account and assume the each of the 1, 2, 3, or 4 services' rankings are an estimate of the true rankings, which I treated as random variables. Of course, it's all more complicated than that and even than you stated above. In addition to what you listed, there is a program to program bias for certain shapes and sizes. Vipers don't fit a stand 4-3 very well, and so they are underrated nationally compared to their worth to our defense, etc etc etc.

I certainly get the complexity of legitimately quantifying rankings. It has definitely been shown that there is a strong correlation between number of stars and ability in college and draft stock. You are right, it's stronger 5 vs 4 than it is 4 vs 3, and that makes sense to me given that the recruiting industry has limited resources to evaluate players. Still, no one with a brain denies that rankings matter to zeroth order. The point of this post was to point out that if we boil down recruiting evaluation to the simplest form it can take, independent measurements of random variables, then there is a quantifiable deficit in the value of the industry compared to a few years ago. 

Boogie down

June 26th, 2018 at 5:07 AM ^

Guys, don't forget the fact that southern players get a bump just because they play in the south. Sam was talking about that this winter. More subscribers mean they appeal to members to keep numbers up, so those players get more interest and thus better ratings. He mentioned that northern and midwest players have out performed their ratings more than southern and west coast players. It doesn't mean they were worse to begin with, they were just lower rated because they only give out so many 4 and 5 star ratings.


June 26th, 2018 at 10:58 AM ^

The SEC has had the most players selected in the NFL draft every year for the last decade or so. They have better players and it has nothing to do with recruiting site subscribers. Besides, isn't this blog always claiming a 3 star from Florida is better than a 4 star from Michigan? It's insane how people keep moving the goal posts around here to fit whatever narrative that is convenient to them.

Beilein 4 Life

June 26th, 2018 at 12:31 PM ^

Yeah, Bluey can’t stand when people move the goal posts on an argument. That’s his go to move.

He did have that one post a couple days ago where everyone was surprised at how reasonable he sounded. Then he went on to be the same condescending poster he was back when he was Maizen. I guess we will always have that one single post. Memories


June 26th, 2018 at 6:45 AM ^

I agree. I believe it still holds some weight, but by comparison to even just a couple of years ago as you mentioned, I think the standard deviation has certainly grown. 


June 26th, 2018 at 7:18 AM ^

I always had the feeling Brady Hoke and somewhat Rich Rod looked at the rankings as their primary method of who to target.

Harbaugh seems to have more of a roll up your sleeves proactive approach.  Some misses are unavoidable with an 85 man roster.   The proof will be on the field and by those making NFL rosters.  


June 26th, 2018 at 8:53 AM ^

Hmmm I remember a Wilton Speight quote along the lines of him being the #1 or #2 QB on the staff's board. Didn't Borges have a reputation for being friends with some private QB coach and basically taking his word on which guys were good?

I believe the Mgoblog reaction was mainly, "really???"

Anyway, I doubt Hoke and co. used star rankings even for the offense. They were just wrong or unlucky or bad coaches.

Inflammable Flame

June 26th, 2018 at 7:21 AM ^

Good work on going in depth on the actual quantification. I personally just look at rankings as opinions, not actual fact that the #n-rated player is in fact that #n-rated player. It's cool to see what sites give their opinions on who is at what level talent. But in the end, it is just someone's opinion and I do see how someone can say the #4 DE is better than the #4 WR (for example). Totally different players that mean different things to their teams.


June 26th, 2018 at 7:25 AM ^

Offer list

I find the offer list to be important.  Reflects the view of several coaching staffs.  If top echelon teams of each power five conference are involved, I take comfort more so than star rankings.