On star gazing
The common theme in the past year or so worth of "hello" posts on the front page is the lackluster state of recruiting services. First there was scout/247 consolidation, and then it seemed rivals only scouted rivals camps, and then ESPN fired everyone except Sports Center anchors and Stephen A. Smith. Given all of that, it seems that stargazing is less informative than it was before. I'll quantifying this below.
While it's true that the top talent is still scouted and ranked highly by everyone (as in the top 50 or so), the recruits in the 3.5-4 star range are sometimes barely even scouted by all services, and thus the 247 composite is a lot less informative. You get a sqrt(n) improvement in measurement error for n times more measurements, so if you only have one legitimate scouting measurement of a given recruit instead of four in years past, the predictive power of the composite decreases by a factor of two. That is, a recruit with a composite rating of the 150th best recruit may truly be between 100-200 in the past with four services, but now there is no way to tell if he's between 50-250 if only a single site gave a thorough eval. If two sites have legit evaluations, then the predictive power of the composite is sqrt(4/2) ~ 1.4 times or 40% less informative. Finally, even if all three have legit evals it's still sqrt(4/3) ~ 1.15 or 15% less informative.
It's clear that stargazing is less informative than it used to be. Even a consensus #1 is 15% less certain than it was in the past due to an additional missing independent measurement. It'll be nice to see this in the data in the next few years, but put me on the record predicting starz to correlate with college success and NFL draft stock less than it used to.
It's amazing what people will tell themselves to believe they are right.
Great thread. It's very refreshing to read these level-headed critiques of a severely flawed system.
In addition to the OP's stats-based argument and other valid points, here are a few that have been taking up space in my head:
- Any Michigan commit is going to be a 3-star minimum, because they're at Michigan. And there's only 30 or so 5-stars that are rare finds. So every commit you'll ever see is either 3 or 4 star. Not really much of a distinction. I think of them as "kinda good" or "pretty good".
- The 3/4 star breaking point is completely arbitrary -- .8900 on the 247 composite last year. Why? Absolutely no reason whatsoever. In the 2018 class, there were about 400+ guys ranked as a 4-star with grades ranging in "score" from .9850 down to .8900. Then there are about 2200 guys rated from .8900 down to .7900. Just under 3,000 kids ranked by people of debatable football knowledge with an undefined line randomly drawn in the sand.
- This is not scientific. These guys are not comparable. How did CB Gemon Green from Texas end up with a .8891 3-star grade, but OT Jerome Carvin from Tennessee received a .8904 4-star grade? How do you even draw a comparison, let alone a line, between two guys who play 1,000 miles away in different divisions against completely incomparable competition at different positions on different sides of the ball? Why not flip-flop them? It's only 13/1000s of a percentage point. Gemon Green can't be .8904? Why not? How did they arrive at such a precise difference? (*whispers: because it's a crock of shit*)
- Think about how imprecise the NFL draft is - A full combine, 4 years of detailed video - often against common or at least related compeition, entire staffs of unlimited budget tracking about 2500 guys for 250 draft slots. And they screw it up constantly. They miss on 1st rounders. Football lifers that can't even accurately nail down the top 30 guys coming out of college. -- Now multiply those numbers by about 10. An absolute slew of high school kids shooting for about 3000 FBS scholarships. And I'm supposed to believe Trieu, Wiltfong and Barton Simmons and a couple dozen journalism degrees have the entirety of the high school football playing nation pegged to the ten-thousanth?
I could go on, but I'll close by saying Yes, I'd prefer 5-stars and the high 4-stars because they are usually highly-scouted, top camp performers with impressive high school careers and offer sheets from the top programs in the country. I trust that everybody's on the right track. At the top, #StarsMatter.
At the same time, if Don Brown thinks a no-name, unrated recruit that he's tracked for years and seen play in person can play defense on this team, I cannot understand for the life of me why people here would get bent out of shape because a career journalist over at severely-understaffed 247 doesn't know who he is. It has proven impossible for lifelong football professionals to project a known commodity to the next level with any level of accuracy. Yet, we gnash our teeth over the grade given to a completely unknown player by a career journalist? Don't forget to renew your subscription.
What a racket.
This is a great take Pepto. Ultimately from our perspective it comes down to trust in your coaching staff. Right now, I feel like the fan base is pretty evenly split between having faith in JH and feeling like he is dropping the ball. You are right, stars matter simply because the guys rated as high 4s and 5s have been scouted better, and to the OP's point, have more accurate rankings. That doesn't mean though that some of these guys we are picking up aren't going to pan out to be on that same level (high 4/5). It just means their variance will be higher.
/\ This should end the discussion.
Here's a good rule of thumb:
1. Generic 3 star from a heavily scouted area, probably a 3 star.
2. Generic 3 star from a very lowly scouted area, probably has a lot of traits that the staff found, has high potential to rise once the camp scene gives more scouting. Usually has a
3. 3/4 star in between guy from well scouted area, really depends on fit of the program. Usually is a guy that is a great football player but lacks prototypical size to be a 5 star, or a big lineman without outstanding feet and needs weight room work.
4. Unscouted guys from Florida/Texas/Ohio/Pennsylvania/Georgia: literally just a crapshoot. Could be really good, but trust the coaches.
5. Be excited about any low recruited guy that's a take at this point in the process. Big difference if a lower recruited guy like Joey Velasquez is a take now as opposed to in December. Usually shows they feel very highly about him.
good point. fewer ratings will decrease accuracy, and it also increases the magnitude of the error. if the 1 remaining rating service has a built in bias for or against certain attributes, schools, geographies, height/weight, # of camps, those biases won't be mitigated by the other services. when they get it wrong they are more likely to be really wrong.
but i'm sure harbaugh doesn't care about stars. I found this quote from an article about his recruiting:
Other important factors at every position were also weighed, such as athletic ability, strength, playing speed and character. But when assessing a player, he was always compared to Harbaugh’s created prototype.
“The prototype went all the way down to hand size and shoe size for each position,” said Lamb. “Jim brings as much intensity to recruiting as he does anything else. It’s an everyday thing. His philosophy is that he always wants players who offer upside in potential, which means he looks at height and weight and then analyzes if that player can grow into his body.
although i'm sure he knows perception matters and having the big block M at the top of the 24/7 team rankings can make a difference to future recruits - but winning the conference and winning championships, as well as having a shot at the nfl, makes a bigger difference.
ESPN hasn't fired everyone yet. TomVH is still working there. He's national now I think, which pretty much has made his information so watered down that I have come close to unfollowing him on Twitter (gasp).
Having ESPN in the composite actually seems to do more harm than good, due to how little effort they put into recruiting and how often they update their rankings (basically never)
I disagree it's less informative. For one the advent of Hudl has allowed analysts and scouts to watch tape at the click of a button. The proliferation of 7v7 camps and combines like The Opening an the Rivals 5 star challenge allows recruiting analysts to see this kids in person AGAINST EACH OTHER which is huge. Don't forget these are events are preceeded by regional events and combines. Then you look at the NFL draft data and see that the recruiting sites are actually hitting on guys at a much higher rate than they ever have been. You look at the list of national champions and the last 10 of them have all signed top 5 recruiting classes per rivals within two years of their championship. All the data out there shows the rankings matter and they are for the most part accurate.
Is everything perfect? No of course not. But nothing is. The NFL spends millions of dollars and an incredible amount of time and resources to get their draft picks right and even they don't hit on everyone. At the end of the day saying recruiting rankings don't matter or trying to blast the system is just a defense mechanism for fans of teams who have a bunch of lower rated recruits and want to make themselves feel better.
I think people overrate camps quite a bit. Guys "AGAINST EACH OTHER" seems to happen in very limited and, honestly, somewhat contrived situations. They have some value obviously, but seeing a guy run a couple of drills against some other guy in an event where there are maybe a hundred+ players to watch and multiple events going on means you undoubtedly only get snapshots of guys, plus a couple of measurables. If anything, the combine numbers you pick up are probably the most useful, because at least there you have somewhat-objective numbers to rely on.
Higher-rated guys typically are so because they are good players; nobody is arguing that every 3* kid is a diamond in the rough. But it also isn't a given that if you took the top 25 players in the country every year and put them on a team, they'd win every national title. There is a give-and-taken with how players are recruited to particular teams, and that's, at least to me, the larger argument I've had with people simply point at star ratings as some objective truth of how good two players will be at a particular program.
You make good points but this topic is moving into the realm of religion and politics. Each side has their examples that prove they are right, claim the other sides examples are the exception not the rule, and everyone that has a reasonable take somewhere in the middle gets lost in the shouting of the vocal minority extremists on each side.
This topic wouldn't exist if Michigan was signing a bunch of 4 and 5 star recruits.
Yup. I totally agree.
To an extent, I believe you are correct.
Every time someone asks you about teams that recruit well according to rankings but don't have performances to match, you seem to just put your hands over your ears and shout "OHIO STATE URBAN MEYER ALABAMA NICK SABAN." Every time someone asks you about teams that don't have good recruiting rankings but do have success, you do the same thing.
Want some examples? Here are some teams with recent struggles with top 10 recruiting rankings.
2014: LSU (2), TA&M (5), Tennessee (7)
2015: Tennessee (4), LSU (5), Texas (10)
2016: LSU (2), Ole Miss (5), Texas (7)
2017: LSU (7), Notre Dame (10)
There is a lot of Auburn, USC and Georgia on there too, but I'll leave them out as their success has been more up and down rather than mostly down. FSU was there pretty much every year, so why did just the loss of a QB totally tank them last year? I see a pretty clear bias to teams pulling heavily from "recruiting hotbeds" in that list, IE a lot of kids going to camps and heavily visible to the ranking services. So maybe the point is less that your 4 and 5 star players are overrated and maybe more that your under scouted players are underrated.
I think the star gazer contingent continues to ignore that these teams with top recruiting rankings also have great staffs from top to bottom (except for maybe Zach Smith), incredible facilities, tons of analysts and support staff, etc. There is more to these teams' success than recruiting.
The topic as a whole no, but the stuff I presented in the OP is true. It’s very low level data analysis. I realized it would be fodder for people who are of the belief that recruiting rankings don’t matter. That’s why I pointed out that they do. It’s just that they are less accurate inherently due to the state of that industry.
Uhh, No.... this topic wouldn't exist if we had won three more games last year.
Close but to me more like the arguement between "flat earthers" and the scientific community. One side has mounds and mounds of statistical evidence to back up the claim that ratings matter.
The other side has anecdotal, one-off examples of why rating dont matter.
I also disagree that both sides are entrenched. The dirty secret (hiding in plain sight IMO) is that even those who claim "ratings dont matter" are very quick to point out Michigan is now 2nd in the B1G in recruiting and in the top 10 nationally. Why would that matter if ratings dont matter?
Let Michigan sign a 5 star recruit and then count how many "but ratings dont matter" posts get made.
IMO, the point of "flat earth" is kinda just like recruiting rankings; if you can't trust the ones presenting the evidence, it's smarter to make your own empirical judgment using the things you can trust the most, your own two eyes.
But as Marvin said... "Only believe half of what you see...."
So even than, it's best to believe you don't "know" anything and say "GoBlue"
There are a lot of biases in recruiting rankings, I think that add up to a lot more than anecdotal, one off examples. I just listed 2-3 top recruiting programs per year that haven’t been very good on the field. However, I'm pretty sure I won't get anywhere with you on that one, so let me take a different angle.
How many other data sets have you compared to recruiting rankings to predict success? Let me give you a few lists:
1.) Alabama 2.) Ohio State 3.) Georgia 4.) USC 5.) Michigan 6.) Florida State 7.) LSU 8.) Oklahoma 9.) Auburn 10.) Notre Dame
1.) Georgia 2.) Ohio State 3.) Texas 4.) USC 5.) Penn State 6.) Alabama 7.) Clemson 8.) Miami 9.) Oklahoma 10.) Notre Dame
1.) Texas A&M 2.) Texas 3.) Ohio State 4.) Michigan 5.) Alabama 6.) Florida 7.) LSU 8.) Oklahoma 9.) Tennessee 10.) Penn State
1.) Texas 2.) Ohio State 3.) Alabama 4.) Oklahoma 5.) Michigan 6.) LSU 7.) Auburn 8.) Tennessee 9.) Notre Dame 10.) Florida
And here is what those lists are:
- 2017 247 Composite
- 2018 247 Composite
- Top revenue grossing schools (all sports) 2016
- Top revenue grossing schools (all sports) 2017
Another one that would probably give you a list that mirrors “Top Football Recruiting Schools” list is Football Team total budget and/or total coaching staff salary. I’m sure the response to this will be “See, this proves football recruiting increases gross AD revenue” or “revenue is tied to recruiting rankings” at which point I will give up because I don’t have time to have a 50 post discussion about causation vs correlation. My point is that there are other data sets that correlate well with football success, not just recruiting rankings, and I rarely see these brought into the discussion.
I don’t see it as two sides. Statistics show that the star ratings will prove to be an effective data point on success. Dantonio has also proven and I think Harbaugh is proving that if you are a better evaluator of potential it will pay dividends.
Its like politics. Most people are in the middle somewhere and agree but the screaming done from the polar opposites is polluting the discussion.
Don’t care what stars say or what MGoBlog comes up with or ESPN and Stephen A. Smith. The bottom line is, OSU is beating the crap out of UM because OSU has better football players. You can spin it all you want on this blog but Alabama and OSU are winning and it’s not because they recruit a ton of under the radar athletes.
OSU is beating Michigan because they had a superior coach for a decade. Funny how their lone loss to Michigan was the one year of transition between Tressel and Meyer, isn't it? Did Luke Fickell not have the better players that year? Tressel had the better players the year before, and Meyer had the better players every year since, but that one year all of the good players left? Is that what happened? Were they all hurt?
Michigan got blasted in 2015, but then played OSU dead-even the last two seasons if not for being on the wrong end of a spot and a back-up QB. What was the difference between 2015 & 2016-17? Can you put your finger on it? How did we go from woefully overmatched to dead-even in 12 months? Did we get 85 new scholarship players for 2016? Or did we simply replace an overmatched Defensive Coordinator with Don Brown?
If it's all about the players, can you explain how fortunes can turn so fast without significant roster turnover?
EZ answer, Urbz had no transition dip when he took over. (sure helped that they leaked that he was next year's coach in October, really helped smooth that all out)
For context, OSU's "recruiting dip" between Tressel and Urbz was the #18 class that Fickell put together, very similar to UofMs '18 class in size and rating.
The next year Urbz had put together a top6 class IN A MONTH.... (wink*wink*)
Harbz had the #20 & #37 transition to get to #8.
I don't see the relation to my post, no offense and no big deal.
Regardless, your years are off. OSU's "bad" recruiting class was Tressel's second-to-last in 2010. Luke Fickell was only on the job for 6 months in 2011 (Tressel's May resignation until Meyer's November hire). Urban wrapped up that 2012 class at #5, so Fickell really didn't have any negative recruiting/roster impact.
Yeah, I guess I was answering why Harbaugh wasn't able to sustain the winning in '17, which I attribute more to the '14-'15 transition than anything, but I guess that wasn't what was asked.
And I knew Urbz had an EZ transition but that's just ridiculous Fickell didn't even have one year at the helm during signing day.
Fickell's picture has to be in the dictionary next to the word "heel".
We have enough talent to win it all.
A few things are undeniable:
1) Skill evaluation isn't "simple" or confined to a really good algorithm, not every person or equation is good at it, and some people should probably get better at it, or maybe be the ones listening to others evals.
2) It's woefully and tragically incomplete, subjective and completely handicapped by it's engine being tied to "pay for info" subscriptions.
3) This is the exact thing the NCAA needs to relax it's clenched sphincter for and spread a few $$ around to give high school kids camps that have consistent skill evaluation procedures; plus, ......I don't know........, maybe....
state of the art physical evaluations
review and instruction on how to play the game safely
and maybe some t shirts that at least say NCAA on them, they probably got millions of those laying around.
While I generally agree that having less sites evaluating players will create more variance in the quality of the ranking lists, I don't think the 5* players will be affected in the same way as 4* and 3* players.
The 5* are almost always no-brainer genetic freaks that don't need multiple aggregated evaluations to determine where they should be ranked. There might be a borderline guy slipping through the cracks and #1 might rightly be #8, but these guys are usually pretty easy to spot.
Only problem is that they aren't as "easy to spot" as they are "easy to differentiate".
Problem with this tiered rankings system is, logic tells us the 5* rating is not a real objective scale because there are always @30-35 every year.
If it represented an objective scale, rather than a differentiation scale, some years would have 10 5* and some years would have 50.
Extrapolate that out to 3-4* range and some years the scale could be waaaay off.
I think the OP is 100% onto something in regards to the composite. ESPN stripping resources from this area does seem to add more variance into the composite rankings. But, that does not do anything to refute the argument that rankings matter. Rivals and 247 haven't changed much and it seems they may actually be increasing the resources the sink into scouting.