Rating the Raters: Reviewing Recruiting Services for Success and Bias

Matthew Stafford (Rivals #6 overall, 2006) and Mitch Mustain (Rivals #10 overall, 2006)

With the ESPN150 hot off of the presses yesterday, all four major sites now have an updated Top 150/247/250/300 list available for the 2013 class. I wanted to dive in and look at how each service has performed over the years. Would any site stand out as doing a better job of predicting success? Do allegations of regional bias play out for any specific services?

Rivals

Typically considered the gold standard (unless another site has the player you like rated higher), Rivals online archives are available back to 2002. They have produced a ranked Top 250 since 2008 (Terrelle Pryor was the original #1 recruit) and an unranked Top 250 list in 2006-2007. Since 2002 they have classified between 25 and 33 players as five stars and typically have about 300 four-stars per class.

Scout

Like Rivals, online archives go back to 2002. They're a bit more generous on the five-stars, extending the honor to exactly 50 high school seniors per class since 2008. The larger group of five-stars is offset by smaller group of four-stars that round out the rest of the Scout 300 ranking that has been available since 2005.

ESPN

The Worldwide Leader joined the recruiting party in 2006. No one is stingier with the fifth star than ESPN, offering up between 11 and 18 each season since they went crazy with 42 in their first year. The ESPN four-star threshold is also a bit tougher. Last year the number peaked at 238 but prior to that the total was closer to 200 per class.

247

The newest service is 247 Sports, which did a barebones review of the 2010 class before jumping in head first for the last two completed classes. Their best-of list ranks the top 247 players (just like their name, get it?) and is in line with Rivals in terms of number of five- and four-star rated players. Their later entry into the group has allowed them to provide what is, in my opinion, the best website in terms of navigation and ease of use. For the most part they are excluded from these evaluations since the first class they fully rated were only freshman last season.

Methodology

This is where it gets tricky. Do you evaluate on hits or misses or both? Based on available, accessible information I decided that hits would be easier to quantify and really what you want to know about a service. Who does a better job of predicting future stars? By stars I defined them as players who earned all-conference or AP All-American status for a BCS conference school. First team all-conference honors were weighted double and AP All-Americans were weighted triple. If a player earned awards for multiple years, their value was weighted for each season depending on their level.

Each recruit was given an overall ranking for each service in each season. If there was a formal ranking, I used my method to complete the rankings behind them. I used star values and position rankings to approximate a ranking. All five-stars were ranked first, then four-stars and finally three-stars. Each player was ranked in position order and the positions were allocated based on total quantity in each group. This way the #4 fullback wasn’t rated the same way as the #4 wide receiver. Kickers and punters were excluded.

The square root of the rank was then used to further accentuate the differences at the top end of the rankings. Ratings were capped at 1000 and any unranked player was given that value. ESPN was evaluated solely based on the 2006 and later classes.

Who Rates the Best (at Rating)

Overall, Rivals gave the highest average rating to a future star. The weighted average ranking of a player to earn post-season honors from Rivals was #268. Scout wasn’t far behind at 281 and ESPN lagged further back at 329. Here is how each service did by recruiting class (lower is better):

Rivals dominated from 2002-2006 before Scout picked up a couple seasons in 2007 and 2008. With plenty of eligibility left for the 2009 class it’s still anyone’s game. Rivals has jumped out to an early lead for the 2010 class and the 2011 class is 60% comprised of Sammy Watkins and generally pointless at this point in its lifecycle. ESPN has failed to come close for any completed classes, although the 2009 class to date has been neck-and-neck between all three services with probably two-thirds of the results still outstanding.

Offensive Ratings-Weighted Average National Rank of Post-Season Honorees

ESPN finally picks up a win in the tightly contested quarterback evaluations. As you can see by the lower numbers, picking future all-conference quarterbacks has proved to be one of the easier tasks among rating services. ESPN’s average rank of 135 puts them ahead of both Rivals and Scout.

Scout does the best at wide receiver with Rivals a bit back. ESPN is not very close to the leaders at any offensive positions other than quarterback. Their results for both offensive linemen and running backs are particularly lacking.

Defensive Ratings-Weighted Average National Rank of Post-Season Honorees

It’s a clean sweep for Rivals on the defensive side of the ball. Scout is never far from, but still consistently behind, Rivals. ESPN is a distant third across all position groups, at least 20% higher than Rivals in every category and nearly 25% overall.

Conference Ratings-Weighted Average National Rank of Post-Season Honorees

The criticism of ESPN having an SEC bias and west coast neglect certainly shows up in the evaluations. SEC is the conference ESPN clearly wins and the ACC is a narrow win. All the other conferences are just carnage for ESPN while Rivals again takes a majority of wins. Scout is virtually tied for the Big Ten, Big XII and Pac-12 while lapping the field for the always crucial Big East rankings. It is difficult to tell whether the ESPN success is due to better rankings on players ultimately landing at ACC and SEC schools or if they are just giving a flat lift to those conferences. The fact that 1 in 3 players on the 2013 Top 150 are from Florida or Georgia probably indicates that at least some of the success comes from allocating preferential spots to players from the SEC footprint.

Regional Ratings

Here is how each service allocated their ranking slots to geographies. Higher rankings are weighted heavier and each player’s home state was allocated to one of the five major conferences (sorry Big East) based on their geography.

Players from ACC territory were the most consistently allocated across all four services. ESPN and 247 allocate fewer prime slots to the Big Ten versus Rivals and Scout. ESPN is a major outlier out west as the Pac-12 footprint garners much lower rankings there than at any of the other three. The SEC evaluations pick up about 40% from 247 and ESPN versus 35% and below from Rivals and Scout. This gap has narrowed some in more recent years, but there is still a strong bias from 247 and ESPN towards players from the SEC footprint.

Takeaways

In terms of ability to predict future success, Rivals stands out as the clear winner among all of the services. Scout is not significantly behind and has closed the gap in recent seasons. Rivals predictions proved more accurate at five of the seven position groups and overall for both sides of the ball. ESPN is a distant third in almost every sub-category with the exception being quarterback where they lead the most closely contested position group.

The services appear to be mirrored in their regional biases. ESPN and 247 slant to the Southeast and Mid-Atlantic regions with a clear sub valuation of the West Coast. Rivals and Scout don’t have dramatic swings to any one region but do give less value to the Southeast with the extra spread across the rest of the country.

While Michigan’s 11 [players in the...] Top 150 showing is absolutely a good thing, ESPN has proven to be the least reliable of the three established services. Historically, 30.5% of the weighted post-season honorees originally appeared in the ESPN 150 while Scout and Rivals have each had at least 34% in their Top 150’s. The differences aren’t massive and all sites have had their misses, but overall there is clear evidence that Rivals is the most consistently correct and that Scout is a strong #2. Although the individual players fluctuate, the overall ratings for Michigan’s class to date are essentially identical between Rivals, Scout and ESPN with 247 being lower on it than any other service.

Bonus: Protecting Conference Turf

Not looking at recruiting services but conferences now, I wanted to see which conferences did the best job of keeping the best players from their region in-house. Each state was split between conferences based on number of schools in a given state. States without BCS conference teams were excluded. This isn’t perfect because there is no way Cincinnati and the Big East have the same share of Ohio as the Big Ten and its school in Ohio. But it makes each school among the Big Six theoretically even and provides a good starting point.

*FP=Footprint

The Pts are an estimate of the total value of the recruits within a footprint. The total includes points for all players to any conference with a school in that region. The split is an allocation based on number of BCS conference schools in that state in each conference. Not surprisingly, the conferences with the smallest geographic competition, the Big XII and the Pac-12 signed the highest percentages of their available recruits. After splitting up the states, the SEC actually signs more than their allocation of the footprint. The Big Ten is close behind but work from a much smaller pool. If the SEC is able to make make gains in Texas (they currently have a 12% “share”) with the addition of A&M, the talent gap between the SEC and the rest of the conferences could widen.