NCAA Passer Formula is NOT Meaningful
Several threads have discussed the evaluation of QBs. Clarence Beeks provided some good stuff in “Terrelle Pryor 2008 Statistics”.
Some have suggested the NCAA Passer Rating Formula is viable to evaluate QBs. Bottom Line – It is NOT meaningful. Here’s why.
The Formula contains these 4 components that are summed together:
1) Percent Complete X 100
2) Yards Per Attempt X 8.4
3) Touchdowns Per Attempt X 330
4) Interceptions Per Attempt X – 200
Percent complete is an important indicator for QBs so it would obviously be included in any analysis. It ends up to be about 1/3 of the total number.
Yards per attempt are also important and the 8.4 multiplier doesn’t bother me (even though I have no clue where it came from). I changed the multiplier to 5.0 and 10.0 and it did have any dramatic impact on the order of QBs. With the 8.4 multiplier, most good QBs have a higher score for YPA than CMP%. This is the first strike against the formula. Do we all agree that YPA is more important than CMP%? I sure don’t. The QB has a lot more control over CMP% than YPA.
The next two (TD & INT) are inexplicable to me. Somehow TDs are deemed 65% more important than INTs. The QB has a lot more control over INTs than TDs. And, INTs are a turnover – the one stat that everyone agrees is the primary indicator of which team wins or loses a game. This is strike two.
Fumbles lost by QB is not included at all. Strike three.
TDs scored rushing by QB is not included. Strike four.
QB rushing yards are not included. Strike five.
Total yards are not included at all. Thus, if 2 QBs have everything else equal but one QB has twice the attempts and twice the total yardage, they are considered equivalent QBs. Strike six.
Final note (and yes, some may consider this anecdotal), here is the list of Michigan’s 10 greatest QBs since 1949 if NCAA Passer Formula is used:
Elvis Grbac 148.1 (6460 Total Yards)
Jim Harbaugh 145.6 (5449 Yds)
John Wangler 145.2 (2994 Yds)
Todd Collins 145.0 (5858 Yds)
Michael Taylor 141.6 (2194 Yds)
Drew Henson 140.8 (2946 Yds)
Tom Brady 136.4 (5351 Yds)
Chad Henne 133.9 (9715 Yds)
Brian Griese 131.4 (4383 Yds)
Rick Leach 130.0 (4284 Yds)
If you think this list is correct (sorry #8 Henne), then the formula works for you.
January 13th, 2009 at 4:39 PM ^
Why would fumbles, rush yards or rushing TD's have anything to do with a passer rating? Maybe this is your point, but a passer rating is supposed to identify one's passing ability, one aspect of playing QB.
January 13th, 2009 at 4:57 PM ^
I agree, but, according to Wiki, the NCAA Passer Rating "is a measure of the performance of quarterbacks ....."
It is almost always shown just for QBs and labeled simply as "Rating" with no other rating formula shown (or that exists as far as I know).
And, I sure think I have seen it in a context such as "Sam Bradford is the highest rated QB in 2007 and 2008." This is misleading at best.
So, even if it were accurate, it only provides one part of the evaluation. Where is the other portion?
January 13th, 2009 at 4:43 PM ^
If you insist on relying on total yards and total completions, you believe the best 2 QB's ever are Tommy Chang and Colt Brennan.
YPA should be more important than completion % because completetion % is included in yards per attempt (to the extent that lower completion percentages drive down yards per attempt).
I would agree that TD's are over-valued.
January 13th, 2009 at 4:47 PM ^
It's Timmy Chang. Or, rather, Timmychang.
January 13th, 2009 at 5:09 PM ^
Wow, Chitownblue, this is like the 3rd time you have absolutely mis-stated my post.
I have never said Total Yards (and never even mentioned Total Completions as an indicator) should be the ONLY indicator.
I said, "if all else is equal, then Total Yards indicates the better QB."
If 2 running backs have the same YPC, and one has twice as many carries and twice as many yards, are they equal? Not to me.
In respect to TTebow, I stated that since he was #38 in Total Yards in 2008 and had 2,000 yards less than Sam Bradford and was not dramatically better in any other stat (#4 in Rating, #23 in CMP%, #4 in YPA) this meant he was not one of the "top 5 college players of all time".
January 13th, 2009 at 5:18 PM ^
Which is using a single season to evaluate what will be three years of starting. It is also omitting the fact that Tebow, in two seasons, has run for 1700 yards more than Bradford.
Further, the point about Chang and Brennan was to illustrate that I don't find raw total passing yards meaningful.
January 13th, 2009 at 5:38 PM ^
Actually if you include 2007 & 2008, TT still has the same problem. Bradford blows him away except in rushing stats.
Let me be clear, TT is a very, very good QB. If his stats are similar next year, he does have a good case to be a great QB.
But, he must compete with: Montana, Elway, Frazier, Flutie, McMahon, Ward, Marino, Testeverde, Staubach, Peyton Manning, Leinart, Young, Bradford, Namath, (just to name a few) just to be a Top 5 QB let alone Top 5 College Player of All Time.
January 13th, 2009 at 5:38 PM ^
Again - omitting rushing stats intentionally removes a large portion of Tebow's value. You can't do it. He has carried the ball over 400 times in 4 years for over 2000 yards and over 40 touchdowns. You can't ignore that.
January 13th, 2009 at 5:45 PM ^
I was not trying to omit TT rushing stats.
Do you believe that considering TT rushing stats, he is better than all the QBs I listed?
When I do include TTs rushing stats (subjectively since there is no formula that I know of), I still do not conclude he is even a Top 5 QB (based on the list of QBs I posted).
This is just my opinion, I am interested in yours.
January 13th, 2009 at 5:48 PM ^
I honestly have never looked at Staubach's or Montana's college stats ever.
Note, I'm not even sure Tebow is one of the best QB's of all time - I haven't even looked into enough other QB's to say that. What I've largely objected to is that the stats you select to quote (total yards) and not quote (rushing) seem to be cherry-picked to cast him in the least appealing light.
January 13th, 2009 at 6:10 PM ^
Stats aren't necessarily the best way to determine who is the best player
January 13th, 2009 at 6:49 PM ^
Statistics measure performance. How would you do it?
January 13th, 2009 at 7:04 PM ^
Personally, I evaluate players based on how often commentators use words such as "grit," "hustle" and "moxie" to describe them.
January 13th, 2009 at 7:11 PM ^
I search for how many coaches are in their family tree.
January 13th, 2009 at 4:52 PM ^
It's not VERY meaningful, and like all other statistics, should be taken with some salt - but to say it's not meaningful at all is selling it short and speaking in absolutes about a very non-absolute discussion. I guess the problem is you're saying it's not meaningful for some really shittastic reasons like QB rushing totals - which have nothing to do with your grading as a passer. Fumbles lost is another ridiculous statistic for reasons discussed ad nauseum re: turnovers. A QB who fumbles the ball 20 times but recovers all of them is considerably more lucky than a QB who fumbles 5 times but loses all of them.
It's obviously not a be-all statistic, but it's better than grading players based on raw numbers.
January 13th, 2009 at 4:53 PM ^
I'd subscribe to this POV.
January 13th, 2009 at 5:45 PM ^
"It's obviously not a be-all statistic, but it's better than grading players based on raw numbers."
Using a formula to manipulate raw numbers still leaves you with raw numbers. Other than that, I agree with your statement. The beautiful thing about football is that it's the closest thing to a true team sport. Even if you have an amazing QB, you could have receivers that are not good at getting open and a young, inexperienced o-line, leading to a bad year for said QB. Any stat in football is largely dependent on someone else, thus comparing stats isn't really all that meaningful.
January 13th, 2009 at 4:52 PM ^
Strike seven: The following three passes have the exact same result in the passing efficiency formula. A 50 yard touchdown pass where the qb hits the WR in stride in the front corner of the endzone, just before the safety can get over. A 15 yard crossing route where the QB steps up in the pocket, throws through a throwing lane and the receiver breaks a tackle for a touchdown. A screen pass where a combination of broken tackles and good blocks leads to a 50 yard touchdown.
Like most football stats, passer efficiency rating is only helpful within context. The context for the passer efficiency rating is that it is much more telling of the overall efficiency of the offense (unless the offense is of the Ga. Tech variety) then it is of the individual skill of the quarterback. Does it help to have a great quarterback? Sure. But it also helps to have a balanced offensive attack with great players at the other positions.
January 13th, 2009 at 4:59 PM ^
Sure, YPA is not completely in the QB's hands. But a QB in an offense that calls for him to routinely throw 15 yards down field will post a higher YPA than Tim Couch, who threw 3-yard passes. Obviously, both will have things providing distortion (ie, broken tackles, etc...). But over the course of 270+ passes a year, the guy who throws longer passes will have a higher number.
You can't find a catch-all statistic. Raw numbers are largely worthless for comparison sake because then you'd decide that the best QB's ever played only for June Jones or Mike Leach.
Completion % is useful, but can be distorted by offenses in which the QB is throwing exceedingly easy passes (like Tim Couch).
YPA is useful, but can be distorted if you have a machine like Benn from Illinois (though, I still think my point above stands).
Rushing yards need to be taken into account.
Attempts per INT is useful, as it shows how often they throw a pick.
Attempts per TD is usefull, as throwing a pass 12 yards downfield from the 10 yard line IS harder than completing a 12 yard pass from the 50.
Passer efficiency compiles most of the above factors. You can disagree with the multipliers applied to each, I understand.
January 13th, 2009 at 5:21 PM ^
I agree with this. I don't think total yards is terribly important. If one team has a solid run game, they are going to throw the ball fewer times than a team with no run game. If one QB is only called on to throw 20 times a game instead of 40, his ability as a QB should not be affected negatively.
January 13th, 2009 at 6:09 PM ^
"Sure, YPA is not completely in the QB's hands. But a QB in an offense that calls for him to routinely throw 15 yards down field will post a higher YPA than Tim Couch, who threw 3-yard passes."
Really? If you're throwing 15 yards down field, your completion percentage, if you're a good QB, is likely to be in the 55% range, meaning on nearly half your throws, your YPA is going to be 0. The quarterback who can throw those 3 yard passes, as you point out, is likely to complete a high percentage of those passes (70-80%). If your offense is effective, the QB who is throwing those exceedingly easy 3 yard passes is still going to see them turned into 15+ yard passes on a regular basis and is going to see less of those 0 YPA incompletions. Thus, the YPA can be equal, irregardless of the average difficulty of the QB's pass attempts over the course of the season
Pretty much every passing statistic is somewhat relevant, but the offensive scheme and non-QB personnel can greatly skew each and every statistic. If you don't put your QB in a position to make the hard throws on a regular basis, he's going to have a lower INT%, if you have great YAC skill players you'll have higher YPA and TD%, if your WRs are not great at creating separation, your QB will have more throwaways and thus a lower Comp% and YPA. If your line stinks you're pretty much going to be lower on everything.
Finally, the combination of semi relevant passing stats into another passing stat does not make the subsequent passing stat any more relevant than the underlying data.
January 13th, 2009 at 6:20 PM ^
I don't know what "irregardless" means.
January 13th, 2009 at 6:33 PM ^
yeah, yeah, i screwed up... Next time I'll be sure to carefully edit my post to meet your internet standards.
January 13th, 2009 at 6:40 PM ^
Just playing around man. No worries.
I have pretty low internet standards - real words only.
January 13th, 2009 at 11:42 PM ^
orly?
January 13th, 2009 at 6:55 PM ^
I'm not going to disagree that having good players helps a QB be good.
RE: YPA. The poster-boy of high-completion/low yard per completion is Tim Couch. He had a 69.8% career completion percentage, and averaged 7.3 yards per attempt - or less than Terrelle Pryor, who completed 10% less of his attempts.
(note: this isn't an endorsement of Pryor, merely some evidence to support my point that a QB who completes fewer passes but throws, on par, deeper passes, can balance out in ypa)
January 13th, 2009 at 5:18 PM ^
In addition to the objections above, I'm not sure I agree with the idea that a QB has "a lot" more control over INTs than he does over TDs. Do you have any data to support that or is it just an assumption?
If it's the latter, I don't really agree. I could see a QB having more control over INTs than TDs in certain circumstances, but I don't know if I'd characterize it as "a lot" more control.
January 13th, 2009 at 5:29 PM ^
Data could exist but I don't have it.
Basically, I believe just watching football games indicates that the vast majority of INTs are the QBs fault. Tipped balls, would be the exception. In the recent NFL playoffs, the INTs were primarily the fault (i.e. control) of QB.
As stated by others above, if the QB throws a 2 yard pass (to Westerbrook?) and the receiver runs 70 yards and scores a TD, why does the QB get credit?
My impression (without hard data) is that a lot more INTs can be directly attributed to QB than TDs.
January 13th, 2009 at 5:34 PM ^
There are 2 reasons QB's get credit for the touchdowns they throw, even when the ball is caught outside of the endzone. The first is that because the QB threw the ball, he made a read as to who he should throw the ball to. Assuming more than one person went out for a pass, decided who the best option was. That decision resulted in a touchdown.
The second reason is that how well a ball is thrown can determine whether or not the reception is a touchdown. If a ball is under thrown or behind the receiver, he still may catch it but be tackles shortly thereafter. A perfect throw will put the receiver in better position to continue on and score. So even if the ball is only in the air for 20 yards of a 60 yard TD pass, the QB has a lot to do with whether or not it goes in for 6.
The only exception to this, that I can think of, is when the receiver breaks tackles. Not perfect, but certainly meaningful.
January 13th, 2009 at 6:51 PM ^
How well a ball is thrown can also be the difference between a slant going for 8 yards or 40.
January 13th, 2009 at 6:47 PM ^
So we're saying that any time it looks like the QB is at fault for an INT, he is, despite:
a. We do not know the play call
and
b. Therefore we do not know, unless someone makes a very very demonstrative gesture on the field to indicate so, if the intended receiver actually ran the correct route.
Am I right?
January 13th, 2009 at 6:56 PM ^
Also this.
January 13th, 2009 at 6:53 PM ^
I don't agree with your impression. I might agree that a higher percentage of INTs should be attributed solely to the QB than to other factors, but I don't agree that it would be accurate to term that higher percentage as "a lot". I think the percentage of INTs that are tipped balls or otherwise not the QBs fault is probably pretty similar to the percentage of TDs where the QB played little part in the eventual score (like your anecdotal Westbrook example.)
Moreover, while turnovers are certainly one of the major factors that tends to influence who wins a game, points on the scoreboard are still THE factor that decides the game. So basically I'm not buying this point at all.
January 13th, 2009 at 7:18 PM ^
is a terrible stat to evaluate a QB, but it is better than all of the rest.
Career passing yardage leaders at Michigan:
1. Chad Henne
2. John Navarre
John Navarre's absence on your list of the top ten Michigan passers based on the NCAA formula speaks volumes.
I don't think there is not a person on this site that would argue that John Navarre was a top 10 QB in history at Michigan.
Judgment must be used in evaluating these QBs. How good were their WRs and OL; how much did their rushing (granted, the NCAA is a "passer formula") add to the offense; what era did they play and the rules changes and offensive philosophy play a part in their numbers. Level of competition, leadership and wins have to be considered.
In the top-ten list, Harbaugh, Taylor and particularly Leach's rushing was exceptionally valuable to the team. Sacks are not incorporated, and Navarre, Collins and Grbac were negative yardage rushers due to their frequent sacks.
Enjoy Life, I have some issues with the Efficiency rating, and would switch around the names of Michigan's top ten listed above. But the list is pretty damn close to the 10 best in Michigan history in order. [My biggest exception is I have always thought Steve Smith was a top 10 QB due to his combination of rushing and passing, but that is my prejudice.]
Provide any other statistical formula that is gives a better top ten (hint: it better not include John Navarre).
January 13th, 2009 at 9:50 PM ^
Finally talking about the list presented above.
Navarre does not crack the above list, imo, and now that you bring up Smith does not even make the top 11.
January 13th, 2009 at 9:57 PM ^
I would be open to any argument about Elvis being the best QB in Michigan history.
I think a lot of folks who only have watched this decade dont know just how great he was.
In 3 + years of starting, Elvis lost 5 starts. Henne lost 14 starts.
The Elvis-led teams were the first truly dangerous, downfield teams Michigan had and often needed a special defensive performance just to stop them.
January 13th, 2009 at 10:15 PM ^
Yeah - the Elvis/Desmond/Hayes/Alexander combo was crazy. He's definitely a QB that balanced out a lower completion % with longer completions. I'd buy that he was the best passer in Michigan history.
January 13th, 2009 at 11:51 PM ^
range from good to bad, but I'm not sure why any of them would lead you to conclude that the passer rating is not meaningful.
What I do want to know is where those weights came from and why. They didn't just leap out from the ether, someone decided they were a good idea for some reason. If someone did some math, I'd like to see it.
Also: if some guy went through the trouble of determining the weights, he must have been thinking about wins, points and yards. So why isn't passer rating denominated in wins, points or yards? There could definitely be a wOBA and wRAA for quarterbacks.
January 14th, 2009 at 1:00 AM ^
"The pass-efficiency formula is computed to make 100.0 the average
quarterback or team rating by using the multipliers 8.4, 3.3 and 2.0 with factors two, three and four, respectively."
See pages 14 and 15:
http://web1.ncaa.org/web_files/stats/Stats_Manuals/Football/2008%20Foot…
Hope that helps explain.
January 14th, 2009 at 12:12 PM ^
Well, there are probably 100's of weighting factors that would accomplish the same result (average =100).
I love that it is "carried out to 2 decimal places" and published. No way does the data support this. It should be rounded to probably the nearest 10 if you want to reflect the variability of the data.
I too would like to know where the weights came from (the link does not provide this -- but it obviously does exist).
BTW, the entire point of my original post is that this formula is obviously flawed. Who decided to use this specific one?
Why?
If you are going to the trouble to develop a formula, why not include as many pertinent data factors as possible?
With these new fangled computer thingies, it is easy to crunch data.
Oh, and I did forget a big one (Won-Loss record) and another one (Championships).
January 14th, 2009 at 12:20 PM ^
Your original post did not assert that passer rating is flawed, it asserted that passer rating is meaningless. Big difference.
Using GPA and a standardized test score when determining a high school students ability to be successful in college is flawed as well. It's not perfect. But it's the best metric to use right now, so colleges use it. I don't think that makes them meaningless, though.
January 14th, 2009 at 1:02 PM ^
We just have a different interpretation of "meaningful".
The dictionary is not much help but one definition is "significant quality" (Webster on-line).
The passer rating is being used as a measure of QB performance. Shouldn't it provide "significant quality" that the results provided are accurate??
Oh, and in case you haven't noticed, I (and many others) do tend to use hyperbole quite often.
I could have used the title "Not as good as it could be" but that just doesn't have the same pizazz!
January 14th, 2009 at 1:13 PM ^
As a big fan of pizazz, I understand. In fact, it would be cool if we could change the color of our font and add emoticons. Brian?
January 14th, 2009 at 1:30 PM ^
OMG, don't even think it. Some folks get upset if anyone uses CAPS too much!
January 14th, 2009 at 1:34 PM ^
NOT ME!!!
January 14th, 2009 at 1:25 PM ^
I always wondered why the NCAA rating was different from the NFL rating. IIRC the NFL system was designed to make a 100 rating about the best one could expect over the course of a season.
January 14th, 2009 at 1:38 PM ^
Well, it seems a bit more complicated.
http://en.wikipedia.org/wiki/Passer_rating
The explanation for the NFL formula is "Passer rating is determined by four statistical components, each of which is computed as a number between zero and 2.375. The benchmarks for these statistics are based on historical averages. If any of the raw components are beyond the limits of zero or 2.375, the component is set to limiting value of zero or 2.375 as the case may be."
I readily admit I have no idea even what this means!!
No such explanation exists for the NCAA formula.
And the NFL formula is far different, even to the point of bizarre.
For just %Complete: [(COMP/ATT) X 100 - 30]/20