the just released schedules were a flat-out statement that the B10 doesn't believe SOS will matter in playoff selection
Analyzing the Validity of the Hennechart
Filed under:
After reading Coach Schiano’s excellent diary on visualizing The Hennechart, I started to wonder about the validity of the HenneChart (damn you experimental psychology grad program!). Basically, does the HenneChart measure what it’s intended to – QB performance. So, I compared the HenneChart with 2 established metrics of performance: Completion percentage and NCAA QB Rating (not to be mistaken with NFL QB Rating).
To do this, I played around with assigning different weights to the Hennechart variables as well as which variables to include. Then, I assigned each variable into a Good category and a Bad category. From there, I took the percentage of points that fall into the Good category [ = Good points / (Good points + Bad points)].
As I mentioned, I manipulated the weights and which variables to include. I tried the following:
- Limited-1pt System: Good = (1 * DO) + (1 * CA). Bad = (1* IN) + (1 * BR)
- All-1pt System: Good = (1 * DO) + (1 * CA) + (1 * MA). Bad = (1* IN) + (1 * BR) + (1*TA) + (1 * PR)
- Limited-multi pt System: Good = (2 * DO) + (1 * CA). Bad = (1* IN) + (2 * BR)
- All-multi pt System: Good = (3 * DO) + (2 * CA) + (1 * MA). Bad = (2* IN) + (2 * BR) + (1*TA) + (1 * PR)
- 0 false 18 pt 18 pt 0 0 false false false /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"Times New Roman"; mso-ascii-font-family:Cambria; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Cambria; mso-hansi-theme-font:minor-latin;} Modified 1pt System: Good = (1 * DO) + (1 * CA). Bad = (1* IN) + (1 * BR) + (1*TA) + (1 * PR)
From there, I ran a series of bivariate correlations. Of course, this is limiting given our low sample size (5 games). I don’t have the time to do previous seasons (maybe this weekend…).
Here is a table comparing these 5 systems of analyzing the HenneChart. Correlation r-values are provided with p-values (statistical significance) in parentheses.
For those that are interested, Completion % and QB Rating is r(3) = .56, p = .33.
As you can see, using the HenneChart with the Modified 1pt System (excluding MA throws) appears to offer the best validity. Now, this is not perfect. There may be yet a better way of assigning weights, deciding which variables to use, etc. Any advice is welcome. So, this would suggest that Coach Schiano's visual illustration of the HenneChart should have Marginal throws as not appearing in Blue or Red; Brian's suggestion of using MA as a zero-point is a good one.
October 9th, 2009 at 6:16 PM | 
Formatting Diaries
Formatting Diaries
October 9th, 2009 at 6:20 PM |
Think that's a bit better.
Think that's a bit better.
Think that's a bit better. You should be able to change everything yourself from there.
October 9th, 2009 at 6:32 PM |
Say What?
Say What?
I'm confused. As far as I know, the reason for the HenneChart is to more accurately reflect the performance of the QB by adjusting the "raw data".
Thus, I would NOT expect that the HenneChart should correlate with the lazy metrics of % Complete or College Passer Rating. If the correlation is high, why bother with the more complex HenneChart?
Life should not be a journey to the grave to arrive safely in a pretty and well preserved body, but rather to skid in broadside in a cloud of smoke, thoroughly used up, totally worn out, and loudly proclaiming "Wow! What A Ride!" HST
October 9th, 2009 at 7:06 PM |
Explanation
Explanation
Well, I wanted to accomplish a couple things:
1) Assess if the HenneChart strongly correlates with other measures of performance (it does)
2) If it needs to be weighted (which my analyses showed it does not)
Also, I've gone through and added one more system (Modified 1pt)
October 9th, 2009 at 9:57 PM |
Interesting. Agree with the
Interesting. Agree with the
Interesting. Agree with the comments that the Henne chart perhaps isn't supposed to correlate with completion percentage and or QB rating.
But, if you want to do this right, go back in time and get all historical Hennecharts for all QBs with, say, a minimum of 15 attempts per game.
Convert each Hennechart metric to a percentage of attempts. Then run a canonical correlation model (http://en.wikipedia.org/wiki/Canonical_correlation) with the Hennechart metrics predicting completion % and QB rating (you may have to run two models or you may be able to run one only). Such a model will give you the optimal weights for each Hennechart metric for predicting completion % and QB rating -- if that's what you want to do.
I'm on hazy memory ground right now, but you'd need to drop one of the Hennechart metrics from the model (probably marginal) so that your predictors/X's aren't a complete linear combination. If you prep the data I think I can run this in Stata or something else -- but I'm rusty.
Of course, your modified 1 point system works reasonably well, and it certainly has simplicity going for it, which would not be a virtue of the canonical correlation model.
Wanting something to be true does not make it true.
October 10th, 2009 at 8:01 AM |
TA's
TA's
I don't think they can be filed as "bad." Many times, the TA is the right read, and, especially in Tate's case, TA's often time are + yardage scrambles.
October 10th, 2009 at 9:53 AM |
I like the idea, but I think
I like the idea, but I think
I like the idea, but I think the methodology is a little funky. Why not run a series of correlations between the raw data and your dependent variables without worrying about deciding what is good or bad? If you have STATA or some similar program, it should be able to tell you how to weight them by producing correlation coefficients for each independent variable (%DO, %MA, etc) that are readily comparable. That would be the real benefit of a study like this--similar to Visualizing the Hennechart, your work would be able to tell us how to read the chart (e.g. by teaching us that a DO is worth 127% of a CA). Just some food for thought.
October 10th, 2009 at 11:41 AM |
I see what you're trying to do...
I see what you're trying to do...
...but it seems flawed to me.
The whole point of the Hennechart is to measure QB performance independent of receivers. This is (for the most part) easily done if you watch the play--a ball that hits a receiver in the chest should be caught, and if it isn't, you don't want to penalize the QB for it in his stats.
The statistical measurements you're looking at correlating with (completion percentage and QB rating) are conflated with receiver's stats too. A drop will show up in the QB's completion percentage, but not in the Hennechart. That's the whole *point* of the Hennechart.
So seeing ~80% correlation to completion is actually almost perfect--you could go further with this and try to show that 20% of the Hennechart score is either a) dropped balls that aren't penalized in the Hennechart, or b) marginal balls that got caught but are still penalized.
60% to QB rating isn't bad, but QB ratings never made that much sense to me (the formula is 'pretty' but weird).
In short, I'm not sure you'll get much better correlation than what you have now, but I'm not sure that what you're testing can tell you if the Hennechart is 'valid' or not. It has an essentially different purpose than the factors you're comparing against. Remember, 'time of posessesion' is a time-tested 'statistic' too :)
October 10th, 2009 at 7:04 PM |
if i was going to use a metric to compare to
if i was going to use a metric to compare to
it would be yards plus interceptions per pass, weighting INTs by average yard value of INT, which is 40-50 yds.
...the Canadians make up for it with their emotion and classic ice-dancing skill.
Also, if anyone knows how to get rid of that annoying code gibberish, that'd be great.