Tater

August 12th, 2015 at 12:38 PM ^

Michigan was terribly predictable under Brandon.  Oops, I mean Hoke.  That is why the team looked so much worse than it really was last year.  And that is why they are going to surprise the idiots in the media who are predicting 6-6 this year.  

This year, opposing defenses aren't going to know where the ball and the skill personnel are going before the ball is snapped.  It is going to make a big difference in the outcomes.

Zone Left

August 12th, 2015 at 11:38 AM ^

Intuitively, I'd say college teams are more predictable, because they're typically good at less than NFL teams. Most college teams can't pass and run well--they're lucky if they can do one of the two.

However, some offensive coordinators in college are doing some really cool stuff with run-pass options. I think the real trick would be to figure out somehow which player the quarterback for those teams is going to read on a given play. That would be a coup.

WolvinLA2

August 12th, 2015 at 11:39 AM ^

This is not surprising to me at all.  But I also don't disagree with it.  In the pros, the offense has such an advantage over the defense that you should always go with the highest percetage play.  There's no reason to get fancy, unless you're significantly overmatched.  Since most teams aren't, you go with what works.   

ebv

August 12th, 2015 at 11:43 AM ^

From the article:

2014 Dallas Cowboys at Jacksonville Jaguars

  • Total # of Plays: 119
  • Total # of Plays Correctly Predicted: 109
  • Total # of Plays Incorrectly Predicted: 10
  • Percent of Plays Correctly Predicted: 91.6%

Is 91% really high accuracy? Sounds like a recipe to get gashed by play action to me.

LBSS

August 12th, 2015 at 12:23 PM ^

Yeah and overall accuracy was something like 75%. I wonder if how much better that is than a fan guessing at home based purely on down, distance, and personnel, let alone an NFL defensive coordinator or a middle linebacker in the midst of the game. 

Now that I think about it, probably a lot better than the casual fan. But than the pros? 

steve sharik

August 12th, 2015 at 1:51 PM ^

is call the defense that is best suited to stop the plays, but the call is still sound against all offense, so you would be protected against big plays (in theory). If you guess wrong, the result should be about a 5 yard gain (assuming everyone carries out his assignment, uses good technique, and makes the play when presented).

99% of big plays happen b/c the defense missed an assignment or didn't make the play (e.g., missed tackle, failure to deflect pass, etc.).

ReegsShannon

August 12th, 2015 at 11:46 AM ^

To be fair, it doesn't mean anything until you test it on new data. This kind of stuff annoys me, the model was designed to spit out good answers for data between 2011-2014. It's the same thing with draft models that predict player success AFTER THE FACT. You have to give it new data to truly test it.

NittanyFan

August 12th, 2015 at 12:05 PM ^

I knew even before opening the link that this model would be some sort of gradient boosting/random forest model.  

And yep, I was correct.

Those types of models are VERY good at fitting data in an after-the-fact matter.  But they're only valuable if they can predict.  And they're typically more questionable on the predictive front.

trustBlue

August 12th, 2015 at 7:36 PM ^

Came here to say this.  Their "test" set was simply a subset of the same data that they had fit their model to.  That's a huge no-no.  It's meaningless until its forward tested against new data that wasn't used to train their model.  

If its average accuracy is only 75% now, its will likely drop appreciately once its starts trying to predict new data.  If its drops to say 60-65%, you're basically doing only slightly better than a coin flip.

It's still sounds an interesting and fun project, but we're not exactly talking Sabermetrics here.

Trebor

August 12th, 2015 at 11:50 AM ^

75% accurate over the 20 random games they selected from the same timeframe that they built the model on? Doesn't sound that 'highly accurate' to me, especially when you could pick 'pass' every time and be right almost 57% of the time.

LJ

August 12th, 2015 at 11:56 AM ^

All they're doing is predicting run/pass.  75% does not strike me as phenomenal -- I wouldn't be surprised if NFL coaches can predict at least that successfully after prepping for a team's tendancies.  I bet most serious fans would be around 70% -- you're probably at 55% just predicting run on every down, and I would think at least 10-15% of plays are very easily predictable given down & distance, 4th quarter leads, etc.

Space Coyote

August 12th, 2015 at 12:08 PM ^

Over 20 games, they predicted run/pass (note: not the type of run or type of pass) 75% of the time. I actually think defensive coordinators could do better with the information they take into game day (which is essentially the stats that this model uses, such as down and distance tendency for specific quarters/situations). Especially as you start getting into more predictable situations, such as the 4th quarter, your accuracy just goes up.

Yeah, in a vacuum, predicting run/pass 91% of the time seems nice. But that's not nice enough to sell out one or the other even. When you realize the real percentage over 20 games is only 75% of the team (meaning it's wrong a quarter of the time simply guessing run/pass), I actually thought this would do better.

I honestly think you could set up a neural network that would do significantly better, and closer to ~90%, but that only gains you a slight advantage, it doesn't really equate to predictability.

Bocheezu

August 12th, 2015 at 12:42 PM ^

Dallas was up 24-7 at the half and up 31-7 at the end of the 3rd quarter.  Not hard to figure out who's going to be passing and who's going to be running.  Would like to see the prediction accuracy vs. quarter.  Like 1st quarter is 60% and then 3rd and 4th quarter are 100% to boost the overall up to 91.6%.

The 2013 Ravens/Broncos game was close at the half (17-14) and then the Broncos blew up in the 3rd to take 42-17 lead early in the 4th.  So a very predictable 4th quarter there.

2011 Giants at Patriots.  Actually a really close game.  0-0 at the half, ending at 24-20 Giants.  The only one I would tout as an actual accomplishment for the model.

Bb011

August 12th, 2015 at 1:40 PM ^

I would argue a certain amount of predictability is good though. It is the high predictability that allows for big plays. When you are predicted to run because of the situation and the trend of how you have been calling the game in the past is when you can hit them for a huge playaction or something similar that they aren't expecting.