If you've been around here a minute you've probably seen many of these on/off charts from Hoop Lens:

On one level these are relatively straightforward: what happens when a guy is on the court versus what happens when he's off the court. But there are a lot of moving parts with these things. More than once I've headed over to Hoop Lens seeking to confirm a hypothesis and gotten confounded. Also sometimes the toplines can mislead.

This post is an attempt to clarify what is and isn't important in these things, and how we try to use them.

Rubric 1: Beware massive three point swings

Eli Brooks's on/off is some kind of education in this department.

Big Ten games + Toledo

I think we can safely assert that if Michigan played a billion possessions with Brooks on the floor and a billion with him off the floor, opponents would not be shooting 14 points(!) better from deep when Brooks is playing. Similarly, losing a 37% three-point shooter—and mostly replacing him with Chaundee Brown, a 40% three-point shooter—shouldn't result in a 14 point(!) drop in 3FG%.

Three point shooting is swingy for whole teams over the course of a whole season, and 3P% defense is even swingier. Via Kenpom:

Pomeroy says "there’s just very little predictive value in a team’s early 3-point defense numbers," and when we look at subsets like Brooks's above we're looking at about 4 games of data—not even the 9 in the graph above.

That three point shooting anomaly makes the topline look like Michigan's defense is basically the same whether Brooks is on the floor or not. But if you look at literally any drill-down other than three-point shooting you'll see it's advantage Brooks. Michigan forces almost 50% more TOs, allows fewer offensive rebounds, puts guys on the line far less often, and has slightly better two-point D. They also force more of those very bad twos when Brooks is on the court. We're looking at a guy who should be on the All-Defense team who happens to be on the receiving end of some bad luck from behind the line.

[After THE JUMP: more rubrics!]

That's not to say that there's never any data in those lines. I just want to see that 3FGA/FGA rate move in tandem. I think Austin Davis's on/offs do demonstrate a weakness in his game:

all games because slashing out cupcakes takes out too many Davis possessions

Opponents shoot better from 3 when he's on the court, and they get a ton more threes. Those two lines together are Donta Scott and Micah Potter getting clean pick and pop looks.

I do think that there may be some reality to the Michigan 3P% drop when Brooks isn't on the court because he's a senior who knows what he's doing on the court and helps the offense flow into good shots even if he's not being credited with assists as he does so. To confirm or disconfirm you'd have to actually chart a bunch of shots and see their relative quality. The data here isn't dense enough to make a call.

Rubric 2: Beware Kenpom time effects

This one doesn't come up a ton because usually you're not blasting the whole conference into smithereens but there's a notable effect for some players this year because some of their time is spent with the motley crew of walk-ons and deep bench players who get crushed in Kenpom time. This is mostly Terrance Williams, who looks like a disaster factory if you just take a simple on/off:

Drop out possessions he shares with Adrien Nunez—who has only gotten on the floor as part of the Kenpom kids and IIRC has not missed any of those possession—and things look less alarming:

(This is a spot where the 3FG% on offense is somewhat real since Williams is 0/9. He is part of that denominator.)

This also tends to make the starters all look like the straw that stirs the drink. 38 of the 244 possessions without Brooks have been the Kenpom kids, which doesn't sound like a ton but has a big impact on the bottom line because that lineup is giving up 1.18 PPP and allowing 61.1 shooting from two. Drop those out and Michigan's non-Brooks D PPP goes from 0.9 to 0.85.

All the starters have a 3-5 point swing in two point D in their favor, and ~all of that is because the "off court" section is getting distorted by Kenpom time.

Rubric 3: Two-point shooting is the most reliable thing available

Two-point shots are a plurality of things that occur in a basketball game. They have the most sample size of any of the drill-down stats except maybe turnover rate. Also, two point shooting conflates two different things: 1) where those shots are coming from and 2) whether they go down. A guy who contests well at the rim will see his two point D on/offs prosper. A guy who doesn't get back cut and stays in front of his guy and forces midrange twos will also prosper.

One of the items I'm most confident asserting from these splits is judging a center's defense. Here's Giorgi Bezhanishvili versus Kofi Cockburn:

A seven-point gap in two-point D is big, and both sides of this equation have a lot of possessions. Also when I flip it—Giorgi on, Cockburn off—those numbers stay the same. (Sometimes you get confounding variables like Kenpom time or both guys playing together that mean the relationship is not symmetrical.) You can usually get a clean split between two centers and that data is likely to mean something.

Rubric 4: TO rate gaps are also notable

Brooks's numbers above offer some insight into why Juwan Howard loves him and Michigan's gotten clunked the last two times Brooks has missed a game (Minnesota this year, Wisconsin last year). Despite being the lowest-usage guy on the team, Eli Brooks's primary skill—knowing where to be and what to do at all times—really pops off the chart. Otherwise a metronomic ability to get into sets and limit clean looks for the opponent is almost unnoticeable if you're not a coach.

You can see similar effects across the league. Sasha Stefanovic is a low-usage Just A Shooter but since he's getting replaced by freshmen whenever he's not on the court the banana peels multiply:

There is also a stark effect at Iowa. Ace caught this in his latest Big Ten Reset: CJ Fredrick is kind of important for them.

Fredrick returned after missing most of the last five games and Iowa had 10 turnovers in a 10-point win over Rutgers in their best performance since he got hurt. 

No matter the usage, a guy who holds down his teams TOs is a tentpole for that team, and losing them is going to hurt.

Miscellaneous Rubrics

Bullets:

• OREB differences are more important than DREB differences but OREBs are so individualistic that I don't think these charts tell you much more than a plain old OREB rate.
• FTA/FGA is largely in the same boat. Last year Michigan's FTA rate went up 10 points when Zavier Simpson, a very hackable PG who created a ton of shots for his teammates, was on the bench. There are maybe some contexts in which I'd single it out as an interesting piece but I don't know that I've run across one yet.
• FT% is mostly useful to highlight the fact that things are out of your control sometimes.
• I try to check different combinations of players on and off sometimes, as it can be hard to tell who's driving a particular bus. There is no particular player driving Michigan's defense, except maybe Brooks. The huge 2P% D gap Mike Smith has is evidence enough that you have to be careful with these things.
• 3FGA/FGA is an important indicator. Fredrick above has a ten point gap, which is partly about Fredrick and partly about the fact that Iowa's backup guards are not shooters.