Friday, October 16, 2015

Threats Perceived When There Are None

Sendhil Mullainathan is one of the most thoughtful people in the economics profession, but he has a recent piece in the New York Times with which I really must take issue.

Citing data on the racial breakdown of arrests and deaths at the hands of law enforcement officers, he argues that "eliminating the biases of all police officers would do little to materially reduce the total number of African-American killings." Here's his reasoning:
According to the F.B.I.’s Supplementary Homicide Report, 31.8 percent of people shot by the police were African-American, a proportion more than two and a half times the 13.2 percent of African-Americans in the general population... But this data does not prove that biased police officers are more likely to shoot blacks in any given encounter...

Every police encounter contains a risk: The officer might be poorly trained, might act with malice or simply make a mistake, and civilians might do something that is perceived as a threat. The omnipresence of guns exaggerates all these risks.

Such risks exist for people of any race — after all, many people killed by police officers were not black. But having more encounters with police officers, even with officers entirely free of racial bias, can create a greater risk of a fatal shooting.

Arrest data lets us measure this possibility. For the entire country, 28.9 percent of arrestees were African-American. This number is not very different from the 31.8 percent of police-shooting victims who were African-Americans. If police discrimination were a big factor in the actual killings, we would have expected a larger gap between the arrest rate and the police-killing rate.

This in turn suggests that removing police racial bias will have little effect on the killing rate. 
A key assumption underlying this argument is that encounters involving genuine (as opposed to perceived) threats to officer safety arise with equal frequency across groups. To see why this is a questionable assumption, consider two types of encounters, which I will call safe and risky. A risky encounter is one in which the confronted individual poses a real threat to the officer; a safe encounter is one in which no such threat is present. But a safe encounter might well be perceived as risky, as the following example of a traffic stop for a seat belt violation in South Carolina vividly illustrates:




Sendhil is implicitly assuming that a white motorist who behaved in exactly the same manner as Levar Jones did in the above video would have been treated in precisely the same manner by the officer in question, or that the incident shown here is too rare to have an impact on the aggregate data. Neither hypothesis seems plausible to me.

How, then, can one account for the rough parity between arrest rates and the rate of shooting deaths at the hands of law enforcement? If officers frequently behave differently in encounters with black civilians, shouldn't one see a higher rate of killing per encounter? 

Not necessarily. To see why, think of the encounter involving Henry Louis Gates and Officer James Crowley back in 2009. This was a safe encounter as defined above, but may not have happened in the first place had Gates been white. If the very high incidence of encounters between police and black men is due, in part, to encounters that ought not to have occurred at all, then a disproportionate share of these will be safe, and one ought to expect fewer killings per encounter in the absence of bias. Observing parity would then be suggestive of bias, and eliminating bias would surely result in fewer killings.

In justifying the termination of the officer in the video above, the director of the South Carolina Department of Public Safety stated that he "reacted to a perceived threat where there was none."  Fear is a powerful motivator, and even when there are strong incentives not to shoot, it is still a preferable option to being shot. This is why stand-your-ground laws have resulted in an increased incidence of homicide, despite narrowing the very definition of homicide to exclude certain killings. It is also why homicide is so volatile across time and space, and why staggering racial disparities in both victimization and offending persist.

None of this should detract from the other points made in Sendhil's piece. There are indeed deep structural problems underlying the high rate of encounters, and these need urgent policy attention. But a careful reading of the data does not support the claim that "removing police racial bias will have little effect on the killing rate." On the contrary, I expect that improved screening and better training, coupled with body and dashboard cameras, will result in fewer officers reacting to a perceived threat when there is none.

---

Update (10/18). I had a useful exchange of emails with Sendhil yesterday. I think that we both care deeply about the issue and are interested in getting to the truth, not in scoring points. But there's no convergence in positions yet. Here's an extract of my last to him (I'm posting it because it might help clarify the argument above):
Definitely you can easily make sense of the data without bias. The question is whether this is the right inference, given what we know about the processes generating encounters.

Suppose (for the sake of argument) that whites have encounters with police only if they are engaging in some criminal activity, while blacks sometimes have encounters with police when they are completely innocent. This need not be due to police bias: it could be because bystanders are more likely to think blacks are up to no good for instance (Gates and Rice come to mind).

Suppose further that those engaging in criminal activity are threats to the police with some probability, and this is independent of offender race. The innocents are never threats to the police. But cops can't tell black innocents from black criminals, so end up killing blacks and whites at the same rate per encounter. If they could tell them apart, blacks would be killed at a lower rate per encounter. What I mean by bias is really this inability to distinguish; to see threats when none are present. 

I believe that black cops are less likely than white cops to perceive an encounter with an innocent as threatening. If a suspect looks like your cousin, or a guy you sit beside to watch football on Sundays, you are less likely to see him as a threat when he is not. That's why I asked you in Cambridge whether you had data on officer race in killings - when the victim is innocent the officer seems invariably to be white. So a first very rough test of bias would be whether innocents are killed at the same rate by black and white officers...

I've found the twitter reaction to your post a bit depressing, because better selection, training and video monitoring are really urgent needs in my opinion, and the absence-of-bias narrative can feed complacency about these. I know that was far from your intention, and you are extremely sympathetic to victims of police (and other) violence. You also have a responsibility to speak out on the issue, given your close scrutiny of the data. But I do believe that the inference you've made about the likely negligible effects of eliminating police bias are not really supported by the evidence presented. That, and the personal importance of the issue to me, compelled me to write the response.
---

Update (10/19).  This post by Jacob Dink is worth reading. Jacob shows that the likelihood of being shot by police conditional on being unarmed is twice as high for blacks relative to whites. The likelihood is also higher conditional on being armed, but the difference is smaller:


This, together with the fact that rates of arrest and killing are roughly equal across groups, implies that blacks are less likely to be armed than whites, conditional on an encounter. In the absence of bias, therefore, the rate of killing per encounter should be lower for blacks, not equal across groups. So we can't conclude that "removing police racial bias will have little effect on the killing rate." That was the point I was trying to make in this post. 

---

Update (10/21). Andrew Gelman follows up. The link above to Jacob Dink's post seems to be broken and I can't find a cached version. But there's a post by Howard Frant from earlier this year that makes a similar point.

10 comments:

  1. Rajiv: I think I get your point. How could we test it?

    Suppose we were talking about men and women. Assume 50% of each in the population. If (say) 60% of police encounters were with men, and (say) 70% of those shot by police were men, I don't think we would infer that police were biased against men. We would probably be looking at data on percentage of homicides done by men?

    ReplyDelete
  2. Nick, a simple first test would be to compare the rates of killing per encounter by white versus black officers. All the examples mentioned by Sendhil (Tamir Rice. Eric Garner. Walter Scott. Michael Brown) involved white officers. I believe that if a suspect looks a bit like your brother or cousin or the guy with whom you watched a football game last Sunday you are less likely to perceive a threat when there is none.

    Look at the quote from Leon Lashley in my post on the Gates arrest. Lashley defended his white colleague, but added: “Would it have been different if I had shown up first? I think it probably would have been different... black man to black man, it probably would have been different."

    http://rajivsethi.blogspot.com/2009/11/leon-lashley-and-gates-arrest.html

    APAA, yes, the numbers could be biased the other way... my point was that the numbers don't tell us what he claims they do, and to give reasons why I think they are biased in a particular way. Obviously we need better tests (see my response to Nick above).

    I'm aware of the statistics in violent crime, and linked in the post to a paper of mine with Dan O'Flaherty that tries to explain the disparity. Fear and preemption are central to the argument:

    http://www.sciencedirect.com/science/article/pii/S0094119010000343

    Part of the problem is the extremely low clearance rate for homicides with black victims, as discussed by Danielle Allen here:

    http://t.co/icCif0UXrN

    Thanks to both of you for the comments.

    ReplyDelete
  3. "This, together with the fact that rates of arrest and killing are roughly equal across groups, implies that blacks are less likely to be armed than whites, conditional on an encounter. In the absence of bias, therefore, the rate of killing per encounter should be lower for blacks, not equal across groups. So we can't conclude that "removing police racial bias will have little effect on the killing rate." That was the point I was trying to make in this post. "

    I'd actually be very cautious about making claims like this, given that my analysis uses a "per-arrest" metric that doesn't capture differences in armed/unarmed arrests. I should maybe try to remedy that…

    Just a clarification!

    ReplyDelete
  4. Thanks for the clarification Jacob, but if likelihood of killing conditional on both events (armed/unarmed) is higher for black versus white, while unconditionally they are about the same (Sendhil's point) doesn't there have to be a composition effect (blacks more likely to be unarmed)? What am I missing?

    ReplyDelete
    Replies
    1. I think it comes down to what we mean by likelihood of killing conditional on arrest being "about" the same for the two races.

      It is about the same, but not *exactly* the same in the dataset I'm using: 0.000132 for black, 0.000115 for white. That's a difference of .0000169. If we split by armed vs. unarmed, we get differences of .0000159 and .0000280, respectively. And there are far more armed crimes overall, so the overall mean is closer to the armed mean.

      What we're seeing is a small difference that matters a lot, especially (for unarmed crimes) if we look at the proportion increase, rather than the absolute increase.

      Delete
  5. Jacob, with all due respect (and "good job", etc.), I don't think your analysis tells us much new. " basically a noisy proxy for Sendhil's "shooting bias". The error bars are quite wide, first of all - wide enough that this new approach doesn't really contradict Sendhil's story. But more importantly, what we need is an estimate of the ratio of the probabilities, which is not equal to a ratio of the two estimates (and will also have wider error bars).

    Also, to test Rajiv's hypothesis, what we need is an estimate of the ratio of ratios (or if you want, the difference in differences) that shows that the rate of deaths/arrest is more skewed toward blacks when no gun is present. That would show that nonthreatening encounters are more likely to result in killings when the subject is black.

    Also, I'm not clear on how you proxied for the number of arrests by race in each state, without having arrest data to begin with. How did you do that?

    ReplyDelete
    Replies
    1. Hey Noah,

      Thanks for the feedback. I think I'm a little confused by your overall view of what would constitute getting at the 'real' question. Here's how I see the dialectic going:

      Sendhil: There is not really a shooting bias, once you control for arrest-rate, or if there is, it's trivial when compared to other racial biases.
      Me: There does seem to be a shooting bias, even after controlling for arrest rate. And it's not trivial, it's quite substantial in size: a 30% increase overall, and (given some assumptions), potentially a 2-fold increase for unarmed arrests.

      Hopefully this makes sense? I'm also confused by many of your points:

      "The error bars are quite wide, first of all - wide enough that this new approach doesn't really contradict Sendhil's story."

      This point doesn't make sense to me. I tested the trend in my post, and it's statistically reliable. This suggests the estimates I described in the previous paragraph are not just due to chance. Given the variability, it of course could be the case that the effect size is smaller than what I've estimated. Or it could be bigger. There's no reason to suspect either direction: all we know is that the effect size probably isn't zero (that's what the statistical test suggests). On balance, our best guess is the 30% measure I cited before— a substantial effect.

      "But more importantly, what we need is an estimate of the ratio of the probabilities, which is not equal to a ratio of the two estimates (and will also have wider error bars)."

      Why do we need to test the ratio of the probabilities? Which probabilities?

      "Also, to test Rajiv's hypothesis, what we need is an estimate of the ratio of ratios (or if you want, the difference in differences) that shows that the rate of deaths/arrest is more skewed toward blacks when no gun is present. That would show that nonthreatening encounters are more likely to result in killings when the subject is black."

      I disagree that this is what's needed. There is an overall bias, and it is robust when considering *only* the unarmed cases (I didn't show this test but I can post it if you'd like). It doesn't really matter whether it's stronger than the bias in armed cases.

      "Also, I'm not clear on how you proxied for the number of arrests by race in each state, without having arrest data to begin with. How did you do that?"

      I have arrest data for the whole US, split by race. I just partitioned this to each state, according to its population. This is a horrible and noisy method, but it shouldn't be a biased one, and therefore shouldn't effect any of the conclusions.

      Delete
    2. Noah, thanks for your comment but I'm a bit puzzled by it. If you look at the argument I sketched out in the first update it predicts a racial disparity in killings per encounter if and only if you condition on the encounter being objectively safe. And Jacob's test is supportive: the disparity is small for threatening encounters, much larger for safe ones. So I have to agree with Jacob's response here.

      Delete
    3. Here's how I see the dialectic going

      I think you mean "dialogue"... ;-)

      There does seem to be a shooting bias, even after controlling for arrest rate. And it's not trivial, it's quite substantial in size: a 30% increase overall, and (given some assumptions), potentially a 2-fold increase for unarmed arrests.

      It's a very noisy proxy, and also potentially biased. If blacks are arrested at a lower rate per encounter, this will drive your estimates above Sendhil's. Plus the noise of your proxy is not included in your error bands.

      Sendhil isn't controlling for arrest rate, he's just measuring killings per encounter. Which is what we really care about, since many encounters presumably don't lead to arrests.

      I tested the trend in my post, and it's statistically reliable.

      What test did you perform?

      Why do we need to test the ratio of the probabilities? Which probabilities?

      P(killing|arrest,black)/P(killing|arrest,white)

      P(killing|arrest,black,unarmed)/P(killing|arrest,white,unarmed)

      P(killing|arrest,black,armed)/P(killing|arrest,white,armed)

      And, to test Rajiv's hypothesis, we'd want:

      P(killing|arrest,black,unarmed)/P(killing|arrest,white,unarmed) - P(killing|arrest,black,armed)/P(killing|arrest,white,armed)

      or

      P(killing|arrest,black,unarmed)/P(killing|arrest,white,unarmed)/P(killing|arrest,black,armed)/P(killing|arrest,white,armed)

      It doesn't really matter whether it's stronger than the bias in armed cases.

      It doesn't if we're just discussing Sendhil's hypothesis. But if we're discussing Rajiv's hypothesis, we do need that, since Rajiv makes a different hypothesis from Sendhil's.

      Delete
    4. You know what? I think you're right about some of your concerns (though I disagree with some of your reasons). I don't think I can honestly estimate the statistical reliability of the effect: treating each state as an individual and building a confidence interval or statistical model around that isn't really sensible.

      Of course, if I skip the step of computing things state-by-state, and I just take the overall nationwide counts, it doesn't really change the pattern in the data:

      http://i.imgur.com/KxplqRE.png

      It's still the case that there is a disproportionate risk for African-Americans, even controlling for arrest rates. It's still a really big effect. And it's still the case that the magnitude of the effect appears to be similarly big for both armed and unarmed crimes (maybe even bigger for unarmed crimes).

      Anyways, thank you for making me think harder about this. I think you're probably right that, with this limited, 2015-only dataset, it's not really possible to test the hypothesis that there's a "difference in differences." I'm still interested in seeing how this looks with a larger database.

      Delete