Monday, July 11, 2016

Police Use of Force: Notes on a Study

A new empirical analysis of police use of force by Harvard economist Roland Fryer is attracting national attention. The paper deals with both lethal and non-lethal force, using a variety of different data sets, some public and some painstakingly assembled by the author and his team. Given the harrowing events of the past week, it's likely that his results on shootings will attract the most attention, but it's worth carefully considering both sets of findings.

Fryer provides evidence of significant racial disparities in the experience of non-lethal force at the hands of police, even in data that relies on self-reports by officers. Using official statistics from New York City’s Stop, Question and Frisk program, he finds that blacks and Latinos are more likely to be held, pushed, cuffed, sprayed or struck than whites who are stopped. This remains the case even after controlling for a broad range of demographic, behavioral, and environmental characteristics. And using data from a nationally representative sample of civilians, which does not rely on officer accounts, he finds evidence of even larger disparities in treatment.

But Fryer also reports an absence of racial bias in police shootings for a select group of jurisdictions. He recognizes that a proper analysis of police bias in the use of lethal force requires data not only on those incidents in which shootings occurred, but also those in which suspects were successfully pacified and disarmed. Data of this kind is extremely hard to come by, but he has managed to obtain incident reports on arrests in Houston that can be used for this purpose. 

The focus is on arrest categories that are more likely to involve incidents resulting in justified use of lethal force. It turns out that in this arrest data 58% of the population is black, while in the shooting data the corresponding share is 52%. This immediately implies that in the absence of controls for other features of the interaction, blacks in the arrest population are less likely to be shot than whites. He finds that controlling for other features of the interaction "does not significantly alter the raw racial differences." Here is how Fryer characterizes these findings:
Given the stream of video "evidence", which many take to be indicative of structural racism in police departments across America, the ensuing and understandable outrage in black communities across America, and the results from our previous analysis of non-lethal uses of force, the results displayed in Table 5 are startling... Blacks are 23.8 percent less likely to be shot by police, relative to whites.
He describes this as "the most surprising result of my career."

While it is entirely possible that the Houston Police Department doesn't exhibit systematic racial bias in the use of lethal force, I'm not sure such an emphatic conclusion is warranted. A close look at the arrest data (Table 1D) alongside the shooting data (Table 1C, column 2) reveals a number of puzzles that should be a cause for concern. In the arrest data only 5% of suspects were armed, and yet 56% of suspects "attacked or drew weapon." This would suggest that over half of suspects attacked without a weapon (firearms, knives and vehicles are all classified as weapons). Moreover, there are large differences across groups in behavior: two-thirds of whites and one-half of blacks attacked, a difference that is statistically significant (the reported p-value is 0.006).  

What this means is that the pool of black arrestees and the pool of white arrestees are systematically different, at least as far as behavior is concerned. So the raw data comparison described as startling in the quote above is not really valid. (I made a similar point in response to a piece by Sendhil Mullainathan a few months ago). Still, Fryer controls for these differences in behavioral and contextual characteristics and finds that the basic picture doesn't change. This has to be taken seriously. The key question, to my mind, is whether these controls are adequate. 

I personally would be more convinced if the arrestee pool looked more like the shooting victim pool. For instance, 18% of arrestees, but only 4% of shooting victims are female. I suspect that many of the interactions in the arrestee pool are not threatening, even from the subjective perspective of the officers involved. And others are so obviously threatening---for instance those involving suicide-by-cop---that no discretion or judgement is really necessary. Pruning these from the data might give us a clearer picture of bias in the use of discretionary lethal force. 

Despite these concerns, I think that there is a case to be made that there is no systematic bias against blacks in the lethal use of force within the Houston Police Department. What one ought not to conclude, however, is that this applies nationally. The analysis of other jurisdictions considered in the paper is restricted to encounters in which shootings actually occurred, and cannot therefore be used to answer the same kinds of questions that the Houston data allows. 

One last point about shootings: I'm not sure why there are quotation marks around the word "evidence" in the above quote. Video evidence, for all its flaws, is still very powerful evidence. It was video evidence that led to the indictment of Micheal Slager on murder charges, and the conviction of Sean Groubert for assault and battery. It is selective and cannot establish the presence of racial bias in individual cases, but surely it can't be dismissed out of hand.

Finally, consider Fryer's analysis of non-lethal force, which is consistent with earlier findings. Aside from being fundamentally unjust, disparities in the use of non-lethal force have some really important implications for crime rates. The harassment of entire groups based on racial or ethnic identity is a major obstacle to witness cooperation in serious cases, including homicide. In fact, given the importance of corroboration, a belief that other witnesses will not step forward can be self-fulfilling.

With witnesses routinely unwilling to come forward in some neighborhoods, people can be killed with near impunity. And this significantly increases the incentives to kill preemptively, in a climate of reciprocal fear. Low clearance rates for homicide are directly responsible for high rates of killing, and both of these are held in place by distrust of the criminal justice system by potential witnesses. The excessive and discriminatory use of non-lethal force by police thus ends up having indirect lethal effects.

26 comments:

  1. It should also be pointed out that Houston spent literally millions of dollars (and fired literally dozens of police officers) in the process of fixing a known systemic racial bias in just the last few years. Comparing this city to a city that has done neither is an exercise in bad statistics.

    ReplyDelete
    Replies
    1. Important point... do you have a link for this?

      Delete
    2. http://www.houstonchronicle.com/news/houston-texas/houston/article/McClelland-presided-over-culture-change-at-HPD-6845901.php

      I found an article that seems to back up what Hardison posted.

      Delete
  2. Harrison - Interesting point. A related one is that large cities like Houston tend to be more diverse, with police departments that are also more diverse, than smaller communities and suburbs. Their police officers are therefore more likely to be more trained, experienced in (and to have been evaluated on) their interactions with minorities. This can also influence the conclusions of analyses that are skewed toward statistics from larger cities.

    In this regard, it's notable that (if I recall correctly) the more apparently egregious, video recorded shootings of black men all took place in suburbs (the NYC choking incident being a notable exception).

    ReplyDelete
    Replies
    1. Yes, lots of variation by location, this is a good point.

      Delete
  3. I'm just a lowly undergraduate econ major who has yet to take an upper-level methodology class, but doesn't the data indicate that the HPD discriminates against white suspects in situations where lethal force is used? That sounds really silly when I say it aloud, but I don't understand why he concludes racial disparities don't exist in use of lethal force given his data and analysis.

    ReplyDelete
    Replies
    1. The arrest data is 18% female, the shooting data 4% female. Does this mean the HPD discriminates against males? Not at all. The interactions are not qualitatively the same, or equally threatening to the officer on average. So you can't just make an inference from the raw data. Once he controls for behavioral and contextual factors, there is no statistically significant difference.

      Delete
  4. I find this study difficult it to believe. I would want to see the raw data, the complete methodology and the stated objectives of the study. I lived in Houston.

    ReplyDelete
    Replies
    1. I'd like to get my hands on the data too, but even then it would be difficult to replicate since the incident reports have been interpreted and coded my multiple individuals over a period of time. So one would need the incident reports also, or at least a random sample to see how this was done.

      Delete
  5. Prof Sethi, can you please comment on the following sentence from the study, specifically on the matter of statistical insignificance? Does the fact that the reported data is statistically insignificant lend to your critique of overinterpretation?
    "Blacks are 23.8 percent less likely to be shot by police, relative to whites. Hispanics are 8.5 percent less likely to be shot but the coefficient is statistically insignificant."

    ReplyDelete
    Replies
    1. The "significance" there refers to Hispanics, the black-white differences in the raw data are significant. But this doesn't mean much, without controlling for behavioral and contextual attributes of the encounters. Once controls are added, significance is lost.

      Delete
  6. Thanks for clarifying! My initial thoughts on the study was that the data did not support the conclusions/interpretations as reported in NYT. I wondered in particular if the claims of "startling" findings would have help up in peer review. I also wondered if it is customary in Economics to frame untested (i.e. non peer reviewed) conclusions in the media (or allow such framing). Many biomedical science journals (e.g. Nature, Science) place an embargo on media prior to peer review or publication.

    ReplyDelete
    Replies
    1. I was very surprised (indeed quite startled) to see tentative and location specific findings being described as startling. I think it was an invitation to irresponsible reporting, which is what we saw at least initially.

      Delete
  7. I have just written a blog for Huffington Post on this issue and I looked at the instances in which police killed UNARMED whites and blacks---the Washington Post of every police killing of civilians in the US in 2015----unarmed whites were a bit more likely to be depicted as "attacking" police than black ones. This suggests there is a higher bar to shoot whites and is consistent with the analysis above. Whites are more apt to have to attack to get shot.
    https://blogger.huffingtonpost.com/tmp/10903644_p.html

    ReplyDelete
    Replies
    1. Tried the link but it seems broken...

      Delete
  8. Many things that seem obvious are actually quite wrong. By your reasoning (comparing arrests to shootings) there is significant gender bias against men in police shootings. You have to look at the nature of the interactions, especially if you're interested in the rates of innocents getting shot. But I don't think you're interested in thinking about this at all, since the answer is so obvious.

    ReplyDelete
  9. There would be no gender bias since arrests of women for violent crime are minuscule. If there's any significant bias against blacks, it would have to show up in aggregate data. The opposite is the case. The bias goes the other way. African American police victimization rate is significantly lower than their participation in violent crime.

    ReplyDelete
  10. Look at the Fryer data: women are 18% of the arrest pool, 4% of the shooting pool. I don't think this reflects anti-male bias but this is where your logic leads. In any case, since your mind is made up already there's no productive conversation to be had.

    ReplyDelete
  11. Miraculous. Amazing what you can do by picking and choosing among databases to confirm your preconceptions.

    ReplyDelete
  12. Did I miss this part of people's analyses?

    The data are not on lethal force by race, they are in lethal force by race GIVEN that an individual has been stopped by the police.

    That right there is a huge difference. Is it possible that Incident Reports are written up in such a way tends to justify whatever interactions occurred?

    Isn't this one of the major reasons video "evidence" has become so important--i.e. look at all the times the video proves that officers outright lied in their reports (eg Laquan McDonald, Sandra Bland).

    ReplyDelete
  13. I'm not a statistician, but the fact that black people are more likely to experience non-lethal force but then have that gap vanish for lethal force -- to me, that seemed to suggest that black people may be better at deescalating police encounters. This makes sense to me, considering:

    1. Black parents give "the talk" about dealing with police more than white parents, so black people get more education on how to deal with police safely.
    2. Black people are more likely to be pulled over, stopped and frisked, searched, etc -- so they get more real world practice with dealing with police safely.

    Do you think that idea might have any merit?

    ReplyDelete
    Replies
    1. JTown, your point number 2, that blacks are more likely to get pulled over, illustrates a possible serious flaw in the methodology used. If Fryer looked at the percentage of police stops that resulted in police shootings, then it's quite relevant that blacks are pulled over far more than whites for frivolous reasons. (For example, one police chief admitted that in a population that was 40% black, all 63 people stopped for seatbelt violations that year were blacks.) Since the police rarely stop whites for trivial violations, the whites stopped are far more likely to be serious criminals, and therefore more likely to get shot. If all of the armed and dangerous whites driving down the street were pulled over and shot (3 out of 3, say), and all the armed and dangerous blacks were pulled over and shot (4 out of 4, say), and 2 of the 63 black seatbelt violators were shot (by mistake, you've seen the videos, so you know it happens), then Fryer would report that a far greater percent of the whites THAT WERE PULLED OVER were shot (3 out of 3, 100%, in this made-up case) than the percent of blacks that were pulled over that were shot (6 out of 67, 9%) since all of the trivial stops were of blacks, and they rarely (but not never) ended in shootings. These numbers don't tell us anything.

      The numbers you want are the percent of white citizens who were shot by police in a time period vs the percent of black citizens who were shot. Or, you could look at white/black males that were shot, or drivers, or pedestrians, but NOT the % of white and black people PULLED OVER who were shot.

      Delete
    2. JTown, your point number 2, that blacks are more likely to get pulled over, illustrates a possible serious flaw in the methodology used. If Fryer looked at the percentage of police stops that resulted in police shootings, then it's quite relevant that blacks are pulled over far more than whites for frivolous reasons. (For example, one police chief admitted that in a population that was 40% black, all 63 people stopped for seatbelt violations that year were blacks.) Since the police rarely stop whites for trivial violations, the whites stopped are far more likely to be serious criminals, and therefore more likely to get shot. If all of the armed and dangerous whites driving down the street were pulled over and shot (3 out of 3, say), and all the armed and dangerous blacks were pulled over and shot (4 out of 4, say), and 2 of the 63 black seatbelt violators were shot (by mistake, you've seen the videos, so you know it happens), then Fryer would report that a far greater percent of the whites THAT WERE PULLED OVER were shot (3 out of 3, 100%, in this made-up case) than the percent of blacks that were pulled over that were shot (6 out of 67, 9%) since all of the trivial stops were of blacks, and they rarely (but not never) ended in shootings. These numbers don't tell us anything.

      The numbers you want are the percent of white citizens who were shot by police in a time period vs the percent of black citizens who were shot. Or, you could look at white/black males that were shot, or drivers, or pedestrians, but NOT the % of white and black people PULLED OVER who were shot.

      Delete
  14. Thanks. Fryer's data capture methodology was actually (in my opinion) the most important part of the work. Non-police scientists attempting to code police narratives have fallen victim repeatedly to what I call "Starsky & Hutch Effect": Having grown up on TV shows and movies, we believe truly we understand policing, when what we really understand is policing on cop shows (https://nselby.github.io/Are-You-Certain/). As you say, law enforcement data is hard to get but, speaking as a police detective and subject matter expert in law enforcement data I will say that while I agree that video isn't "evidence," it is rather interesting anecdotal information. Your comment about Walter Scott and Michael Slager notwithstanding, so much in law enforcement is highly nuanced that one cannot rely on video to be consistently valuable, or at least consistently valuable in a consistent manner. As I wrote (http://goo.gl/Gdz1Sc) with Rachel Levinson-Waldman of The Brennan Center, a pair of videos (https://www.youtube.com/watch?v=Gu0wMBJn-r8) from a Texas police department illustrates the point neatly. The first, from a dash-cam, appears to show an officer being overly aggressive with a driver he had pulled over. The second, from the officer’s body camera, shows the driver reaching towards a gun on the seat. There are many examples of this.

    Also it must be clear that video is not widespread - I have seen many estimates. My research (http://amzn.to/1q0pkXx) shows video was available only in 26 percent of the cases in which an unarmed person died after an encounter with police. In two 2015 cases, eyewitness-shot video was crucial in showing that officers had lied to investigators. In two other cases, video exonerated officers. But in all four of these cases, the video was taken by bystanders or surveillance cameras, not by police-operated video (https://goo.gl/fYsryR). There's just not enough.

    Sadly, the best and most consistent data we have around the country is 911 call data, which is only evidence that someone is saying that a crime is in progress. The data is "best" for the simple reasons that 911 call-taker/dispatcher training is consistent across the country and asks consistent questions about suspect and caller (see, eg, https://fortress.wa.gov/cjtc/www/images/training/T1-%20Combined%202011%20Student%20Resource%20Guide%20010914.pdf). This allows us to gather, from across the country, 911 call logs that may be searched for consistently REQUESTED (not granted) information about the race, gender, height, weight, presence of a weapon, drugs, apparent mental state and other crucial questions about crime suspects as reported to police by the community. When compared to those actually confronted by police, we were able to conclude that in the 153 cases in which unarmed civilians died after confrontations with police in 2015, the majority of the decedents were NOT selected by the police but rather by the community (there were no cases of false identification).

    The good part of Fryer's paper and your blog post is the discussion of police data. Typically it is filthy, uncoded, duplicative, based on police narratives that are themselves somewhat suspect, with different data types arbitrarily segregated into different data stores (usually each with its own proprietary, often non-relational, data format) designed in the early 1990s without any capacity to aggregate, let alone clean, de-duplicate and normalize, and forgetting completely the ability to mine.

    I would like to agree that Fryer's "surprise" notwithstanding there are serious problems with his conclusions as stated, but that the patterns he observes do have value - and his method of translating and coding police narratives, while simple and relatively common in other areas, is instructive to non-police researchers looking at police data.

    Police data is, like, hard.

    ReplyDelete
  15. Not understanding how Krzys was being stubborn here. I've since done my analysis of Fryer's use-of-force data to show how academic conclusions really don't add up to much "on the ground." Or as I wrote in August 2016 about Fryer's conclusions on lethal use of force, "People still freaked out, specifically about the evidence dealing with police shootings — it went directly against the massively popular narrative that began after Trayvon Martin’s death and erupted in Ferguson, Missouri. Fryer himself tried to answer reader questions. Though, it doesn’t take a “genius” to conclude for the past few years that if a demographic that is 25% of those being killed by police is also 27% of the those arrested (36% for violent crimes), and 43% of the persons killing cops, then there might very well not be nationwide systemic racial bias against that demographic."

    But we can keep on using a obfuscating gender argument, which I'm not sure that has to do with anything. Yes, the data has to be looked at cautiously and carefully; but the big round numbers kinda say the obvious: Policing isn't racially biased when it comes to lethal use of force.

    https://medium.com/@agent.orange.chicago/how-roland-fryers-controversial-study-on-racial-bias-by-police-actually-shows-negligible-bias-ea3a8b1fd293

    Noted points on non-lethal Use of Force that "proves" disparities that Fryer reported:
    1. Prevention the rate of police killings per million for African Americans has fallen by 70 percent over the last four decades but the rate remains as high or higher for whites, Latinos and Asians.

    2. This is the “next step” I’m doing with Fryer’s data published in The National Bureau of Economic Research as translated by the New York Times. Do the simple math on any interaction with New York City police (see graphic above) during the controversial “stop-and-frisk” decade and you get nearly the same percentage for black and white citizens:
    * 1 out of every 64 blacks stopped will have a weapon DRAWN on them (1.5 % of encounters)
    * 1 out of every 77 whites stopped will have a weapon DRAWN on them (1.2 % of encounters)
    Or:
    * 1 out of every 185 blacks stopped will have a weapon POINTED at them (.5% of encounters)
    * 1 out of every 232 whites stopped will have a weapon POINTED at them (.4% of encounters)

    ReplyDelete