There’s another quality policing podcast in which I talk to data analyst Jeff Asher about the Brennan Center’s latest report on crime. Asher had posted this thread about methodological problems in their data and analysis.
Brennan has a new report out showing murder down 2.5% nationally, but there are some major issues with that finding.
1) The figures cited aren’t year-to-date, they’re projected year end numbers based on around midyear counts.
2) Murder tends to pick up over the second half of the year, and any projection using midyear numbers will almost certainly be wrong.
3) They found murder -2.5% but included San Fran’s 2016 count in that. There was no count for 2017. Removing SF makes murder -1.5%.
4) Detroit is estimated to be -27%, but that’s based on Detroit’s open data site.
5) That’s problematic because the open data site is slow to add murders, so any year-to-date count will be wrong.
6) Detroit had over 130 murders as of late June according to the Detroit Police Department, and the 220 murders they project would be the fewest there since 1966.
7) Taking Detroit’s inaccurate count out takes murder in their sample from -1.5% to +0.7% overall. So Detroit’s inaccuracy explains the drop
8) The Phoenix count is similarly wrong. Phoenix had about 150 murders in 2016 but this report says they had 80 and project 60 for 2017.
9) The Phoenix figure was reached by using MCCA midyear data and doubling it, but Phoenix only reported Q1 data to the MCCA.
10) As of May Phoenix had 58 murders year-to-date in 2017 and 56 in 2016. Take away Phoenix and Detroit and suddenly murder is up 1.2% in the sample.
11) Which is to say nothing of the methodological issue of projecting midyear for 30 cities to a full year and calling it a national trend.
12) For what it’s worth, my midyear piece for @FiveThirtyEight shows murder up a few % but rising slower than previous years.
13) Also worth reading is @Jerry_Ratcliffe on why doing year-to-date analysis isn’t a great idea
14) Larger point is that measuring murder nationally is tough, drawing sweeping conclusions from badly incomplete data is a huge mistake in my opinion
This isn’t the first time the Brennan Center has released faulty and misleading reports on the rise in homicide. In July, after the last one, I finally made an attempt to talk to one of the report’s authors. Once I laid out my concerns, the correspondence ended. Today I asked the other author (via twitter) if he wished to be interviewed or engage in a civil discussion of methods. No dice, apparently he’s “alright, thanks.” It’s still an open invitation.
There are numerous problems with their analysis, but the most irksome to me is the straight-up misleading statement. I asked:
Is this statement [from your report] true? “Notably, 55.6% of murder increase 2014 to 2017 is attributable to two cities — Chicago and Baltimore.”
Because I know it’s not true, since about 14 percent of the murder increase from 2014 to 2017 is attributable to Chicago and Baltimore. He replied:
Yes. It’s true for the 30 largest cities (our cohort), not nationally.
This not an explanation as much as a confession because they don’t say “for the 30 largest cities (our cohort), not nationally” in their report.
I understand how they got their numbers; on my calculator, I can replicate their methods. That’s good, but not good enough. Their methods are faulty.
Here are some of my remaining unanswered questions I posted on twitter.
Since 2013, what is the change in homicides in those 30 cities? I get a decrease in 3 cities and an increase in 27. Is this correct?
Do you understand problems in saying a “percentage of increase in sample“? Substantively meaningless & statistically absurd.
If you have three years of data, why do 2017 tables only compare with last year, 2016?
It may turn out to be true, but still seems a odd choice that only mention of (20%!) 2-year homicide increase is as “short-term fluctuation”
If twitter can’t do this justice, I’d be happy to interview you for @QualityPolicing podcast.
I asked if we could “continue w/ a civil discussion of your methods?” Alas, the reply was: “I’m alright, thanks.”
For two main reasons, I’m not OK. I’d like the Left to stay committed to the truth. The generally decent Brennan Center should be above Heritage-Foundation-style BS.
But more importantly: when you say murder is down when murder is up, it’s not just an issue of truth. It’s also an attempt to make the murder victims — disproportionately poor young black men — disappear from our consciousness. As if they never existed. Do their lives not matter, too?
I, as well find this data hard to believe, when Chicago experiences a huge wave of violence every year on the fourth of July weekend and the numbers seem to keep going up every year. Also, it's hard to take those numbers seriously when they're predicted year end numbers. I would like to know where we stand right now with these numbers.
it seems like the Brennan Center is trying to pick and choose which statistics they want to add in the study. they are making generalizations on mid-year findings then doubling the findings to calculate the year-end statistics. the way they came to the conclusion is faulty, and it makes me wonder how many other studies that you hear on the news or read online are biased in this sense. it is important to look at studies and analyze predictions but this study should have also included the actual statistics of the year-end findings. then a comparison can be made and discussed to see if similar predictions were made.
This data is really interesting to me, mainly because I do feel that the numbers would be a lot higher but it does make sense that the number are the way they are mainly because of when the numbers are pulled throughout the year.
I heard the statistics and got really sad. I cant believe the crime number are so high! What has our country come to? BTW, your podcast is great and informative