The New York Times reports today on a RAND study (behind the Great Damned Elsevier Pay Wall) by Ruopeng An and Roland Sturm about the lack of “food deserts” in poor neighborhoods. Or more precisely about the lack of link between food deserts and obesity. More specifically, it questions the very notion of food deserts. From the Times:
There is no relationship between the type of food being sold in a neighborhood and obesity among its children and adolescents.
Within a couple of miles of almost any urban neighborhood, “you can get basically any type of food,” said Roland Sturm of the RAND Corporation, lead author of one of the studies. “Maybe we should call it a food swamp rather than a desert,” he said.
Sure thing, Sturm. But I suspect you wouldn’t think certain neighborhoods are swamped with good food if you actually got out of your office and went to one of the neighborhoods. After all, what are going to believe: A nice data set or your lying eyes?
“Food outlet data … are classifıed using the North American Industry Classifıcation System (NAICS)” (p. 130). Assuming validity and reliability of NAICS occupational categories is quite a red flag. It means that if something is coded “445110,” then — poof — it’s a grocery store! What could make for easier analysis? But your445110 may not be like my445110. Does your supermarket look like this:
Well the NAICS says it does because they’re both coded 445. New York is filled with bodega “grocery stores” (probably coded 445120) that don’t sell groceries. You think this matters? It does. And the study even acknowledges as much, before simply plowing on like it doesn’t. A cigarette and lottery seller behind bullet-proof glass is not a purveyor of fine foodstuffs, and if your data doesn’t make that distinction, you need to do more than list it as a “limitation.” You need to stop and start over.
Here’s one way to do it: a fine 2010 Johns Hopkins study edited by Stephen Haering and Manuel Franco. They actually care about their data. Read the first page in particular for the problems of food-store categorization. It matters. And notice the sections titled “residents personal reflections on their local food environment” and “food store owners’ attitudes regarding stocking healthy food.” What a concept for researchers to actually talk to people! (The picture above is from this study.)
I find this so frustrating because so much quantitative analysis is so predictably problematic, over and over, again and again, in exactly the same way. Here’s the mandatory (and then ignored) disclaimer (p. 134, emphasis added):
Possibly even more of a limitation is the quality of the … business listings, although this is a criticism that applies to all similar studies, including those reporting significant fındings…. More generally, categorizing food outlets by type tends to be insufficient to reflect the heterogeneity of outlets, and it is possible that more detailed measures, such as store inventories, ratings of food quality, and measuring shelf space, would be more predictive for health outcomes. Unfortunately, such data are very costly and time consuming to collectand may never exist on a national scale.
So let me get this right, because “all similar studies” use this flawed data, it’s OK? And because getting good data may be “very costly and time consuming to collect,” we’ll simply settle for what we have at hand? Bullshit!
You know, perhaps we never will have good data on a national level about what produce is sold in each and every store in America. I can live with that. But it is neither very costly nor time consuming to simply go into every store in any one neighborhood and see what is there. Do a spot check. Or at least read and learn from the John Hopkins study. I just found it on google without even trying. They managed just fine. And if a corner store sells three moldy heads of iceberg lettuce and some rotting root vegetables, it is not the same as Whole Foods simply because they’re both coded 445!*
Ironically, An and Sturm may still be right about their conclusions, but more by accident than design. Maybe the focus on food deserts is barking up the wrong tree. Perhaps obesity is notcaused primarily by lack of access to good food. Maybe people do not want to eat healthy foods. Or maybe people simply don’t know how to cook. Maybe we need to bring back Home Ec. I don’t know. Certainly, I think we can agree, culture matters. But quantitative people don’t like looking at culture because it’s so hard to count. And who has the time to do time-consuming ethnographies when we’ve all got to get our name on as many co-authored quantitative peer-reviewed journal articles as possible?
There actually is (or was?) an excellent produce store in Baltimore’s Eastern District, Leon’s Produce. Conveniently it was right by a busy drug corner. Talk about one-stop shopping! Seriously, as a cop, I could suppress the corner drug market and buy onions and carrots. And yet people would indeed pass up this local family-run store to buy a cheesesteak or yakomee.
Maybe the problem is intense neighborhood isolation. Drawing a geographic circle around somebody and saying a grocery store is “close enough” may not matter if you’ve never left your neighborhood, don’t have access to a car, or are afraid to walk down the block. Speaking of cars, Sturm also uses CHIS data in which “Only 3% of households … report not having access to a car.”
Well there’s another red flag.
What does “access” mean? I suspect to some it is gathering $10 for a gypsy cab or knowing somebody who may let you borrow their car in an emergency.
The authors acknowledge the limitations of CHIS data, and then go right on using it: “The response rate … remains low, and the current study sample has a large proportion of missing values” (30%, in fact!). If you’re looking at the problems of poverty in America and believe data that say 97% of people have access to a car, you’ve got your head up your ass.
And if you have bad data, it doesn’t matter what fancy quantitative methods you use. It’s putting lipstick on the damn pig of correlation. Garbage in, garbage out:
The primary dependent variables (i.e., counts of food consumption) are regressed on the explanatory variables using negative binomial regression models, a generalization of Poisson models that avoids the Poisson restriction on the mean-variance equality.
Wow! Negative binomial Poisson regression models to avoid the mean-variance equality restriction. I (to my shame) no longer have any idea what that means, even though Poisson regressions were all the rage when I was in graduate-school. But I do remember the fatal flaw of non-random missing data.
I’m not against quantitative methods. I’m against bad research.
And I also believe you need to talk to the people you’re studying no matter what methods you use. I don’t trust your study on poverty if you’ve never talked to a poor person. I don’t trust your research on police if you’ve never talked to a cop. I don’t trust your research on crime if you’ve never talked to a criminal. Nor do I trust your research on obesity if you don’t talk to a fat person. And if you’re going to write about food deserts, you’d better talk to some people who live in one. If you’re not careful, you may learn something before it’s done. Once you quant-heads actually talk to the people you’re studying, then you can go ahead and run all the regressions they want.
*Update (April 29): As one commenter pointed out, a Whole Foods is not coded the same as a corner store (because the Whole Foods is larger). Indeed. But you still get my point.
And here’s a picture of a corner “deli-grocery” in Crown Heights, Brooklyn (NYC):
It was in the Daily News because 14 were arrested for a running a drug ring from it. I strongly suspect it wasn’t a good place for quality groceries.
Having met a few of these right wing think tank researchers, it does not surprise me at all that they would make sweeping conclusions based on irrelevant data without ever stepping into the world they are analyzing. They could easily walk to a food desert in under 20 minutes from their Northwest DC offices, but that is waaaay outside of their comfort zone, and it wouldn't have any impact on their preconceived notions anyway.
Hi Joe
aren't you at work? taking time to read and comment on this blog might constitute "Theft of Honest Services"
How about those Nationals ? !
Historically speaking, RAND often writes what their benefactors wish to hear. They work within a quite limited frame of reference.
If the data doesn't support the desired conclusions- well – they draw them anyhow. RAND's work regarding The War on Them Drugs is revealing, to say the least.
Not revealing regard the issues involved, just revealing in terms of their intellectual prostitution.
RAND would doubtless argue with your use of the term 'right wing' describing anything they do or anyone they hire
"we are non-partisan, impartial, just the Facts, Ma'am, and our best honest analysis"
but hell, in this real world we live in, the elite center consensus is right wing to begin with, with only a very limited band of options available to solve any problem . By any objective measure, (my own, that is) , our 'Socialist' President is center-right at best, not to mention a red fanged imperialist SOB to boot.
Oh well, back to work- I am stealing my own honest services even as I write
Can't wait to show this around the School of Public Policy at George Mason, we live and die on quantitative methods and analysis.
Anecdotal point, many years ago I worked with a woman who'd taught elementary school in DC right out of college (mid 70s), she told me she used to bring fresh fruits and vegetables into her class for the students and they'd never eaten them fresh before. Didn't like them either.
The "Post" did a story a year or so ago on corner liquor stores in Northeast Washington adjusting to the changing demographics of their neighborhoods; carrying fewer 40s of Mad Dog and Ripple, more imported wines, taking down the Plexiglas enclosures, etc. Maybe fresh fruits and veggies are yet to come?
Been through B'more twice in the last week, Amtrak last weekend and in the car yesterday, can't get over the razed blocks over by Hopkins Hospital. I wonder how many 445120s are over there? I've heard of Leon's but have never been there, old Lexington Market guy myself.
Very, very good write-up. I'm a PhD student in health behavior and collaborative research, and also live in Detroit, so this issue is near and dear to my heart. I'm frustrated by the Times' shoddy reporting and the fact that they didn't give credit to the long history of research in this area.
You hit the nail on the head with your recommendation that people get out of their offices…I truly believe that working in a collaborative manner (my bias) and using both qual and quant data can push us far ahead in understanding obesity. Many researchers shy away from actually talking to people, and I don't understand that. Why sacrifice the truth so you can claim to be objective, when any outcome will be "tainted" by subjectivity anyway?
I think that some quantitative folk don't like talking to people for a combination of reasons:
1) It goes against what they (mistakenly) believe are properly scientific methods.
2) It is too time consuming.
3) They lack the social skills to actually engage strangers in conversation. They're also, in the case of some Baltimore or Detroit neighborhoods, simply afraid.
4) There is a bias in the academy and in journal publications toward quantitative research. So they're on the winning team.
I have heard of “food deserts” before, but I guess my experiences are an anomaly. Perhaps because the cities I’ve lived in (after I graduated college) and the VERY poor neighborhoods I’ve lived in were in the south, where farms and farmer’s markets were the norm. However, I have little doubt that big cities in the northern U.S. do have fewer places to buy fresh produce and/or less processed foods.
Ironically something different has happened where I am now. What happened seems to be similar to what happened in parts of California. The ultra-rich took a liking to our town and now it’s not just an average town, but upper crust.
So I experience a similar problem, some ingredients for the food I make are hard or impossible to find. The best I can figure is that many of the people who have moved here eat out most of the time or buy foods that are already mixed up. So instead of making their own hummus they buy it prepared.
And the one that impacts me the most is baking bread. There must only be a handful of us who bake our own bread; yeast is nearly impossible to find. When I can find it, it’s in little packets instead of jars.
As someone who cooks a lot, I find it a bit shocking how much of up-scale "grocery" stores is filled with prepared foods.
Luckily in my neighborhood there are still produce stores (open 24 hours), butchers, fish mongers, and bakers galore. This local food swamp is a big part of the reason I moved here (Astoria).
As to the yeast issue, not that this is a cooking blog, but might I recommend keeping a starter? Always there for you. And better tasting bread, too.
I'm going to make some hummus right now. You know, from scratch, with chick peas that are simmering as I type. It's not hard. And it sure is cheap.
I'm not a quant person, but I agree with your critique of how this data on alleged food deserts was gathered. I think there's a major problem in any study about this issue, which is that defining what makes an acceptable grocery store is pretty subjective. I think a lot of full-fledged grocery stores in nice NYC neighborhoods have wretched produce, so does that make them food deserts? Michelle Obama doesn't seem to think so.
Your point about maybe people don't want to eat healthy food is my theory about what causes obesity. It's not "food deserts" (whatever that means to you) it's what people, rich and poor, choose to eat. Even when people in America have plenty of access to nice produce, they often eat a whole lot of junk food.
That's my main beef with the whole concept of the (alleged) food desert: that it's a major cause of obesity. I think people eating junk food is a major cause of obesity. Even when there is plenty of produce to be found, a lot of people choose to buy and eat Coke and Cheetos.
Yes, hummus is very easy and inexpensive, and let’s not forget darn tasty! 🙂
Bumped up it again! So far all I’ve been able to get are cans of chick peas. I haven’t been able to find bags of uncooked beans. For the moment I’m not sure how much $ I’d save cooking my own beans, but at least they wouldn’t have preservatives in them. I’ve only been able to find one brand of garbanzos without preservatives and it was at a store that is not close to me.
Baba Ghanouj is another one of my favorites, the only non-locally grown produce (for me) is lemon (not counting cumin, tahini, and olive oil that some recipes call for).
The good news is there seems to be attempts to nurture inner-city gardens; I saw a report 5-10 years ago about a retired basketball player now devoting all his time to get them started.
Anyway, it’s good to call out that kind of research.
I think I’m going to go get a snack which will involve my homemade hummus. I hope yours turned out great!
Does not 445110 exclude convenience stores. For example, see census.gov/econ/census02/naics/sector44/445110.htm.