Let’s play a game called “write the headline.” Here’s the story from the New York Times:
According to the latest federal figures, which were part of an annual survey, Monitoring the Future…. The report looked at a wide variety of drugs and substances. It found, for example, that drinking was steadily declining, with roughly 40 percent of high school seniors reporting having used alcohol in the past month, down from a peak of 53 percent in 1997. Abuse of the prescription painkiller Vicodin is half what it was a decade ago among seniors; cocaine and heroin use are at historic lows in almost every grade.
Cigarette smoking has also fallen precipitously in recent years. For the first time since the survey began, the percentage of students who smoked a cigarette in the past month dropped below 10 percent. Roughly 8.5 percent of seniors smoke cigarettes on a daily basis, compared with 6.5 percent who smoke marijuana daily, a slight increase from 2010.
[Also] More than 12 percent of eighth graders and 36 percent of seniors at public and private schools around the country said they had smoked marijuana in the past year. About 60 percent of high school seniors said they did not view regular marijuana use as harmful, up from about 55 percent last year.
How would you summarize this story in one headline?
No matter what you pick, I bet you can beat what what the Times editor came up with: “Increasing Marijuana Use in High School Is Reported“! The exclamation point is mine.
Well, good news is not news, is it?
Pete!
Why in hell should we trust a survey given to high school students to have any external or internal validity? Should we just assume that a teenager taking a no-consequences survey is telling the truth? Why?
We shouldn't put a lot of faith in surveys about illegal activity. But we probably can assume some consistency in error over time. In other words, are people less likely to tell the truth on those surveys now than they were back in the late 80s when I took them? Probably not.
But I agree, there are serious validity problems that probably (and understandably) underestimate illegal activity… Especially compared to surveys in countries where drug use isn't so criminal, like Portugal or the Netherlands.
Think of it like a weight scale that is 10 lbs off. It's not too useful for any one measurement, but there's still validity over time, as long as people are stepping on the same scale.
Consistency in errors and the non-random response of missing data are the two major errors of way too much survey data.
You have every reason to be suspicious!