Few rhetorical weapons are wielded with as much glee and as little caution as the phrase “a new study shows.” Science, this tries to tell us, has spoken, and any dissent or skepticism about the new findings betrays an opposition to empiricism, if not Truth itself. Although this happens in every field, it’s particularly common in social science, where research often intersects with matters of family and morality.
This is not how science is supposed to work. Good science values consensus over novelty and requires new theories to provide better answers than their predecessors’. As such, open-minded skepticism is the best response to research that tries to overturn decades of established theory. Even when new theories are confirmed, they tend to illuminate subtleties and nuances that had been missed, rather than overthrow our basic understandings.
For every potential Copernicus or Darwin in a given field, there are tens of thousands of researchers exploring narrow—but important—questions that expand our existing knowledge. In this sense, good science is overwhelmingly conservative, not revolutionary.
Unfortunately, science, especially social science, exists in a highly politicized and media-saturated environment that celebrates novelty over consensus and drama over diligence. When presented with university press releases making bold claims, harried reporters often neglect to ask the kind of informed, challenging, and skeptical questions they’d ask anyone else. Even when journalists intend well, research that confirms popular worldviews tends to be welcomed without reflection. At the same time, scholars whose conclusions are out of fashion are often subjected to hostile scrutiny based less on evidence than ideology.
Problem, Meet a Potential Solution
These failures in science and science communication affect the real world. More and more often, politicians and (increasingly) judges look to social science to explain current trends and the likely effects of new laws and regulations. How difficult is it to get an abortion? What are the likely effects on the environment of having more children? How much does marriage affect public health?
These kinds of questions have enormous ramifications on society and public policy. But with credulous reporting—and original research often hiding behind paywalls, mountains of statistics, and academic style—the public has had few places to turn for accessible and useful information.
That’s the problem The Unskewed Project intends to fix. Launched late last year, and co-sponsored by The Wheatley Institution, The Austin Institute, and The School of Family Life at Brigham Young University, Unskewed reviews new social science to explain how it was done, why it matters, and how it’s being spun.
“The peer-review system was set-up to provide a check on bad science, but we felt there was a need to provide an extra critique for research that was impacting the public dialogue” said Brian J. Willoughby, the editor-in-chief of Unskewed and an associate professor in the School of Family Life at Brigham Young University. “The echo chambers of the media and academia have led to a situation where research findings are less scrutinized and more blindly accepted more than ever before. Unskewed was created to provide a counter-point to flawed social science research being disseminated in the public.”
The results are sometimes unsettling, sometimes funny, but always illuminating. To get a better sense of the problem, here are just some of the most egregious examples from the past year.
1. Bad Premise, Worse Metric
Can C02 emissions released hundreds of years in the future save the environment before it’s (supposedly) destroyed in the next few decades? If you answered in the affirmative, you might have a career in academia ahead of you.
In a paper published last year, two researchers in Sweden investigated whether governments were recommending the kind of drastic lifestyle changes that can, they said, stave off environmental disaster. Those changes included giving up cars, turning down transoceanic flights, eating vegetarian, and having fewer kids. In fact, having one fewer child, they said, saves 58 tons of C02 per year, an enormous sum that outweighed all other options they explored.
This is, in a word, crazy. To begin with, it assumes that humans are always a drain on the environment. “Much of our hope in reducing emissions is pegged to the development of unproven technologies,” writes Spencer James in his Unskewed review. “[B]ut, if we lose the combined potential of millions of unborn, such developments may take longer.”
Second, the statistic the researchers use holds babies born today accountable for a percentage of the carbon emissions of all their descendants, some of whom won’t be born for centuries. Yes, really. This method makes even less sense when you consider that the researchers’ premise was that we need drastic changes now to prevent imminent disaster.
2. Smug Conclusion, Obvious Advocacy
In a display of remarkable hubris, researchers at the University of Michigan concluded that any further research comparing monogamous and non-monogamous couples was “not imperative.”
To justify this incredible claim, they cited their own research, which was well-conducted. However, they breezed past its limitations, including that they never asked whether participants were married, cohabiting, or in some other form of committed relationship, a major oversight that, even among the sexually “liberated,” should have been considered.
But the kicker came when the researchers argued that the strength of their findings mattered less than their potential for positive social change: “[C]onvenience sampling [such as through Craigslist] is an effective way to reach members of stigmatized groups. Despite potential problems … this type of research has had tremendous significance for the lives of marginalized groups. For example, social science research was cited in the monumental Obergefell v. Hodges marriage equality case and in related lower court rulings as well … despite the fact that much of the research cited relied on self-report measures and nonrepresentative samples.”
“Put simply,” writes Unskewed’s Alan Hawkins, “social justice trumps science.”
3. Conclusion First, Evidence Second
In another example of putting activism before empiricism, researchers at the Guttmacher Institute investigated whether women who traveled either 100 miles or out of state to procure abortions were inconvenienced and, if so, what the effects of that inconvenience were. Unsurprisingly, most reported troubles involving “travel-related logistical issues, system navigation issues” and similar frustrations that had less to do with abortion than with traveling a moderate distance. “In other words,” writes Unskewed’s Mark Regnerus, “their sample criterion guaranteed they would come to the conclusions they did.”
At least willing to be honest, the researchers concluded that they could find “no direct link between any one barrier and any one consequence … and no clear patterns of barriers and consequences according to women’s demographic characteristics.” So why bother to conduct a study with such an obvious conclusion?
Regnerus noticed the paper makes repeated reference to the U.S. Supreme Court’s 2015 Hellerstedt decision, which ruled that states cannot impose undue burdens on women seeking abortions. As such, he suspects that this paper was intended as “a political and legal hammer,” allowing abortion advocates to show that such burdens have been documented in a peer-reviewed journal.
4. Humble Research, Arrogant Reporting
Even when a study is careful and circumspect, journalists can ruin good work with sensationalist headlines and bad reporting. That’s precisely what happened to Dimitry Tumin, a researcher at The Ohio State University College of Medicine. In a recent paper, Tumin concluded his research showed that the (well-documented) positive health benefits of marriage had somewhat diminished in recent years, especially among young people. Tumin suggested this may be less a commentary on the benefits of marriage than on the limitations of his study and, perhaps, how current sexual mores blur the lines between marriage and singleness, and suggested that more research be done.
As you might imagine, the popular press ignored his conclusions and offered readers of a superficial gloss of his results with some commentary of their own: marriage doesn’t make you any healthier, so why not just—wink, wink, nudge, nudge—enjoy the single life?
“[P]opular media,” writes Alan Hawkins, “have missed the most important implication of [Tumin’s] research: The deinstitutionalization of marriage may be eroding the benefits of marriage for its members, reducing overall health and well-being in our society.”
Like so much reporting, Hawkins argues, it was a wildly successful exercise in missing the point.