How The Replication Crisis Undermines Government By Bureaucracy

How The Replication Crisis Undermines Government By Bureaucracy

Perhaps the best indicator of how the experimental irreproducibility crisis threatens secular progressivism is how lopsided the political response to solving it has been.
Lewis M. Andrews
By

For decades, critics of secular progressivism have been skeptical of the many billions Congress and state legislatures annually spend on university research, especially in the social sciences. What leftists like to tout as a valuable investment in better understanding and improving the human condition is increasingly seen by others as a cleverly camouflaged marketing effort to justify bigger and more intrusive bureaucracy.

It has been especially hard to ignore that almost every academic study includes two implications. The first is that government officials are even wiser and more worthy of the public’s trust as a result of the study. The second is that the results of the study require “further investigation.” Thus, studies generally suggest a mutually self-serving relationship that trades generous research subsidies for the apparent scientific endorsement of statist policies.

The late cultural philosopher Irving Kristol coined the phrase “New Class” to describe the growing number of economists, environmentalists, health-care administrators, policy analysts, and other public workers with a vested interest in hyping any experimental justification for their authority.  Kristol did not suggest anything as blatantly corrupt as a coordinated effort by bureaucrats to elicit favorable research results. He simply noted the natural human tendency of government officials to support the academics whose research seemed to confirm their expertise, as well as the natural temptation of academics to tell their funders what they want to hear.

In the years since Kristol’s work, progressivism’s skeptics have frequently highlighted the inverse relationship between the expansion of scientifically based public programs and the growth of the problems the programs were supposed to solve. As writers like City Journal senior editor Steven Malanga have pointed out, the states with the largest and supposedly most sophisticated bureaucracies almost always have the lowest-quality schools, health-care facilities, and mass transit systems. Indeed, there seems to be a direct connection between how much a state spends to “scientifically improve” its public services and the percentage of its citizens eager to migrate elsewhere.

Scholars have also been able to show that when academic studies do occasionally contradict left-wing agendas, they have a hard time getting the peer endorsements needed for publication. This is true even when the rejected papers are just as well documented as far more liberal studies.

There are even a small number of anti-progressive (or at least fair-minded) academics who have been willing to go public about the blatant funding bias they see in colleagues’ research. In theory, faculty have an objective “truth-finding” and “truth-telling” role in our society, says University of California-Berkeley law professor Stephen D. Sugarman, but “because of their own financial interests, [they] may be dishonest in what they say they have discovered and/or how they describe the state of knowledge in their field.”

Musa al-Gharbi, a Paul F. Lazarsfeld Fellow in sociology at Columbia and a research associate at Heterodox Academy, similarly believes that nearly every academic paper touching on “how society should be best arranged” has likely been subject to “prejudicial design,” based on how the outcome would personally profit the author. In his recent book on “The Administrative State,” University of Nevada professor John Marini observes that the names of university subject specialties and government departments now so closely mirror each other as to render the presumption of academic independence an obvious fiction.

But perhaps the biggest threat to secular progressivism’s claim to scientific validation is the growing awareness of just how little of that validation can be replicated experimentally. In 2005 Dr. John Ioannidis, co-director of Stanford University’s Meta-Research Innovation Center, wrote a now-famous paper showing that much of what has passed for “settled” research in the social sciences and related disciplines is, in fact, the product of unreliable statistical testing, the use of small sample sizes, unwarranted credence is given small effects, unshared experimental data, and other scientifically dubious methods.

Ever since Ioannidis’s paper, it has become painfully clear that even many of the most influential experiments in sociology, political science, social psychology, economics, climate science, anthropology, education, and medicine cannot be duplicated. Studies supposedly confirming the leftist belief that discriminatory behavior stems from unconscious stereotyping, for example, turn out to be based on highly unreliable methodology.

In 2015, the journal Science tried to replicate the findings of 100 articles published in three prominent psychological journals during 2008 and got significant results for only 36, compared with the significance claimed by 97 of the originals. A similar study one year later in the Finance and Economics Discussion Series of the Federal Reserve could not reproduce the outcomes of a majority of prominent economics articles.

Ioannidis believes that up to half the discoveries published in peer-reviewed social science and medical journals are probably wrong, an opinion he shares with The Lancet’s medical journal’s respected editor-in-chief, Richard Charles Horton. National Association of Scholars President Peter Wood has argued that many of the regulations, laws, and social programs routinely passed by the U.S. Congress on the presumption that they reflect rigorous research have no real scientific basis.

Theoretically, of course, there are reasons other than the desire to please a government funder that a particular experiment might not be reproducible. These include researcher incompetence, sloppiness, damaged equipment, the psychological pressure to find positive results, and even the hope that a faked result will someday be proved correct.

It is also true that not all social science, education, and medical studies are done for some state or federal agency. Non-profits, pharma, and other companies, professional associations, and public employee unions also sponsor university research, although their dependence on politicians for regulatory approvals, tax breaks, member licensure, employee benefits, and other operating advantages does not quite make them ideologically neutral.

But perhaps the best indicator of how the experimental irreproducibility crisis threatens secular progressivism is how lopsided the political response to solving it has been. In February, the Independence Institute, a prominent free-market think tank, sponsored the first national conference on “Practical Solutions for the Irreproducibility Crisis.” Among its seemingly commonsense recommendations were to spend more money on replication studies, prioritize grants to researchers who pre-register their protocols and meet new best-practices standards, and require government-funded experimenters to make their data and research protocols publicly available.

The leftist social media response to these suggestions has been almost hysterical, labeling them everything from misogynistic to white supremacist to fronts for climate change denialism. Two graduate students set to speak at the Independent Institute conference were forced to withdraw out of concern for career sabotage, according to organizers.

Such attacks continue, even though the authors of the most detailed expose of academic bias, “The Irreproducibility Crisis of Modern Science,” went out of their way to show that some leftist causes, such as reversing climate change, could benefit from stricter research standards. While “the short-term thrust of [experimental reproducibility] reforms may seem to favor the political agendas of American conservatives,” write David Randall and Christopher Welser, “every American who cherishes the scientific pursuit of truth should seek to solve the problems that beset contemporary science.”

What the left undoubtedly fears from the effort to end funder bias is an intellectual setback far more damaging than just losing the ability to affect legislation. “Not all irreproducible research is progressive advocacy,” as NAS president Wood puts it, and “not all progressive advocacy is irreproducible, but the intersection between the two is very large.”

The good news is that experimental reproducibility is so fundamental to science that no ideology or self-serving relationship between government and the academy can prevent past studies from eventually being rerun under more rigorous conditions. As a 2017 Nature editorial put it, “Replication studies offer much more than technical details. They demonstrate the practice of science at its best.”

Neither is it possible for new studies to remain as corrupted by self-interest as they have previously been. Indeed, leaders in the field of psychology are encouraging study authors to include suggestions on how to reproduce their results as well as to publish their protocols. This is why the biggest cultural change in the coming years could well be inspired, not by political activists or party platforms, but by the pages of the latest scientific journals.

Dr. Andrews was executive director of the Yankee Institute for Public Policy from 1999 to 2009. He is the author of “Living Spiritually in the Material World” (Fidelis Books).

Copyright © 2020 The Federalist, a wholly independent division of FDRLST Media, All Rights Reserved.