The perils of natural experiments and randomized controlled trials

Synopsis: The RCTs that have swept economics and won Nobel recognition remain rather unreliable in comparison with clinical trials. We should be sceptical about the policy conclusions of randomized control trials used in economics and social sciences.

Introduction

Randomized Clinical Trials have revolutionized medicine and led to many life-saving treatments, drugs and vaccines. But, we should be cautious before applying conclusions based on Randomized Controlled Trials (RCTs) to economics and social science fields.

What are the issues with applying RCTs in economics and social sciences?

i). The fallacy of composition: The human body, the object of clinical trials, is a very complex but generally closed system. The impact of a trial drug on one person is independent of its impact on others.

The results of a randomized clinical trial for, say, a vaccine, after it has gone through the quality checks of three phases, can be safely applied to the whole population.

Not so in the case of a Randomized Controlled Trial (RCT).

This is so because in economics, individuals are part of a complex and open system, their decisions and interactions are dependent upon one another. Thus, at the economy-wide level the impact on one individual may not be extrapolated to be applied on the whole population.

Example:

Rise in minimum wage: A rise in the minimum wage in one district or industry may not have an adverse impact upon employment and costs and prices, since firms invariably have some margin of profits to absorb the hike. But an economy-wide minimum wage rise may push up prices, unless the central bank tightens policy to offset that.

Education subsidy: A certain subsidy leading to more education for some individuals may benefit them, but nationally, it may not.

ii). Accuracy of data: The second vital matter is the accuracy of the data that researchers generate by conducting surveys to conduct their RCTs. In their pioneering 1994 study (for which Card recently got Nobel in Economic Science), David Card and Krueger generated their own data from phone interviews. Given ideological biases, researcher- generated survey data is intrinsically suspect.

What is the way forward?

The academic debate continues. The question to ask is:

Of all the studies using RCTs published in the top 10 economics journals in the past two decades, how many generated their own data versus used publicly-available data? – This would broadly indicate the reliability of non-medical RCTs that have swept the field.

Source: This post is based on the article “The perils of natural experiments and randomized controlled trials” published in Livemint on 26th Oct 2021.