It is unfair to blame survey methodology when results disagree with a view point

Source: The post is based on the article “It is unfair to blame survey methodology when results disagree with a view point” published in The Indian Express on 12^th July 2023.

Syllabus: GS 3 – Growth & Development, Planning

Relevance: concerns associated with various surveys conducted in India

News: In a recent article titled “The sample is wrong,” Shamika Ravi, a member of the PM’s Economic Advisory Council, expressed concerns about national surveys in India. However, it is not a correct picture.

What concerns were highlighted by Shamika Ravi?

Under-Representation: She argued that indicators used by the National Sample Survey (NSS) and the World Health Organization (WHO) have failed to capture the true progress of India, and they express concerns about the deterioration of the official statistical system.

The overestimation of the rural population in national surveys conducted by NSS and NFHS (National Family Health Survey) is cited as an example of this shortcoming.

However, the national data gathering system has gone to great lengths to make the samples statistically representative.

For instance, separate samples are drawn for rural and urban areas, and any excluded geographical regions are clearly specified for reasons beyond the organization’s control.

Moreover, the population estimates derived from the NSS have consistently been lower than the numbers reported by the Census or its projected figures.

Although this disparity has been investigated in the past, no specific faults in the survey methodology have been identified. 

Share of the Rural Population: Shamika Ravi further argued that the share of rural population estimated by NSS/NFHS post 2011 Census is much higher than the projected share of rural population from the Census.

However, population projection is based on assumptions on fertility, mortality and migration using appropriate statistical models.

The urban population is projected using the urban-rural growth differential (URGD) from the period 2001-2011, which implicitly takes into account increases both in statutory and Census towns.

Moreover, the primary objective of the NSS and other survey agencies is to generate statistically valid estimates separately for rural and urban areas.

They typically report percentage values rather than absolute numbers across different categories. Their methodologies are not designed to estimate the population size in the survey year.

Conservative in Urban Projections: Shamika Ravi argues that the Census has been conservative in its urban projections as India achieved the urbanisation level projected for 2016 in 2011.

However, using the method of URGD, even the United Nations (UN) has projected a much higher share of urban population in India.

It is also worth noting that the last Census data is over 13 years old, and despite several representations, the Registrar General of India (RGI) has not made efforts to reassess its urban projections.

Response Deficits: Shamika Ravi observes a relationship between response deficits in surveys and wealth levels. It is true that household surveys are becoming difficult due to non-cooperation of households and non-access to richer households in urban areas.

However, the NSS utilizes a substitution method for non-cooperating households, ensuring a near 100 percent response rate. Hence, the observation that the response rate of men falls significantly with wealth needs investigation.

What is the way ahead?

Sampling theory and practice is a scientific discipline. Even some of the surveys conducted by MOSPI have also been found to be of indifferent quality due to engagement of outside agencies.

Hence, it is unjust to solely blame the survey methodology when survey estimates diverge from a particular narrative.