Skip to content

Using Big Data to transform the poverty conversation: a Small Area Estimation approach

Social protection programs and appropriate policy are impactful drivers of poverty reduction but they need up-to-date, comprehensive and accurate data in order to effectively tackle the causes of poverty. Here we talk about advanced statistical techniques using Big Data to generate granular poverty estimates at a much lower cost and more frequently than ever before, enabling more dynamic conversations to help end extreme poverty.

Current methods and analysis techniques often used in poverty reduction policies rely on expensive surveys and infrequent census data. This means that policy discussions can often lack dynamism, and might be at risk of sliding off the political agenda between census years.

Also, current poverty estimation and analysis often lack granularity, with a focus  frequently  at an ‘urban vs rural’ level rather than comparing districts or even villages. This can result in public resources being inefficiently deployed rather than being tailored for  the specific  area that needs it most.

Thanks to new estimation techniques, harnessing Big Data, our ‘small area estimation’ method allows policy makers to generate more frequent, granular poverty estimates. This provides country-wide statistics at a fraction of the cost of a census, lowering costs for governments, NGOs and think tanks. It can also help to determine how current policies are impacting poverty rates and therefore guide future policy.

What is small area estimation?

Small area estimation is a statistical modelling technique. It allows a user to estimate data for small sub-populations that are not necessarily included in a household survey. It does this by using common covariates (a variable related to the dependent variable being studied) from a different dataset.

Small area estimation is not a new technique. Second paragraph: Small area estimation is not a new technique. Our guide illustrates the use of the Fay-Herriot model, which was first published in 1979. However, in the past, it has always used a combination of sample surveys and population censuses, resulting in researchers having to wait 10 years for the next census in order to generate the succeeding poverty map.

Now, using our innovative estimation technique, we can use granular geospatial data as an alternative to population censuses, enabling us to:

  • Generate poverty estimates in and out of sample areas;
  • See the spatial distribution of poverty across the country; and
  • Generate poverty estimates much more frequently.

This new development has been made possible by technological improvements such as high-speed computers, as well as key innovations in modelling tools and data integration which allow analysis between geospatial and household data.

DEEP’s guide: an 8-step procedure

DEEP’s SAE guide provides an 8-step procedure which outlines the whole process of small area estimation using geospatial data, from sourcing the geospatial data online to producing poverty maps like that of Bangladesh in Figure 1.

Figure 1

The first step is to gather the geospatial data and harmonize the data into a single rasterized dataset. We can then merge the geospatial and household datasets and fit the Fay-Herriot model, carrying out the necessary diagnostic checks. After running the model, the results are visualized on a country-wide map.

The geospatial data DEEP uses include accessibility data, demographic maps, nightlights data and various datasets from WorldPop. This provides over 150 geospatial covariates which give us sufficient explanatory power for out-of-sample prediction.

An example: Comparing methods to estimate poverty in Bangladesh

In Figure 1, there are two maps showing poverty in Bangladesh. The variable illustrated reflects the likelihood that a household in the upazila (a small administrative region) is among the poorest 20% in Bangladesh.

The graphic on the left shows data we used from the DHS (Demographic and Health Survey). The grey areas indicate areas not covered by this data. We found that 157 out of 522 upazilas were not included – more than 30% of the regions.

For the graphic on the right, we used our small area estimation model to estimate poverty. This provides poverty estimates for all regions, which may be especially significant because we believe that areas not covered by the initial data, like the Chittagong Hill tracts, may be disproportionately disadvantaged.

At DEEP we intend to repeat this process on our other focus countries to learn if we can identify any common characteristics. Furthermore, we are producing time-series poverty distribution maps in Bangladesh to observe which areas are responding best to current policies and which areas might be lagging behind.

Adding momentum to the poverty reduction conversation

By using Big Data, this innovative small area estimation method adds another key layer of analysis to a normal household survey. We can robustly extrapolate results to generate poverty estimates in small regions across the whole country.

It could influence and improve policy evaluation and social protection programs. Frequently updated, granular poverty estimates can reveal which areas respond most positively to different policy measures, enabling governments to make services more bespoke to the area in need.

Furthermore, the public availability of geospatial data ensures that this method is available to a wider audience of agencies. This should add valuable momentum to the poverty reduction conversation.

The method also has academic benefits, since  it:

  • Closes data gaps by producing policy estimates for previously hard-to-survey areas.
  • Integrates data by combining geospatial and household data.
  • Improves the precision of estimates compared to previous methods.

Research from UC Berkeley has found using geospatial data in small-area estimation to be very precise. When compared to ground-truth data in Togo, the small area estimation explained 84% of the variation in wealth (Blumenstock et al., 2022).

A real world purpose

High-frequency mapping changes the policy landscape and will bring dynamism to the poverty conversation.

Our guide will not just serve as an interesting academic exercise in the precision of new big data techniques and data integration but should serve a real-world purpose of helping to alleviate poverty.

If you’d like to learn more about the small area estimation technique, our guide, big data, or anything else related to this innovative method, please get in touch with our team.


Related Updates