Urban Water and Sanitation Survey Dataset

Background: the global problem of inadequate access to water and sanitation for the urban poor

Over the past two decades most developing countries have experienced rapid urbanisation as part of a global trend that has brought millions out of poverty, and helped to bridge the gap between the developed and the developing world. However, many of the people migrating to urban environments are concentrated in low-income informal settlements (commonly referred to as ‘slums’), either within the central city or in peri-urban districts at the city’s ever-growing periphery. Since 1990, an estimated 171 million more people are living in such settlements. This number will only increase over time, with Africa’s urban population predicted to triple to 1.23 billion people by 2050.

Faced with this trend of mass urban migration, governments and utilities are proving unable to expand their water and sanitation systems at sufficient pace to respond to the increasing consumer need. This is due to a complex web of constraints that characterise the urban water, sanitation and hygiene (WASH) sector in many developing countries. These constraints include: limited institutional and government capacity at the national and city level; lack of technical expertise to serve low-income areas effectively; lack of the necessary fiscal and tariff-setting autonomy for mandated service providers to fulfil their role; unsupportive and/or poorly defined institutional frameworks; insufficient levels of government investment in water and sanitation; poorly targeted international financial institutions (IFI) investments; and weak implementation of existing national policies and strategies.

Without functional WASH sectors that can deliver new service delivery models at scale, huge numbers of people living in slums will continue to remain without access to the most basic services. Diseases related to inadequate WASH remain among the world’s most serious public health problems, and the associated impacts on economic productivity and children’s cognitive development are likely to have profoundly negative impacts on national development.

Water and Sanitation for the Urban Poor (WSUP)

WSUP is a not-for-profit company that helps transform cities to benefit the millions who lack access to water and sanitation. WSUP believes that access to safe and affordable water, improved sanitation and improved hygiene practices underpin poverty reduction through impacts on health, education and livelihoods. This is coupled with a conviction in two core principles. Firstly, WSUP believes in the transformative power of markets and innovation: enormous strides can be made by bringing the local and international private sector into the urban WASH space, by driving business thinking in low-income service provision, and by viewing low-income people as consumers who make choices. Secondly, WSUP understands that the dynamising power of markets can only partially solve the problem: deep political commitment, institutional change, and a functioning WASH sector are equally critical. Sustainable at-scale progress in urban WASH will only be achieved if political leaders prioritise government investment and policy change, and if WASH service providers are assisted to radically enhance their capacity to deliver effectively on that investment.

Valid International

Valid International (Valid) is a limited company registered in the UK. The company was founded by Drs Steve Collins and Alistair Hallam in 1999 to improve the quality, impact and accountability of global health and nutrition interventions. Valid pioneered the Community-based Management of Acute Malnutrition (CMAM) model for addressing Severe Acute Malnutrition (SAM) at scale as well as the local production of Ready to Use Therapeutic Food (RUTF), and was the central force behind the advocacy leading to the endorsement of the model by the UN and multiple national governments in 2007. To date, the approach has been adopted in over 60 countries worldwide.

Over the past ten years, Valid International has been the primary provider of technical services for the set-up, training and monitoring and evaluation of CMAM programmes all over the world. As part of this, Valid International has developed various tools and methods that facilitate the provision of these services. For CMAM setup and training, Valid has published a field manual for CMAM¹ that gives detailed guidance for the set up and implementation of CMAM. In addition, Valid has supported the development of CMAM training modules with support and funding from FANTA².

Valid International’s assessment arm Valid Measures started work 15 years ago to address the urgent need for a survey method that could accurately measure the coverage of selective feeding programmes such as CMAM. We have now developed several innovative approaches for needs assessment and monitoring and evaluation of the coverage and effectiveness of nutrition interventions, including the Simple Spatial Survey Method (S3M), Rapid Assessment Method (RAM), Semi-quantitative Evaluation of Access and Coverage (SQUEAC), and Simplified Lot Quality Assurance Sampling Evaluation of Access and Coverage (SLEAC).³ Taken together, these approaches have improved assessment outputs over the standard cluster sample approach used in health and nutrition surveys, including improved precision of results, speed of data collection, understanding of barriers and boosters to service access, spatially even distribution of data collection, and detailed spatial mapping of indicator results.

Whist this work was initially focussed on the measurement of coverage of feeding interventions it has now expanded to assess multiple indicator sets, making the methodologies suitable for evaluation of a broad range of multi-sectoral interventions and practices including the assessment of WASH indicators.

Background to the citywide surveys

In order to measure sustained universal coverage in urban areas of WSUP programme countries⁴, WSUP will be conducting citywide surveys in one selected city for each of our six programme countries in early 2017 and early 2020. These citywide surveys will collect data allowing service levels across the entire city to be characterised, while also allowing more detailed data to be collected in areas of the city of particular interest to WSUP. These surveys are intended to generate useful information for others working in the sector including the municipality, regulator and local utilities.

In support of these citywide surveys, Valid International has been contracted to provide technical support with the design of the citywide surveys, with particular focus on the technical design of variable density sampling approaches that enable a general characterisation of the entire city, while allowing higher-resolution data to be obtained in areas of specific WSUP interest. This should be achieved while minimising the sample size as far as possible provided that it retains the ability to generate statistically significant conclusions.

WSUP information requirements and other survey design considerations

WSUP would like to be able to report findings that are representative for three specific populations of interest:
- The general city-wide population;
- Low income/poor population within the city; and,
- WSUP-identified population of interest⁵.
As such, the design will ideally have 3 sampling frames (one for each population of interest) with each having their own sample size requirement so as to be able to report estimates of the various water and sanitation indicators. The sample size requirement (see section on sample size) in each sampling frame would be the same so at the minimum, this approach will require triple the sample size from doing a survey of only one population of interest.

The specified populations of interest also provide some challenges that need to be considered when designing the survey. These challenges pertain to identification of these populations and determining where they can be located or found within the city. For low income/poor populations, the first issue to address is who to consider as having low-income or as being poor given that there are a multitude of concepts and metrics used to define this categorisation (e.g., income-based, asset-based, multi-dimensional poverty, etc.) and varying levels of available data that can provide information to determine who is low income/poor. In addition to knowing who is to be considered low income or poor, it would be important to know which areas of the selected city are considered low income / poor areas. Ideally, this should be presented in a map of the city indicating generally (or specifically) where areas of low income / poverty can be located. This can usually be done using map outputs of work done by other researchers on poverty mapping. However, most of this work on poverty mapping have results of low resolution (i.e., poverty is mapped down to district level only) and rarely present poverty mapping at smaller units of a city.

As for WSUP-identified populations of interest specifically populations who live in areas within the city where WSUP provides interventions, the challenges are similar in identifying who these populations are and determining the area in the city where they can be located. Whilst locating the general areas where beneficiaries of WSUP interventions live can be relatively simple, knowing the full and specific extent of where they are located in the city can be difficult given that they may not necessarily fall neatly into known administrative units/boundaries.

An additional issue with having a focus on WSUP-supported areas is that it has additional sample size requirements especially if the purpose of drilling down on these focused areas is to report on experience and satisfaction of users of WSUP services and interventions (which is what is indicated in our preliminary review of the questionnaire) specifically. We think that to get such information requires a different approach to data collection and cannot be easily nested within a bigger cross- sectional requirements without complicating the whole process with .
The type of indicators that WSUP would like to assess through the survey and the questionnaire that is to be used to collect information to calculate these indicators will also influence to some extent the design of the survey. Hence, they are factors to consider at design phase.

From the current questionnaire that we have seen, the target respondent of the survey are households with the head of the household present during survey time as the key person of interest to provide answers to the questions. So, this is primarily a household survey.

Within the questionnaire, some form of household roster is documented such that the head of the household is asked about the other members of the household and specifically their ages and/or their age groupings, their gender and other specific categorisation / groupings of interest such as persons with disability, pregnant women⁶, and in some cases persons living with HIV/AIDS (PLWHA). This is not a standard household roster per se as the data collected on the members of the household are still attributed to the household itself (the data on the members are individual variables on the household data itself) rather than a separate data branch from the household data (which is what is typical of a household roster). From our review, it seems that the data from household members are used to report indicator results of access and coverage to water and sanitation services specific to the different categories of interest. Such kind of reporting has sample size implications but as of yet it is not clear whether this is indeed what the purpose of the household roster is. So, for the purpose of this design document, we focus on the idea that this survey is targeting households and reporting results on households only rather than different categories of interest to which members of the household belong to.

The questionnaire also indicate that most likely majority of the indicators that WSUP needs to be reported are proportion-type of indicators and mostly based on respondents’ self-report with most questions requiring a yes/no answer whilst some require responding based on a scale⁷ which are used for eliciting satisfaction from services that accessed and/or provided. The questionnaire doesn’t require measurements to obtain data (numerical data) but a module on poverty assessment⁸ uses a numeric scoring scale to produced/report an index score for a household that is then used to assess the level of wealth/poverty of the household. Given that majority of the indicators are proportion- type, sample size calculations will take this into account.

Finally, it should be noted that by nature, indicators on access to water and sanitation services are highly clustered. That is, source of drinking water or type of sanitation facility tend to be delivered or provided in a similar way among groups or sets of households specifically those living within a block or a neighbourhood. Hence, responses taken from households that live near each other or on the same street or neighbourhood will most likely be similar if not totally identical. This is the nature of a water and sanitation questionnaire and their associated indicators in a cross-sectional survey and should be factored in when estimating sample size.
WSUP would like to have as much spatial disaggregation of the results across the city as possible as this will provide more nuanced information for the organisation with regard to variances in service provision and access throughout the city which can in turn help guide programme development, beneficiary targeting and programme implementation. At the minimum, WSUP would like to be able to report results specific to the three main populations of interest mentioned above and ideally have an even finer breakdown of results across the low-income/poor groupings and the WSUP-supported communities.

This requirement has implications on the sample size (both overall and at each level of disaggregation) hence should be taken into account. The general rule is, the finer the resolution of results needed, the higher the sample size required.

Survey design

Based on the considerations discussed above, we propose the following overall design that can be applicable to all of the six countries that WSUP plans to survey.

Sample Universe

The survey is to take place at household level. There are three populations to be surveyed:

A representative sample from across all of the city, to estimate WASH services coverage (e.g. indicators like percentage of the population using an improved, non-shared sanitation facility)
A representative sample from identified low-income/poor areas of the city, assessing the same indicators
A representative sample from one or more WSUP-identified areas of interest, either for previous work locations or proposed future locations, again assessing the same indicators

Sample frame

We propose a sampling frame with at least three levels of spatial stratification across the city and a two-stage spatial sampling design within each of the spatial strata.

The three levels of spatial stratification that we propose are:

A first level sub-division of the city – this would usually be a formally recognised and/or official sub-division of the city along known or set administrative boundaries. However, it should be noted that formally recognised and/or official sub-divisions of a city may or may not always include urban sprawl or peri-urban areas or conurbations that extend the city.

These areas are usually a result of influx of people from rural areas and other areas of the country to the city who form communities around the borders of the city and/or in specific areas within the city. The people who form these communities tend to be low-income/poor. Hence, it would be important that further investigation be made whether formally recognised borders of the city include these conurbations and are part of the first level sub- divisions of the city that will be used for the sampling. If they are not, then a decision has to be made whether the areas that have not been included fall under WSUP’s other populations of interest such as the low-income/poor (category b above) and/or those that receive or benefit from WSUP interventions (category c above). If these areas are deemed important to include in the sample universe, then these areas are added to the sample universe either as a separate sub-division within the city⁹ or included within the existing sub-divisions of the city.

The rationale for having this first level of stratification is to serve as a unit of disaggregation to which the survey will be designed to be able to report indicator estimates on. This, therefore, has an impact on sample size as the more the first level sub-divisions there are, the higher the overall sample size needed (see sample size section for detail). This would mean that in some cases for cities with numerous first-level sub-divisions, some kind of grouping of first-level sub-divisions would be necessary so as balance the need for spatial disaggregation of results with the level of resources available for the surveys.
A second grouping based on low-income/poor stratification – this would address WSUP’s requirement of reporting results specific to the low-income/poor population of the city. In this section, we refer to low-income/poor stratification as a geographical grouping based on areas that are known to be low-income/poor. This information should be based on known or established concepts of low-income/poor which have been mapped and areas and boundaries of which are well-defined. These maybe available for some cities (as described in the section on design considerations).

It is possible that the first level sub-division stratification described in point 1 above overlaps with the second stratification on low-income/poor. As mentioned, there maybe specific first level sub-divisions that include certain first level units being classified or categorised as low-income/poor¹⁰. If so, then the first level and second level stratification proposed here will be the same as it already includes a grouping based on low-income/poor status. On the other hand, these areas can be smaller units interspersed across the first level sub-divisions (i.e., slums). In this case, we will need to define these areas separately as a second-level sub-division within the first-level units.

It is also important to clarify that this stratification on low-income/poor does not necessarily imply that all respondents within areas that are considered low-income/poor are indeed low- income/poor (based on actual metrics of poverty such as wealth index, etc.) and the same for areas that are classified as not low-income/poor. Variations in wealth status at the household level can still vary within these areas and this variation will be captured through the survey itself. What this geographical stratification on the basis of low-income/poor does is it allows us to provide a specific sampling frame and hence a specific dataset for this area to be able to provide more robust disaggregated spatial results. This is based on the recognition that these areas of low-income/poor have more likely very different characteristics compared to the more formal and organised areas that are not considered low-income/poor. Such characteristics (e.g., housing structure, available services, etc) would make these areas function in a different way than others and the households within them live in a different way than other households in other areas. Capturing this variation specific to this area would therefore be important particularly in an urban-setting survey where clustering is very common.
The third grouping are WSUP-identified areas of interest either based on previous work locations or proposed future locations. These are even more localised areas and are often not clearly delineated geographically. This will require work to delineate and map these areas as clearly as possible so that they can server as third level sub-divisions with their own sampling frame.

It should be noted that this third-level of sub-division will require additional sample size which should be considered weighing in the desirability of having results specific to WSUP-identified areas of interest with resources available.

Each of these levels of spatial stratification will have their own sampling frames and as such will require their own sample sizes.

Sample Size

The following sample size calculations are for each spatial strata described above which will address WSUP’s requirements for spatially disaggregated results based on defined sets of populations to report on.

Sample size for estimating proportion-type indicators Based on the nature of the indicators that WSUP requires the survey to report on, the sample size calculations should be powered to be able to report proportion-type of estimates. In addition, sample size required for the survey depends on:

Precision of the estimate required (as determined by the width of the confidence interval around the estimates); and,
Variance inflating factor (VIF) or design effect (DEFF) for the chosen design (e.g., simple random sample, cluster sample, etc.), reflecting the increase in sample required to use a cluster design. The general formula to calculate sample sizes to estimate a proportion (sometimes called prevalence) indicator is:

\[ n = z ^ 2 \times{\frac{p(1-p)}{c ^ 2}} \]

where

\(n = \text{sample size}\)

\(z = \text{z-value for} ~ 95\% ~ \text{confidence interval}\)

\(p = \text{expected proportion/prevalence}\)

\(c = \text{level of precision}\)

To be able to use this formula, we set out the following parameters specific to the survey requirements and the sample frame described above:

\(z = 1.96 ~ \text{(for} ~ 95\% ~ \text{CI)}\)

\(p = 50\% ~ \text{(assume proportion requiring highest sample size)}\)

\(c = 0.10 ~ \text{(} 10\% ~ \text{precision)}\)

This gives us the following sample size:

\[ n = 1.96 ^ 2 \times{\frac{0.5(1 - 0.5)}{0.1 ^ 2}} \approx 96 \]

We therefore need a sample size of \(n = 96\) assuming a simple random sample survey design.

However, given that we will be using a clustered sample design, we need to take into consideration design effect or DEFF. DEFF is related to the indicators being studied as well as the planned sample size per cluster, and is best calculated empirically from previous similar surveys in the area being studied. The best reference survey to use to estimate DEFF from is the Demographic and Health Survey (DHS) which is conducted every 5 years in most countries. In general, DHS includes an urban sample from the capital city and from other main cities of the country which can be used to estimate the intra-cluster coefficient (ICC). The intra-cluster coefficient gives an estimate of how correlated the responses are for specific questions / indicators within a cluster thereby providing a metric for the loss of variance as a function of the cluster survey sample design. The purpose of DEFF is to inflate the base sample size based on a simple random sample design so as to increase the variance of the survey sample. This counteracts the clustering that happens with a cluster sample design and that is created by the type of indicators being surveyed (such as water and sanitation indicators). DEFF can be calculated using the following formula:

\[ DEFF = 1 + (c - 1) \times \rho \]

where

\(c = \text{cluster size}\)

\(\rho = \text{intracluster coefficient (ICC)}\)

This equation shows that DEFF increases with increases in cluster size and/or in ICC. Since there is very little we can do to change ICC, the general idea will be to reduce cluster size as much as possible as this will bring down DEFF. The smaller the cluster size, the more that the survey design approximates a simple random sample.

Studies that have estimated DEFF using survey datasets from various countries have shown that DEFF for water and sanitation indicators can range from as low as 1.5 to as high as 7. DEFF should therefore be estimated per country whenever possible so that a more precise sample size estimate can be calculated.

Once DEFF has been estimated, the base sample size of \(n = 96\) can be adjusted as follows:

\[ n_{adjusted} = n_{base} \times DEFF = 96 \times DEFF \]

Sample size for classifying proportion indicators Another way of reporting proportion indicators is by classification, that is determining whether a certain threshold or thresholds is/are reached by the results of the indicator enabling its categorisation into specific classes of achievement (i.e., high or low, success or failure, adequate or inadequate). The approach of classification is called lot quality assurance sampling (LQAS) which is an analytical technique developed and used widely in the industrial field (pharmaceuticals, manufacturing to name a few) as an efficient and cost-effective way of checking and controlling the quality of voluminous products manufactured¹¹. LQAS requires much smaller sample size than the classical estimation approach mainly because of the type of information it provides (classification rather than estimation). From a quality control perspective particularly at an industrial level, a classification result determining whether a specific lot or set of goods produced are of good quality or not is just the information needed to be able to make decisions on whether or not to release the lot of goods to the market and whether an inspection of a specific production line needs to be performed to check its manufacturing process. Such decision-making do not require results of high precision. For example, if a drug company sets its quality control standard at \(80\%\) of medicine in a batch produced meets the quality requirements, then a survey with a sample size aimed at reporting an estimate with acceptable precision (as described above) that gives an estimate of 60% has the same decision-making value as a survey using a much smaller sample size (usually \(n = 40\)) that classifies the batch of medicines as failure (below \(80\%\)). Both sets of result will guide plant managers in determining that the specific batch of medicines are of poor quality hence should not be brought to market. However, the LQAS approach provides this information with about half of the sample size.

The use of this quality control technique has been adapted for use in health monitoring and health surveys, which include water and sanitation indicators, to a great degree because of its small sample size requirements¹² which lends itself well to routine and repeated applications over time and location.

LQAS generally requires a sample size of \(n = 40\) . Adjustments to this sample size are made based on survey design and the cut-offs used for classifying with a cut-off of \(50\%\) requiring the highest sample size (\(n = 40\)). Factoring cluster design, this minimum sample size can go up to \(n = 60\) which is the typical sample size used for LQAS applied to water and sanitation indicators¹³.

Stage 1 sampling

Stage 1 sampling is the the selection of clusters or primary sampling units (PSU) in which the sample of households will be collected from. We propose to select survey clusters using a variable grid spatial sampling approach. Cluster locations will be chosen for each sampling frame (non-slum, slum, and operational area) using centric systematic area sampling (CSAS). This will be performed through the following general steps¹⁴:

Get appropriate maps for the city to be surveyed. The types of maps that will be appropriate for this urban sampling approach are:
1. Map data files¹⁵ that delineate the boundaries of the whole city and each of the three levels of the sampling frame¹⁶;
2. High resolution gridded population maps¹⁷; and,
3. Map data on roads, buildings, residential areas, landuse and places of interest for the city¹⁸.
Using maps with boundaries of the different administrative levels with an overlay of the high resolution gridded population maps, draw a square grid around each spatial strata that has been identified for the city. The size of the grid will be dependent on the number of sampling clusters that has been decided when calculating sample size. The grid should cover only areas with populations as indicated by the population maps.
Get the centroid of each quadrat (square) of the grid. This identifies the primary sampling unit within the specified area where the stage 2 sampling will be conducted. The geographical coordinates of the centroid will be captured and recorded and put on the stage 1 sampling list (and drawn on appropriate maps) for enumerators to use during data collection.

It should be noted that the stage 1 sample is not selected proportional to population size (PPS) which is a stage 1 sample selection approach that is commonly used in most population surveys. Instead the stage 1 sample is drawn spatially through an even square grid across the city. This approach addresses the limitation of a PPS sample which gives a cluster with higher population a higher probability of being included in the sample compared to clusters with smaller populations. Also, a population-weighted stage 1 sample is drawn from a specific sampling frame hence can only be representative of that frame. Any form of disaggregation of that dataset to provide results at finer resolution will not be possible to perform.

For the spatial sampling approach, the dataset can be disaggregated and aggregated accordingly by just applying a cluster weighting approach (using the population at each cluster as the weights) used in analysis of stratified data in estimating the indicators of interest. This is what is called posterior weighting approach. Population estimates obtained from WorldPop remote sensing data can be used for the purpose of weighting the cluster data.

Sampling Stage 2

At each cluster, a total of two households will be selected. The cluster will be identified using GPS coordinates on enumerators’ devices, and the start household will be identified as the building closest to the GPS coordinates. In the case of a multi-household or apartment building, the household will be chosen randomly. To select the second household, we recommend a general approach of choosing from the building or housing structure five doors to the right facing the door of the start household.

It is very likely that the spatial organisation of the various cities to be sampled will be different between each other. Also, within each city, there will be differences in the setting and organisation of buildings and houses in different parts of the city which will require adjustments or variations to the general approach for stage 2 sampling described above specifically for selecting the second household. This is something that will require investigation and testing prior to the start of the survey in every city and experiences documented in the process document produced during and after the survey process. This will help in producing more nuanced guidelines on other approaches to stage 2 sampling.

Ernest Guevarra

2024-04-12