Skip to main content

Center for Global Democracy

 

  2. Quality Control

Quality Control

LAPOP Lab has long been a pioneer in innovations in survey research. From its development and early use of cognitive interviewing to its attention to cross-country and within-country nuances in the use of language, LAPOP Lab has consistently striven to maximize the comprehensibility of its survey questions, even among the poorest and least well-educated populations. Our ability to innovate has increased exponentially due to the widespread availability of relatively inexpensive connected tablets and smartphones that allow for real-time in-the-field downloads of questionnaire revisions, precise geo-location of interviews using GPS technology, and real-time uploads of completed interviews to cloud web services. Once there, they become simultaneously available to field offices and LAPOP Lab at Vanderbilt and immediate audits of the quality of each interview can take place. In the 2016/17 round of the AmericasBarometer, we made significant advances in the use of electronic devices for data collection in the field. Handheld Android mobile devices were deployed for data collection in all AmericasBarometer countries in Latin America and the Caribbean; as always, the U.S. and Canada studies were conducted via the internet because of the extraordinary costs of face-to-face interviews in those countries. With the exception of Haiti, where approximately 50% of interviews were conducted using paper questionnaires, all surveys were conducted using mobile connected devices. In the 2016/17 round, we used the SurveyToGo© (STG) software (app and web-based platform) to assign interviewers to precise locations based on each country's sample design, to manage and control respondent selection and track non-response and to conduct field interviews using response range controls and skips to maximize response integrity. The use of electronic devices for interviews and data entry in the field reduces data entry errors, supports one-click switching between multiple languages in multi-lingual environments, and permits quality control teams to track the progress of fieldwork on a daily basis, by monitoring the location of interviews, the correct reading of questions, and the timing of the interviews. As part of our integrated package of controls we call our Fieldwork Algorithm for LAPOP Lab Control over survey Operations and Norms (FALCON©), we have developed a series of technological advances that improve the quality of AmericasBarometer data in real time. While quality control has traditionally been performed post-hoc, after the fieldwork is complete, LAPOP Lab monitors and corrects problems as they emerge while the survey is still in the field. For a detailed assessment of the gains in quality control that come from this approach, see Assessing and Improving Interview Quality in the 2016/17 AmericasBarometer . See below for detailed information about several of these advances.

Geo-fence Module

LAPOP Lab’s Geo-fence Module helps maintain the integrity of the sample design by ensuring that interviewers are in the assigned work area. This is done through the creation of a series of electronic circles that are placed around selected census segments or municipalities (which are the local level geographic areas that LAPOP Lab typically uses as Primary Sampling Units). We use three elements to build these geo-fences: shapefiles, centroids, and radio.

LAPOP Lab obtains shapefiles (electronic files that store locations, shapes, and attributes of geographic areas in the form of polygons) from census bureaus and/or open source websites. With these files, we use the ArcGIS program to calculate the GPS coordinates for each polygon’s centroid. We then draw circumferences around the selected municipalities or, ideally, if the shapefiles are available, census segments. The Geo-fencing Module then instantly informs interviewers if they are outside the boundaries of the fence, and flags the quality control team. All interviews conducted beyond the fences, through the use of a LAPOP Lab-developed algorithm are programmed into the data collection software. As soon as each interview is completed, the results upload automatically, without interviewer intervention, to a cloud server via the mobile device data signal or, if available, a Wi-Fi signal. Through this process, supervisors and interviewers are quickly notified if the interviews took place in the wrong location, and appropriate steps can be taken to correct this problem.

Distance Audit Module

LAPOP Lab’s Distance Audit Module (DAM) allows us to assess interviewers’ distance from the bounds of the geo-fence. Once each interview is uploaded, the team monitoring fieldwork is able to instantly determine whether an interview was carried out in the correct place, and if not, whether the violation was major or minor. The DAM returns the distance in fractions of kilometers between the interviewer’s location and the closest point of the circumference around the census segment or municipality (i.e., the limit of the geo-fence). A negative number indicates that the interview was carried out within the fence’s boundaries. A positive number means a wrong location. 

Location Consistency Check

Some location errors occur by mistake, with interviewers carrying out interviews in the wrong place because–for example–the selected neighborhood shares a name with an area in a different location within the same country. LAPOP Lab’s Location Consistency Check (LCC) ensures that interviewers are in the right location before an interview takes place. The LCC works as follows: Fieldwork supervisors assign interviewers to work areas (such as a census segment), and the interviewer is informed of that selection. Before starting data collection on any given day, the enumerator affirms their location for the selected area. To do so, the interviewer selects the Primary Sampling Unit (municipality or neighborhood) in which they believe they are located. If they select an area that does not correspond to the selected sample location, the software immediately informs the interviewer of the problem so that it can be corrected. The LCC provides another check, which in conjunction with the Geo-Fence, helps ensure that interviewers collect data from the location selected in the sample and not from another community with an identical or similar name. 

Multi-Tiered Auditing

LAPOP Lab implements a multi-tiered quality control process for fieldwork. In addition to the above checks, we program the SurveyToGo software to silently record a subset of items over the course of each interview (all respondents are informed before the interview begins for the purpose of quality control snippets of the enumerator’s work will be recorded). Fieldwork teams listen to the recordings from 100% of interviews to ensure that enumerators adhere to best survey practices. Auditors take note of and record in the web-based version of the program, the number and nature of errors using LAPOP Lab’s Quality Assurance Chapter (QuAC). Interviews with many errors, based on a scoring system we have developed, are canceled and then replaced by field teams. A second-level quality control team at LAPOP Lab listens to a random subset of these recordings to ensure the quality of interviews and the quality of field teams’ checks. This system allows us to provide quality feedback to interviewers and field supervisors as the survey proceeds in real-time, thus correcting errors in study implementation early in fieldwork, canceling and replacing low-quality interviews, and giving appropriate recognition to high-quality work.