Center for Health Statistics Research

Source: Mathews et al., 1998. Only states with an increase of 150% or higher are included.

Latina workers are widely dispersed throughout the North Carolina economy. Despite the fact that North carolina ranks fifth in the US in number of migrant farm workers (Employment Security commission, 1996), a minority of Latinas work in the primary sector (Johnson-Webb and Johnson, 1996). the proportion of Latina births from migrant farm workers is unknown. However, preliminary analysis of 1995 NC vital records shows that seasonal variations of the number of Latina births is small suggesting that only a minority of Latina births is from seasonal workers.

In states with emerging Latina populations, new efficient sampling techniques must be developed for two main research design options that can be used to study perinatal outcomes in Latina women (see Figure II.1). One option is a prospective cohort design, in which participants are enrolled early in pregnancy and followed through pregnancy, childbirth, and some period into the postpartum period. This option will be called an early-pregnancy prospective design. Latina women in this longitudinal design are sampled and enrolled as early in pregnancy as possible. This way the researcher benefits from gathering important information to explain perinatal outcomes. Follow-up into the postpartum period is done to examine the implications of pregnancy and childbirth on early early childhood development and other health sequelae for the woman. The other research design option is a retrospective cross-sectional design in which selected women are enrolled soon after the birth of the child, and information about the referent pregnancy and childbirth event are then obtained. We refer to this design option as the at-birth retrospective design.

Obtaining a representative sample of Latinas during their postpartum hospital stay might be difficult if the population is dispersed in many hospitals with relatively few Latina births. Sampling pregnant Latinas early in pregnancy is even more challenging because they often attend prenatal care late. In 1995, only 69.1% of women of Mexican origin attended prenatal care during the first semester, compared to 87.1% of non-Hispanic whites (Mathews et al., 1998). Identifying women at their first prenatal visit might thus not be an appropriate way to enter Latina women into a cohort study. Population-based sampling is an alternative that should be considered. Sampling a population including many potentially elusive undocumented immigrants will also require careful thought. Again, emerging populations might be especially difficult to locate and to sample. In summary, new techniques will have to be developed to study the newly established Latina populations in North Carolina, the South East, and other regions of the US.

Numerous researchers have commented on an apparent paradox among Mexican American babies (Guendelman, 1998). There are relatively few low weight (LBW, <2500g) births and infant deaths among Mexican Americans, despite their low socio-economic status. Hispanic of other origins generally have slightly higher LBW rates than Mexican-Americans.

Buekens, et al. (1999) have shown in a previous study that the decreased occurrence of LBW among Mexican Americans is probably due to fewer small preterm (<37 weeks) births, but gestational age is difficult to measure. Errors in last menstrual periods (LMP) more commonly occur among low socio-economic status populations in general (David, 1980; Buekens et al., 1984). Late prenatal care is probably another important factor, with Mexican American women having their initial prenatal care later than non-Hispanic whites (Mathews et al., 1998). Late prenatal care could decrease the usefulness of ultrasounds to estimate gestational age. Language barriers may also contribute to errors in LMP.

Causes for lower rates of preterm delivery among Mexican-Americans are unknown. A Mexican cultural orientation has been suggested to be linked to favorable perinatal outcomes (Scribner, 1996). In one study, a lower level of acculturation was associated with reduced prenatal stress and with a low frequency of preterm delivery (Zambraba et al., 1997). Available data do not suggest low rates of bacterial vaginosis among Hispanics (Goldenberg et al., 1996). Future studies on the Mexican "paradox" of LBW should focus on preterm delivery rather than birthweight, with the goal of identifying the factors that protect this at-risk group from high rates of preterm birth.

Low infant (<1 year) mortality has also been reported among Hispanics. This paradox is also unexplained. One hypothesis is an underreporting of infant deaths and ethnic misclassification in death certificates (Guendelman, 1998). Latina children might not keep throughout childhood the relative health advantage they had at birth. A study performed at age 8-16 months in San Diego county showed that for 26% of the infants, their health status was eroded by social conditions (Guendelman et al., 1995). Compared to non-Hispanic whites, Latina children are less frequently immunized, and are at higher risk for infectious diseases, lead poisoning, poor oral health, and unintentional injuries (Guendelman, 1998). However, such observations are from cross-sectional studies, and cohort studies are not available.

Further studies are needed to understand the nature of the Mexican paradox. Better instruments will have to be developed to study perinatal outcomes (e.g., gestational age) and their determinants. Such instruments should be adapted to the characteristics of emerging Latina populations, and to different levels of acculturation. New designs will be needed to perform cohort studies initiated early in pregnancy and continuing during childhood. Such designs should be applicable in both established and emerging Latina populations.

Sample designs for the (at-birth) retrospective and (early pregnancy) prospective design options would be quite different approaches in perinatal outcomes research aimed at Latina women. The reason for varying approaches to sample selection is due mainly to the differing points in the pregnancy and postpartum periods in which the woman is sampled. Figure II.1 is used to illustrate the distinction. Here we see a visual depiction of the course pregnancy in a woman’s life, from onset of pregnancy, through childbirth, and to three years postpartum. The retrospective design would imply sampling the woman shortly after she has given birth, and then referencing back to the latter stages of her pregnancy, followed by the birth of her child, in gathering the study data. The most plausible sampling option (statistically and operationally) is to first randomly choose those health care facilities (usually hospitals) where they give birth. Since relatively few births occur outside of a hospital, hospital-base sampling may provide sufficient coverage to the sample of "at-birth" women. At that point, these women are relatively accessible to the investigator and most knowledgeable about events that have transpired in childbirth and the latter stages of pregnancy. Screening and data gathering is also feasible at this point since members of the target group are relatively accessible to the investigator’s representatives, although the percent of all births to Latinas is quite low and generally scattered in somewhat varying concentrations among hospitals in North Carolina. Stratified sampling of hospitals by the percent of births to Latinas, with oversampling of those hospitals with higher percentages of Latina births, is needed to increase screening efficiency.

The early-pregnancy prospective design, on the other hand, attempts to capture a sample of women early in pregnancy and then gather study data from the cohort sample intermittently during the rest of their pregnancy and some time thereafter (e.g., three years). Sampling in this research design is made difficult by the fact that one wishes to sample a woman "early" in pregnancy (i.e., the first or second trimester), before or soon after the pregnant woman has first intervened with the health delivery system. Further complicating matters are that: (1) the timing of interventions may vary widely through the course of pregnancy, (2) many and varying types of health care providers may be involved in prenatal care, and (3) multiple encounters with the system may favor discovery of women with relatively frequent encounters (Peoples-Sheps, Kalsbeek, and Siegel, 1988). Given these complications, the best design option may to sample through what we call an "area frame," where a sample of area units (e.g., counties, neighborhoods, etc.) is chosen and then women in the general population are screened for race-ethnicity and pregnancy status, perhaps by telephone, before baseline data collection and follow-up begins. One problem with the area frame is that it may not be optimal for targeting migrant and seasonal farm workers, and other elusive sectors of the Latina population. For instance, sampling MSFWs may best utilize other frame sources (e.g., migrant camps) to link into these more fluid segments of the population of Latina women. Another potential troublesome aspect of using the area frame, and to a lesser extent approaches used to sample the fluid component of the Latina population, is the process inefficiency of screening for Latina ethnicity, as well early pregnancy status. While various sampling steps (e.g., oversampling areas with higher concentrations of Latinas, telephone prescreening) can be taken, the question remains as to the relative cost-efficiency of individual options or mixtures of the options.

The fundamental sampling issue in this project is coverage of sampling frames, reflecting the extent to which a probability sample chosen using that frame represents all segments of the population targeted for study (Lessler and Kalsbeek, 1992). A sampling frame is defined here as the explicit or implicit list or lists to which the randomization-based plan for sample selection is applied in choosing a sample from a targeted population. For example, an at-birth sample of women selected by randomly choosing hospitals, and then selecting all births to Latina women during a randomly chosen sequence of consecutive weeks at sample hospitals, would utilize what might be termed a "hospital-based frame." The primary statistical implication of lack of complete coverage of the frame is bias in estimates derived from the sample with improper coverage. The amount of bias is directly tied to the percent of the population that is not covered by the frame and the amount of difference in key study measures between those covered and those not covered. In a broader sense, the essence of most frame problems is a lack of stability in the linkage between the member sets of the target population and the list or group of hierarchical lists which comprise the frame. Linkage is the key, and thus the issue, in sampling elusive populations where the frame-to-population linkage is tenuous.

For some populations of interest to researchers, particularly those at greatest health risk (e.g., MSFWs, the homeless, etc.), the dimension of sample coverage problems is two-fold. One problem may be that the linkage between entries on the frame and members of the population is not fixed but dynamic in time. For instance, if a sample of the homeless population in a target area is chosen by first sampling from a list of shelters for the homeless, the linkage between a homeless person and the set of shelters may be limited to the day or two that this person is a resident at a shelter. Since randomization applied to the frame is key to determining the statistical likelihood of choosing population members into the sample, tenuous frame-to-population linkage implies uncertainty in establishing the selection probabilities for individual population members and thus a breakdown in the process of making inference from the sample to the target population. In this instance, both the outcome of the sampling randomization and linkage uncertainty determine whether or not the population member becomes part of the sample. Lacking the ability to establish the selection probability for each member of the population, the sample becomes invalid and subsequent estimates biased unless one can establish a suitable model for the linkage between frame and population.

A second coverage implication of sampling high-risk populations is that no single frame source provides adequate coverage; i.e., that no single frame source accounts for the entire target population. For example, a list of homeless shelters available from the local social welfare agency may exclude many of the smaller and newer places for the homeless to spend the night, and even a complete list of overnight shelters might exclude those facilities providing shelter for persons for longer than a few nights. The implication here is that linkage to a frame entry does not exist for some members of the population targeted by the study. The probability of being selected in the sample for those not covered by the frame is zero, thus leaving the uncovered portion of the population unrepresented by any selected sample. If the missed portion of the population have study measures that collectively differ from the portion that is covered, bias will result in sample estimates.

Finding remedies for frame problems often requires the use of multiple frames to deal with the lack of complete coverage for individual frames, and/or working around the dynamic state of frame-to-population linkage in conjunction with the sample selection process. While issues of design and inference from multiple frames have been around for over three decades (see Hartley, 1962 and 1974; Cochran, 1967; Lund, 1968; and Fuller and Burmeister, 1972), most attention in recent years has been on the efficient use of multiple frame sources in conjunction with telephone samples jointly chosen by random digit dialing (RDD) sampling and vendor listings of phone numbers. Lepkowski and Groves (1986), Casady (1989), and Tucker (1989) have investigated general design issues for dual-frame designs, while Biemer (1994) and Brick et al. (1995) have studied various process efficiency issues in conjunction the dual-frame strategy, thereby justifying the widespread use of so-called list-assisted samples in telephone surveys. In more recent general research, Choudry (1989) has provided an optimum allocation solution for dual-frame designs when one frame provides complete though expensive coverage and the study variable is dichotomous, while the weighted estimator proposed by Skinner and Rao (1996) is seen as comparable to those previously proposed by Hartley (1962) and Fuller and Burmeister (1972). Casady et al. (1985) compared a set of alternative dual-frame sample estimators, while Durant and Vitrano (1989) and Traugott, et al. (1993) have explored issues of nonresponse in dual-frame designs.

The challenges of sampling elusive populations has been recognized for some time (Kish, 1965), but increased research interest in these populations has been especially keen in the past two decades. Kalton and Anderson (1986), Sudman et al. (1988), Lepkowski (1991), and Kalton (1993) have provide general reviews of the issue and its remedies; Kish (1988 and 1991) has provided a review of terminology on this topic. Most prominent in the segment of the sampling literature is the number and diversity of elusive populations sampling strategies have been attempted. Table II.3 provides a partial list of applications.

Population	Investigator(s)
The Homeless:	Cowan (1986); Frankel (1986); Iachan (1991, 1992, 1993, and 1994)
Undocumented Aliens	Cheswick (1986)
Migrant Farm Workers	Kalsbeek (1989)
African Nomads	Kalsbeek (1982 and 1986), Kalton (1993)
Ex Heroin Addicts	Biernacki and Waldorf (1981)
Female Sex Workers	Fougier (1996); and Fougier and Sargeant (1997)
Missing Children	Sudman (1986)
The Unemployed	King (1990)

Some of these sampling approaches have utilized the sampling strategy called snowball or network sampling. This approach has been widely used for elusive populations that can be discover through social and familial networks of which they are a part, hence the name (e.g., Fougier, 1996; and Fougier and Sargeant, 1997). The related topic of multiplicity estimators to track the prevalence of rare populations traces its long history to the initial work by Birnbaum and Sirken (1965) and Sirken (1970). It is apparent from the above table and a review of the work done thus far, that sampling of elusive populations is a topic that has generated considerable interest among survey statisticians. It is also apparent from this experience that a definitive general solution to the problem has not been found, since sampling needs are particular to the unique characteristics of specific elusive populations. Thus more work is needed to explore the varying contexts in which these populations appear.

In constructing and testing survey instruments, researchers attempt to reduce the possibility of measurement error at all stages of the question answering process. More simply stated, questionnaire designers work to develop questions which respondents will understand and answer as the researcher intended. Measurement effects can be introduced at any point in the question answering process from comprehension of the question and retrieval of relevant information to making a judgment based on available information and reporting an answer. Problems in these various stages, like comprehension difficulties or problems with retrieving specific types of information, have often been reported in the survey literature, mainly in the context of general population surveys.

However, there is now an emerging literature on the occurrence of measurement problems specific to cross-cultural survey research. Constructing culturally appropriate questionnaires for ethnic groups, like Latinos, can present a various of measurement challenges. Hines (1993) has identified three of these potential measurement challenges. They are:

In this section, we will address each of these problems and include examples of each, where possible, from the survey literature. These problems mainly reflect differences in attempting to measure a population, like Latinos, versus an Anglo-American population, which is very familiar to survey researchers. It is also important to consider an additional factor that can complicate measurement, differences among Latinos in terms of their level of acculturation. We will briefly address this measurement challenge.

First, the researcher and the respondent must share a common understanding of concepts measured in the survey questionnaire. This is the idea of conceptual equivalence. Some concepts may not have the same meaning or any meaning at all across cultures. For example, the Western world's view of depression has no equivalent in many cultures (Hines 1993). In another illustration from Hines, a concept like dependency carries very different connotations in Japanese and American cultures. The Japanese associate dependency with togetherness and interdependence, which are valued and important in their culture. On the other hand, American culture places a higher premium on independence and self-reliance and looks negatively on dependence. These examples underscore the importance of understanding the culture of the population being measured when developing a questionnaire. Without this knowledge and understanding, there is danger that respondents may have different interpretations of the concept of interest, introducing measurement problems. In our health research with Latina women, we will investigate possible cross-cultural differences in concepts that may be of interest to health researchers dealing with Latino populations.

Another possible difficulty in developing questionnaires for Latinos is producing a Spanish translation that will be uniformly understood by respondents, that is, achieving linguistic equivalence. Hines has argued that techniques like back-translation, with the original instrument being translated to the target language and then back to the original language, and decentering have improved translation efforts (Hines 1993). Decentering is a translation approach where the original language version and the target language version are developed and revised together with multiple modifications of the original and target language instruments as needed (Marin & Marin 1991). On the other hand, we think there are still several remaining challenges in producing a Spanish language instrument. There are a number of varieties of Spanish in use from standard Spanish to popular varieties that vary depending on the urban/rural location of the speakers (Elias-Olivares & Farr 1991). Furthermore, some Latinos now speak a form of Spanish containing a number of English words or they may code-switch, switching back and forth between English and Spanish. Another issue is differences in language use between immigrants from different Spanish-speaking countries. Cubans and Puerto Ricans are often characterized as strong proponents of a "pure" form of Spanish, while Mexicans quite freely add borrowed English words into their language (McKay & Aguirre 1994).

There is evidence in the survey literature of a number of these problems. In developing the Spanish version of the redesigned Current Population Survey (CPS), McKay and Aguirre (1994) reported that respondents of Mexican origin had comprehension problems due, in part, to the formal Spanish used in the instrument. For example, respondents of Mexican origin did not always know the formal Spanish term for boarder ("inquilino"). They used the term "rentero" instead. Research on the Census long form with respondents of Mexican origin also has pointed to similar difficulties in understanding the more formal Spanish (Elias-Olivares & Farr 1991). The use of formal Spanish like "alquilar" (to rent), "concepto de contribuciones" (taxes) and "vestibulo" (lobby) were unfamiliar and difficult to interpret for Census respondents. The respondents preferred "rentar" (to rent), "los taxes, las taxas" (taxes), and "corredor" or "pasillo" (lobby). "Grupo racial" (racial group) also caused controversy among a number of respondents who actually interpreted it as "racista" (racist). In addition to the specific words and terms used, respondents had difficulties with the formal, complex phrasing of some questions. Overall, the Mexican respondents suggested the use of less formal Spanish and simpler, more direct questions in the Census forms.

While clearly the Spanish version of an instrument can present comprehension problems, many Spanish speaking respondents may be able to complete questionnaires in English and may encounter difficulties with English words. For example, in testing the Census long form with Latinos, some respondents did not understand phrases like "mobile home" and "plumbing difficulties" (Elias-Olivares & Farr 1991). In CPS cognitive interviews with foreign-born respondents, Wellens (1994) identified "born abroad", "American parent", and "citizen by naturalization" as problematic terms for some of these non-native English speakers. In investigating the measurement of perinatal outcomes and their determinants with surveys of Latina women, we will carefully consider as a project team the form of Spanish that might be used in instruments and potential difficulties with both Spanish and English terms in the questionnaire (for example, "health care provider" and "last menstrual period").

A second set of problems with obtaining accurate survey responses from culturally diverse populations are what Hines termed measurement problems. We will highlight two which she discussed: (1) cultural sensitivity, and (2) unfamiliarity with the use of survey measurement tools, like response scales and closed questions. Addressing the first of these, cultural sensitivity, issues like citizenship status or welfare status may be sensitive to immigrant groups, particularly those here illegally. In testing the CPS nativity questions, Wellens (1994) noted that several foreign-born respondents thought that questions about their immigration and status of citizenship were sensitive. The respondents also said that some people might not report honestly. Highlighting sensitive subjects like sexual behavior, Negy and Woods (1992) have argued that self-reporting may be even more difficult to interpret with Hispanics than Anglos because of the strong conventionality which is frequent in Hispanic culture. This can lead to underreporting of socially undesirable behaviors. There is additional discussion of this problem in Bradburn, Sudman, Blair and Stocking (1978), Francoy, Malloy and Gonzales (1984), and Marin and Marin (1989). In examining panel attrition in a face-to-face survey of Mexican American and non-Hispanic female adolescents, Aneshensel et al. (1989) speculated that one reason for the higher panel attrition of Mexican-born women might have been the topic of the survey, fertility-related attitudes and behaviors. It is unknown whether asking about last menstrual period or other aspects of pregnancy which we might want to explore in our research would be considered sensitive to Latina women. It is an important factor to consider in developing and testing instruments for Latinos, perhaps as suggested by Negy and Woods (1992) even more so than with Anglos.

A second of Hines' measurement problems is lack of experience with response scales or closed questions, which force the respondent to choose from a list of prespecified answers. In Census Bureau research with Latinos, the most common problems across English and Spanish versions of the Census forms were difficulties with the survey task (Elias-Olivares & Farr 1991). More specifically, respondents were unfamiliar with the test-taking skills required to complete a survey, for example, choosing the most appropriate response options when provided with a list of alternatives. The interviewers had to do a considerable amount of coaching to help respondents complete the questionnaire. It was simply not a familiar task to many Latino respondents.

These findings tie to the third key set of problems Hines discussed. These are problems related to the survey interview process. People in the United States are often confronted with surveys from work to school to the shopping mall. However, familiarity with the survey process may be quite different in other cultures. It is important to address how respondents from other cultures may react to the survey request and survey interaction (if an interviewer-administered instrument). Aneshensel et al. (1989) hypothesized that greater panel attrition among Mexican-born adolescents versus U.S.-born Mexicans and Anglos in their panel study might be attributed, in part, to the Mexican-born respondents' lack of exposure to surveys. If respondents do not understand and feel comfortable with the role of the respondent and the interviewer in the interaction, they may not be able to provide accurate answers. The effects of unfamiliarity with the survey process and survey interview have not been extensively studied in the context of culturally diverse populations. Yet, they seem key in understanding the context in which survey measurement occurs. We hope to address them at some point in our work.

The potential measurement problems outlined thus far stem for the most part from differences between Latinos and Anglos. However, measurement may also be affected by differences in acculturation level among Latinos. Acculturation is the modification of values, attitudes and beliefs which occurs with exposure to a new culture (Marin et al. 1987). Clark and Hofsess (1992) describe the development of acculturation models over time. They start from the single continuum model of acculturation. In this model, people move from unacculturated on one end to acculturated at the other end, where all aspects of the old culture ultimately are replaced by the new. The more recent model of acculturation is the multidimensional model. Under this view, some of the immigrants' old cultural values and practices may be replaced by the new culture's (learning the new culture's language) but others, like native foods and music, may not. This process is termed selective acculturation. Clark and Hofsess outline a variety of factors that may influence an individual's process of acculturation. These include individual personality, multicultural environments, and socioeconomic status.

Several researchers have examined the measurement of acculturation in a variety of contexts (Negy & Woods 1992; Clark & Hofsess 1992; Deyo et al. 1985; Cuellar, Harris & Jasso 1980; Marin et al. 1987). Measuring acculturation has often involved items about language use (see Deyo et al. 1985; Cuellar, Harris & Jasso 1980; Marin et al. 1987). Differences in language use among population members is an issue that must be addressed when designing questionnaires for ethnic groups, like Latinos, as discussed earlier. For example, some Latinos may feel comfortable with English and thus be able to complete an English questionnaire. However, some words that may be familiar to a native English speaker may not be to a non-native speaker (mobile home, as mentioned earlier).

Besides language specifically, adoption of values, interactions with people outside the immigrant population, and media exposure may be other indicators of acculturation and can affect measurement. Sensitivity to questionnaire items about sexual attitudes and practices may vary depending on respondents' exposure to the more liberal views and attitudes towards discussing sex in the United States. Comprehension problems with concepts related to real estate and employment, which were found in testing the Census long form (see Elias-Olivares & Farr 1991), may be less common among Latinos who have personal or work interactions with Anglos or those who watch news programming in English. The use of racial categories also may differ by level of acculturation, according to Martin, DeMaio and Campanelli (1990). In a split-ballot experiment, they tested two versions of the Census long form, one asking race first then Hispanic origin and the other asking about Hispanic origin then race. They found that Hispanic respondents born in the United States were more likely to identify their race as "white" if Hispanic origin was asked first. However, there was no effect of the ordering of the race/hispanicity items for Hispanic respondents born outside of the United States. Latinos may also differ in their level of experience and exposure to surveys and standardized forms. This may affect how easily and accurately they can provide an interviewer with survey responses in pre-specified categories. In summary, when examining the potential problems associated with measuring health among Latina women, we will need to carefully consider the effects of level of acculturation on our measurements.

As we have identified, there are a number of possible challenges in obtaining accurate survey responses from culturally diverse populations like Latinos. There are differences between Latinos and Anglo-Americans, which most researchers in the United States are used to surveying, in terms of the equivalence of certain concepts, language use, sensitivity of particular topics and familiarity with the survey task and interaction. In addition, within the Latino population in the United States, there are varying levels of acculturation, making measurement even more complex. This can translate into a variety of measurement difficulties, like comprehension problems or response formatting difficulties. We need to understand and address these issues to ensure the reliability and validity of our survey measurements in ethnic populations like Latinos.

While the causes and particular nature of these problems may be specific to cross-cultural research, the general problems themselves are akin to measurement difficulties, like comprehension problems with terms or reporting on sensitive topics, in general population surveys. Thus, techniques, like cognitive interviewing, used to diagnose and remedy these types of problems in general population surveys may be usefully applied in addressing the measurement challenges in surveying ethnic groups, like Latinos.

The following diverse group of researchers will comprise and support the team that has been assembled to address the variety of sampling and measurement issues that arise in doing health outcomes research among Latina women.

William D. Kalsbeek (SAMPLING TEAM, UNC-CH Biostatistics and Survey Research Unit)

The expertise represented by this team spans the full range of skills needed to identify the substantive research objectives, the corresponding statistical issues, and what remedies might be explored to resolve the issues to meet the research objectives. Dr. Kalsbeek’s research and work experience with sample design, design optimization, and sampling issues in dealing with nomads, migrant and seasonal farm workers, and other elusive populations will enable him to identify the key statistical design issues and find remedies for generating plausible designs for future targeted studies. Dr. Lessler’s broad experience with sample design in some of the Nation’s major surveys will provide excellent support to the sampling parts of this project. Other RTI members of the Center team (Drs. Folsom and Shah) also have broad experience in sample design and will be a useful source of review and comment. On the measurement side, Ms. Ashley Bowers will work jointly with Drs Lessler and Willis in identifying, assessing, and resolving the measurement issues that arise in conjunction with respondent screening and interviewing Latina women about perinatal outcomes. Ms. Bowers has consulted with UNC researchers on questionnaire development for a variety of health surveys and she has led questionnaire development and cognitive testing efforts for two major surveys. Drs. Lessler and Willis have been leaders in the application of cognitive methods to the development of health survey questionnaires and they will bring extensive experience in questionnaire design to the measurement team. Drs. Bennett, Buekens, and Siega-Riz will form a team of experts in perinatal and minority health. They will be supported through consultation from Drs. Guendelman and Bazan-Manson, both acknowledged experts in Latina health. The role of these subject matter experts will be to identify independent variables and perinatal outcomes to be investigated in the population-based studies whose survey designs we will investigate. They will also team with the measurement methods experts on the team to develop and validate drafts of instruments to measure nutritional status, gestational age, and other perinatal outcomes among Latinas.

Several topics of relevance to sampling and sample design emerge from this project. They are as follows:

For each topic, Dr. Kalsbeek has considerable significant experience and has done sufficient prior research to enable him to successfully complete the objectives that are set for the sampling part of this project. He has developed sample designs and done research on sampling issues tied to studies of African nomads and migrant and seasonal farm workers. He has also investigated the development of more realistic models for cost-variance optimum allocation among sampling stages in area household samples, and developed an approach to establish optimal interviewer calling strategies for the National Health Interview Survey. He has also studied the implications of oversampling the uninsured in the National Medical Care Expenditure Survey and has designed a provisional time and space strategy for an at-birth retrospective sample of Latina women for a planned survey of perinatal outcomes in North Carolina. This research experience, combine with his career experience in having designed 45 major survey samples, makes him adequately qualified to be able to recognize and deal with the important sampling issues that will arise in this project.

Dr. Kalsbeek’s first exposure to the issue of sampling elusive populations occurred in conjunction with a series of demographic samples he developed in the early 1980s as part of a USAID contract with the International Population Laboratory (POPLAB) in the Department of Biostatistics. This work appeared as part of the design for a survey in Somalia of infant mortality among women of childbearing age. Since nearly a third of the population of that country were then nomadic, a significant part of the survey design effort was to figure a suitable way to sample the nomadic population. This design study included an examination of design alternatives (Kalsbeek and Cross, 1982), the development an analytic framework for understanding the relative error properties of two design models recommended by a UN commission on the sampling of nomadic populations (Kalsbeek, 1986), and an analytical assessment of the relative cost-efficiency of various adaptations of a basic time and space framework for sampling populations with a dynamic (i.e., "fluid") linkage arrangement between frame and population (Kalsbeek, 1988).

The work on nomads provided the springboard to another design study on the issue of sampling migrant and seasonal farm workers (MSFWs) that Dr. Kalsbeek did as part of a grant from the Millbank fund (Kalsbeek and Parker, 1989). The goals of this design study were to study migrant housing and the spatial and temporal dimensions of movement by this segment of the population and, from this background information, to sketch out the structure and a time and space design strategy for selecting MSFWs. As an outgrowth of this study, the previously developed analytic framework for sampling nomads in time and space (Kalsbeek, 1988) was applied to the problem of sampling MSFWs, although no empirically derived data were available from past well-designed surveys of MSFWs, since there were none from which to draw at the time. It is hoped that the planned study of Latina women paralleling the work in this project will provide that information base. The alternative is to rely on special population supplements to federal surveys (e.g., USDA surveys of the farm population).

Dr. Kalsbeek’s prior work on cost optimization the survey design will also contribute to the methods input that will be needed in the sample part. The work by Kalsbeek, et al. (1983) addressed issues in considering the implications of interviewer travel and call scheduling in the optimum allocation of sample sizes at each stage of selection in face-to-face household surveys. The novelty of this work was the use of spatial movement by interviewers in crafting more plausible cost models for the design optimization. Results showed that relatively simple models could be employed in finding the optimum allocation, but that the estimation of the cost parameters for those models would be complex, depending on assumptions made about movement of the interviewers from home to their work assignments and back. Another cost-optimization strategy was followed in a study by Kalsbeek, Botman, and Massey (1994) aimed at establishing the relative cost-efficiency of various cutoffs in the maximum allowable call attempts that are allowed in calling assignments for interviewers in the National Health Interview Survey. The main implication of this work was a justification for reducing the standard for the number of allowable attempts in the NHIS to less than 15, thus causing a reduction in survey cost but also a reduction in the response rate. While Dr. Kalsbeek has used classical cost-variance optimization rules for many of the several dozen sample designs he has developed in his career, these two studies illustrate his experience with other forms of arriving at the best design strategy while considering both statistical and operational factors.

The issue of disproportionate stratified household sampling to more effectively screen for rare events is another relevant part of the work needed for this project for which Dr. Kalsbeek has prior experience. As part of the design development for the National Medical Care Expenditure Survey, conducted jointly for consortium of federal agencies, including the NCHS, Kalsbeek and Cohen (1978) investigated the relative effects of sampling specifically targeted area with suspected higher concentrations of those with a rare attribute of particular research relevance, the uninsured in NMCES. An important conclusion from this research was that dramatic over-sampling of higher concentration areas could be counter productive in that limited increases in the sample size of the rare attribute group would be more that offset by severe increases in the variance of total population estimates, when (as was the case with the uninsured) the trait did not tend to segregate itself out amongst the members of sample area units. In a comparable line of work, Kalsbeek (1992) has also summarized most of the alternative design strategies that one might consider in sampling African Americans as part of research in dental ecology.

Preliminary Sample Design for Planned At-Birth Retrospective Study of Latina Women in NC

The sample design that follows represents some preliminary thinking by Dr. Kalsbeek concerning an approach to sampling Latina women for a planned at-birth retrospective study. The population addressed in this work will include all women ages 18 or more who give birth in any of 30 central North Carolina counties during two time periods in which sample enrollment and data collection will occur. The two 12-month time periods will be July to the following June, with 12 months dividing the end of the first period and the beginning of the second. The 30 county target area was chosen because of its racial, ethnic, and socioeconomic diversity, and because of its proximity to the UNC-CH campus where the study’s investigators are located. In 1994, there were 2,481 Latina births in the 30 selected counties, which corresponded to 79% of the state’s Latina births.

The plan for sampling women separately during each of the study periods is briefly described as a stratified cluster sample consisting of two sampling steps or "stages" (Kish, 1965; Cochran, 1977). As summarized in Table II.4, the samples of 300 women for each of the two ethnic groups (i.e., L and N) will be independently chosen through a screening process in each of the participating hospital. Thus, 1,200 women will participate in the study, 600 for each of two survey administration periods.

The sample for each 12-month period of the study will be chosen from the same random sample of about 20 participating hospitals that will have been selected prior to data collection in the first period. Selection of the hospitals will be done by a sampling method, called probability proportional to size (PPS) sampling (Kish, 1965; Sarndal, et al., 1992), that is widely used to choose clusters of unequal size in population samples like this one.

The two samples of 300 women for each race/ethnicity group will be chosen as follows: The first sampling stage for each ethnic group sample will involve choosing and securing a commitment to participate from about 20 (out of 77) hospitals in the 30-county area. This step will be performed separately within each of four sampling strata defined as the cross-classification of percentage of Latina births ("high" and "low", with categories based on the percent of the prior year’s births in the hospital to Latina women), and proximity to the geographic center of the 30 county target area ("center" and "fringe"). These percentage figures for hospitals are available from the State Health Department’s statewide provider data system. Separately, within the "high" and "low" categories, the list of hospitals will be ordered (in a spiral fashion from the center out to the fringe of the area) and subdivided in such a way that the total number of Latina births to "center" and "fringe" hospitals for both the "high" and "low" categories of hospitals in the target area is approximately the same. This two-way stratum configuration, defined by the cross-classification of the dichotomous percentage-Latina-births and geographic-proximity stratification variables, will form four sampling strata for hospitals, as indicated below. Grouping of hospitals by percent Latina births and proximity will be done using the cum

rules for optimizing stratum boundaries (Cochran, 1977).

Stratification in the selection of hospitals will be done primarily to facilitate the over-sampling of Latina births. While the number of hospitals chosen in each of these strata will depend on the total number of hospitals and on the number of Latina births in each stratum, disproportionately higher sampling rates will be applied in the selection of hospital from the two "high" strata. Selection of hospitals in this stage will be by PPS and with replacement, with the expected annual number of births to Latinas serving as the size measure. The purpose of stratification, and the use of PPS in this sampling stage, is to over-sample (though not to the exclusion of others) those hospitals with the largest numbers of Latina births coming from counties with the largest numbers of Latina births as well. The final number of sample hospitals will not be known until hospital selection takes place. As regards determining the number of the hospitals to include, we will target a set of participating hospitals with at least 750 expected annual Latina births combined. With the 52 week data collection period for each year of the study covering half of that year and an anticipated 80% participation rate among selected Latina women, achieving a final respondent sample size of 300 Latina women requires 750 = 300/(0.50)(0.80) expected annual Latina births in the final set of participating hospitals.

While hospitals are expected to be receptive to participation in the study, any hospital nonresponse that does occur will be handled by allowing substitution of another comparable hospital within the same county. Substitution for initially chosen hospitals will be allowed to handle hospital nonresponse both within and among years of the study. Although widely viewed as a reasonable way to deal with certain forms of nonresponse (Lessler and Kalsbeek, 1992), the use of substitutes will be carefully controlled. To the extent possible, each replacement hospital will be similar in size and located in the same county as the nonresponding hospital it replaces. Moreover, every reasonable effort will be made to enroll selected hospitals (in accordance with a predetermined hospital solicitation protocol) before substitution is allowed.

The second and final stage of sampling will consist of two sub-steps, or "phases" in the vernacular of statistical sampling (Kish, 1965 and Cochran, 1977). In the first phase, a random sample of weeks will be chosen for enrolling Latina births and women. The second phase will be to randomly and separately sub-sample the first phase weeks for enrolling non-Latina births and women.

In the likely event that the expected annual number of Latina births in the final hospital sample exceeds 750 patient enrollment during the full 52 weeks of data collection in participating hospitals will not be necessary. More specifically, if the actual total expected annual number of Latina births is X, then the overall percent sampling rate for weeks will be approximately 100[750/X]. For each racial/ethnic group, the number of weeks (k) out of 52 that are randomly designated for subject enrollment in participating hospitals will vary among hospitals in such a way as to achieve (as close as possible) an overall equal-probability sample of women within each of the four hospital strata. Since the overall probability of selecting a woman in this sample (

) will be the product of the probability of choosing her hospital (

) and the week in which she gives birth once her hospital is chosen (

), the number of weeks (k) out of 52 weeks in her hospital becomes 26 times

It should be noted that, in general for each racial/ethnic group, selected time periods will differ among participating hospitals. The selected time period for each individual hospital will be the same for each of its two years of enrollment. However, the set of women enrolled in the study is likely to differ among the participating hospitals and enrollment years.

For reasons of operational efficiency, the randomly chosen sample of k weeks for a sample hospital will be partially consecutive. This will be achieved by considering the sampling frame to be a circular list of 52 items. After randomly choosing a number (g) between 1 and 52, the gth week plus the next k-1 after that (starting again with the first week, if necessary) will be the designated enrollment period for that hospital. For example, if g=2 in a hospital with k=9, the enrollment period will consist of weeks 2-10, whereas if in that same hospital g=46, the period will be weeks 1-2 and 46-52. By this scheme each week (and thus each birth and associated woman chosen) has the same within-hospital probability (k/52) of being chosen.

The final sampling step proceeding each survey period will be to randomly sub-sample consecutive weeks from the set of weeks designated for enrolling Latina births. Using the same selection procedure as used to sample the weeks for enrolling the Latina group, a separate sub-sample will be chosen for the non-Latina group. Assuming an 80 % response rate among selected women, a selected sample of about 360 non-Latinas will be targeted to yield the required respondent sample size of 300.

Several important issues and limitations tied to the proposed sample design should be noted. The sample will be highly clustered to control screening and data collection costs. Although, as previously noted, the statistical effect of this clustering will be to increase the variance of study estimates, and thus reduce the effective sample size for some comparative analyses. Another feature of the design, with both advantages and disadvantages, is its disproportionality, which is intended to reduce the number of hospitals needed to achieve the needed sample sizes. The main statistical disadvantage of allowing the variable sampling rates that lead to the disproportionality will be the creation of an additional source of loss in the precision of survey estimates (Kish, 1965, Section 11.7). These limitations, of course, will be offset by the variance reduction benefits of sample stratification and PPS selection.

A second issue raised by the proposed sampling design centers on selection of non-Latinas in the final phase of the third sampling stage. As proposed, the samples for this group will be successive sub-samples of the sample of weeks during which Latina women will be interviewed. For example, if Latinas are interviewed during weeks 10-28, non-Latinas will be interviewed during some of the same weeks (e.g., 12-19). A less complex and labor-intensive alternative would be to include the next non-Latina woman after a Latina woman is identified and interviewed. The statistical difficulty with this alternative approach is that the non-Latina sample would technically not be statistically valid (since some women in the non-Latina group would have no chance of being chosen). However, the sets of selected women in this group would serve as reasonable time- and institution-matched control samples for members of the Latina sample. Some effort during the project will be devoted to testing, weighing alternatives, and seeking a resolution to this issue.

A third key issue is births to women outside of the hospital setting and thus the coverage of the sample in this design. While the number of births to Latina women outside of the hospital setting is unknown and is expected to be low, the precise effect of the exclusion of these births on inference to the larger population of all Latina women may be more important, especially if perinatal outcomes of Latina women giving birth outside of the hospital setting are substantial different from Latinas having their children in hospitals. Some of these nonhospital births are logged as part of the State’s vital records system. The sample imbalance created by birth outside of the hospital can, at least, be partially eliminated through the use of post-stratification weight adjustments (Kalton, 1983a).

The measurement team of Ms. Bowers, Dr. Lessler, and Dr. Willis brings a vast amount of experience in developing and testing health survey instruments to this project. With their combined expertise in survey measurement and extensive experience in leading many questionnaire development and testing efforts, this group of researchers has the strong background in survey measurement required to complete Project 1's measurement goals. Drs. Lessler and Willis' leading roles in the introduction of cognitive methods to survey research is an important asset to this team.

As a survey methodologist, Ms. Bowers has directed and been involved in many questionnaire design and testing efforts. While at RTI and now UNC, Ms. Bowers has worked with a variety of health researchers on the design of several survey instruments. She has reviewed instruments addressing numerous health-related topics from nutritional assessments to a survey on Medicaid underutilization by African-Americans in North Carolina. She has collaborated with health researchers on these surveys to identify and remedy potential measurement difficulties, including comprehension and retrieval problems. This experience will be important as she works with subject matter experts on this project to develop survey measures of perinatal outcomes and as she looks to identify possible comprehension and retrieval problems with these measures.

Ms. Bowers also has participated in and led the cognitive testing of various health survey instruments. As a member of the RTI cognitive testing team, she conducted and analyzed cognitive interviews with Medicare beneficiaries for the Consumer Assessment of Health Plans Study (CAHPS). The CAHPS is an instrument designed for health plans to collect information from consumers about satisfaction with their health care. The CAHPS testing effort was funded by the Agency for Health Care Policy and Research and the Health Care Financing Administration. At UNC, Ms. Bowers has led cognitive testing efforts for two major surveys, a survey assessing benefit to subjects in gene transfer research (GTR) studies and an organizational survey involving substance abuse programs in school districts. For both of these projects, she has: (1) identified possible measurement problems, (2) outlined the objectives of cognitive testing to address the potential measurement difficulties, (3) developed a cognitive testing plan (including the number of rounds of interviewing and logistics of recruiting and scheduling subjects) and protocol for implementation of the interviews, and (4) conducted cognitive interviews (or trained interviewers to administer the interviews). Presently, with testing complete, she is working with the subject matter researchers to analyze the results of cognitive testing and suggest revisions to the instruments.

In the GTR project, she collaborated with Department of Social Medicine researchers to develop an instrument assessing perceptions of benefit to subjects enrolling in GTR studies. This instrument is one of the first, if not the first, instruments designed to measure perceptions of benefit in GTR studies. Understanding respondent interpretation of concepts and terms regarding benefit was an important goal of cognitive testing for this project. Ms. Bowers will utilize this experience in developing comprehension probes and apply it to assessing Latino respondents' comprehension of questionnaire items. Constructing survey questions that have a uniform meaning across respondents, a major challenge in the measurement part of Project 1, was a challenge in Ms. Bowers' second recent cognitive testing project which involved interviews with substance abuse prevention staff in school districts. Staff in districts across the country may use different terms than those used by the questionnaire designer/subject matter experts. In addition, there may be variability across school districts, perhaps depending on the level of sophistication of prevention efforts. Once again, assessing comprehension across different types of respondents is critical, as it will be in this project.

Dr. Lessler is Director of the Statistical Research Division at RTI. She is a Fellow of the American Statistical Association and an expert in survey methods. She collaborated with Dr. Monroe Sirken and others at the National Center for Health Statistics on the initial study exploring the use of cognitive laboratory methods for designing survey questionnaires. This project led to widespread use of cognitive science in both the US and internationally. She has published extensively in this area. Dr. Lessler also has considerable experience in questionnaire development for many health-related projects. For example, she has directed the development of the survey instruments for Cycle V of the National Survey of Family Growth under contract with the National Center for Health Statistics. She also developed questionnaires for surveying women who participate in the Women, Infants, and Children (WIC) Feeding Program. With her substantial experience in questionnaire development and extensive expertise in cognitive methods, Dr. Lessler's contribution to Project 1 will be invaluable.

Dr. Willis is a Senior Research Statistician/Psychologist at RTI. He is a widely recognized authority on cognitive interviewing techniques and has considerable experience in conducting questionnaire reviews, cognitive testing, and other forms of pretesting of questions intended for both general population and special population surveys. Previously, Dr. Willis was a senior staff member in the Questionnaire Design Research Laboratory at the National Center for Health Statistics, where he took responsibility for pretesting of questionnaires for health surveys such as the National Health
Interview Survey and the National Health and Nutrition Examination Survey. He now continues to conduct this type of work for RTI, with respect to a wide range of survey questionnaires. Dr. Willis has also conducted targeted survey methodological research to determine the efficacy of various methods
of survey pretesting, such as expert review, behavior coding, and cognitive interviewing. Further, he has written a widely used manual on "Cognitive Interviewing and Questionnaire Design," which has served as a general "hands on" resource for questionnaire developers in the survey research community,
and he has presented and published extensively in the areas of questionnaire design, pretesting methods evaluation, and survey response error. Dr. Willis will apply his considerable experience and expertise to advise the Project 1 team regarding issues related to cognitive interviewing methods.

The main statistical novelty in the methods work to be done in the sampling part of the will be in the use of a basic theoretical framework for evaluating the error implications of sampling migrant seasonal farmworkers, based on earlier work by Kalsbeek (1988). In this framework, the sample design being evaluated presumes that one must sample both in the spatial dimension (e.g., migrant camps and persons within camps) and the temporal dimension (i.e., to decide on which randomly chosen days during the period of study should select camps be visited). Time and space in this two dimensional configuration of sampling units can be sampled in a number of ways (e.g., treating camps as PSUs, worker households as SSUs, and clusters of days as TSUs; or one can reverse the order of the time and space dimensions so that one designates a clusters of days as PSUs, camps as SSUs, and worker households as TSUs; or one can independent sample in the time and space dimensions). Since how one samples in the time and space dimensions impacts resulting survey error, one must formulate the error implications among plausible options and then compare the results. In addition to considering the various combinations of time and space sampling for studies of migrant and seasonal farm workers, we intend to extend the basic theory we have developed for estimating the total count of MSFWs and various measures tied to perinatal outcomes (e.g., average gestational age of the child at birth). In the existing results, we assume relatively simple equal-probability sample designs in the time and space dimensions. In the proposed work we would extend the existing results to include more complex designs involving cluster sampling and disproportionate allocation of stratified samples, thus making our theoretical results more relevant to our evaluation of designs for sampling the transient population of Latinas, where these complicating features of the sample design are likely to be found.

The measurement part of this project will focus on the application of cognitive methods to address many of the measurement challenges found in cross-cultural survey research. One qualitative method often used in questionnaire development and testing is the cognitive interview. Cognitive interviewing is now widely used in cognitive laboratories at the National Center for Health Statistics (NCHS), Bureau of Labor Statistics (BLS), Bureau of the Census and in labs at major academic and private survey organizations. In a cognitive interview, a respondent is interviewed one-on-one by a trained questionnaire design specialist. The interviewer may ask the respondent to "think aloud" as he develops his answers and/or the interviewer may ask directed probes after the respondent has answered the question. With the "think aloud" approach, respondents are asked to report what they are thinking as they answer each question, allowing the researcher to obtain valuable information on their thought processes. The basis of the "think aloud" approach is discussed in Protocol Analysis by Ericsson and Simon (1993), who have used this approach to analyze problem solving. Thinking aloud can be a useful technique for detecting problems with questionnaire items, such as retrieval difficulties (Sudman, Bradburn, & Schwarz 1996). Follow-up probes are another technique often used in cognitive interviewing. An example of this type of probe might be "Did you include X in your response?". These directed probes can help assess respondent comprehension of some term or concept (Forsyth and Lessler 1991; Sudman et al. 1996). There are many examples in the literature of problematic terms and questions that have been discovered through the use of these cognitive interviewing techniques (see examples in Belson 1986; Royston 1989; Bradburn & Miles 1979; Cantril & Fried 1944; Sirken 1989; Willis 1991). In sum, these techniques applied in the cognitive interview can be invaluable in determining if questions are being interpreted and answered as intended.

However, Hines has argued that most researchers developing instruments for culturally diverse populations have failed to utilize these techniques in their work. There is a small but growing literature that is beginning to challenge this assertion. For example, there has been cognitive testing of the CPS with Latinos. Cognitive interviewing with the CPS has identified a number of comprehension problems, including difficulties with formal Spanish terms to problems understanding phrases in English like "born abroad" and "citizen by naturalization" (see de la Puente & McKay 1995; McKay & Aguirre 1994; Wellens 1994). Assessing comprehension problems like these, specifically problems with language and concepts that may arise in cross-cultural surveys, can be an important use of a technique like cognitive interviewing. Beyond this, cognitive interviewing can provide key insight into respondent familiarity with the survey task and interview. To general populations researchers, this may not be a great concern. On the other hand, this is an important concern in cross-cultural research. Qualitative interviews can elicit useful information from respondents about their difficulties with the survey task and help us design instruments that better address these problems.

On the other hand, it is important to note that Nisbett and colleagues have questioned individuals' ability to produce valid introspective reports of their thought processes (Nisbett & Ross 1980; Nisbett & Wilson 1977). However, Sudman et al. (1996) point out that typically in cognitive interviews we are not relying on respondents' interpretation of their cognitive processes but rather we are asking them to report what they are thinking for later interpretation by the researcher. Sudman et al. also argue that thinkalouds do not change respondent thought processes, except perhaps they could encourage respondents to put additional effort into retrieval. There is another possible limitation of the cognitive approach in that thinkaloud reporting may not be complete. Information that is not readily accessible to the respondent will not be reported. The effect of mood on responses and the effect of information provided in earlier questions on later responses often go unreported (Sudman et al. 1996). Wellens (1994) has raised concerns about the use of cognitive techniques in interviewing non-native English speakers since they are having to answer questions in a foreign language and report their thoughts at the same time. However, she has concluded based on her cognitive research with foreign-born respondents that they were very capable of providing their thoughts, particularly when probed specifically for their thoughts. These are important limitations to keep in mind when considering the objectives of cognitive testing and when interpreting the findings from cognitive interviews. Nonetheless, given these constraints and limitations, cognitive interviewing is a powerful technique in assessing a variety of measurement problems as detailed above.

Cognitive interviews also provide an opportunity to gather information about the sensitivity of question items. This may be difficult to assess using direct questions or probes. Rather, it may involve using postinterview projective questions as suggested by Sudman et al. (1996). We will consider this further as we develop and test our questions.

In sum, cognitive interviewing techniques seem to hold promise for helping to address many of the measurement problems of cross-cultural survey research. We hope to build on and expand the use of cognitive methods to our interest in obtaining accurate survey measurements of perinatal health outcomes and their determinants from Latina women.

For time and space sampling the universe sampled consists of the cross-classification between a set of units comprising the frame in the spatial dimension and a set of time units spanning the interval of time over which sampling is to be done in the temporal dimension. Units sampled in space might include some combination of area units (e.g., counties or census block groups), organizations (businesses, hospitals, or office-based physician practices) and/or individual population members (e.g., adults, hospital medical records, pregnant Latina women). Units sampled in time might be individual or grouped sequences of standard units of time like minutes, hours, days or weeks. The conceptual framework for time and space sampling developed by Kalsbeek (1988) to sample migrant and seasonal farm workers is the starting point for the theoretical work that involves time and space sampling to study elusive populations. Other theoretical frameworks that will be utilized to complete other sampling parts of this project are the basic theory of area sampling summarized by Kish (1965) and Cochran (1977), and dual-frame cost optimization first considered more generally by Hartley (1962) and Choudhry (1989), extended in the general case to more complex designs by Skinner and Rao (1996). Much of the recent work on dual-frames has been in conjunction with telephone sampling (e.g., Casady, 1989; Tucker,1989; Durant and Vitrano, 1989; Traugott and Goldstein, 1993; Biemer, 1994; and Schejbal and Lavrakas, 1995).

Our time and space model for sampling actively nonsedentary populations presumes a spatial sampling frame consisting of L units (from which l* units are sampled), a temporal frame consisting of M units (from which m are chosen), and a stable target population of size N whose size is to be from the survey sample. The two-dimensional array consisting of LM cells, corresponding to the cross-classification of the time and space frames, is thus used to choose the study sample. Four approaches to jointly sampling time and space were considered in this work; viz.,

1) Unrestricted Random Sampling --- In which an unrestricted (i.e., simple random with replacement) random sample (i.e., URS) of size l*m cells in the two-dimensional array.

2) Unstratified 2-Stage Sampling --- In which a URS (l* of L) spatial sampling units are first chosen from the time dimension, and a URS (m of M) is separately chosen from each of l* selected spatial units.

3) Substratified 2-Stage Sampling --- In which a URS (l* of L) spatial sampling units are first chosen from the time dimension, and a stratified URS (m of M) is separately chosen from each of the l* selected spatial units.

4) Unstratified 2-Way Sampling --- In which the same URS (m of M) temporal sampling units is used for each member of the URS (l* of L) spatial sample.

The sampling error implications and thus the statistical efficacy of these four approaches were found to depend on which probability sampling methods are applied to each dimension (i.e., simple random sampling or stratified simple random sampling) and the manner in which the probability sampling methods are jointly applied to the two sampling dimension (i.e., independently, hierarchically sampling time first and then space within time, etc.). The way in which the measure of interest distributes itself along the time and space dimensions and strata formed in sampling time also impact the relative precision of estimates from the various approaches. For instance, the intraclass correlation of temporal frame units within spatial clusters and a measure of the effectiveness of substratification of temporal units are important parameter of relative statistical efficiency of the four approaches.

Lacking from this initial round of theoretical work is the use of more complicated methods of probability sampling (e.g., cluster sampling combined with stratification, disproportionately allocated stratified sampling, etc.) for sampling in the two dimensions. These more complex sampling strategies are often needed for sampling in space to reduce frame construction or data collection costs. Practical considerations are also sometimes needed to sample time. For example, sampling clusters of three consecutive days to screen hospital outpatients visits for some medical condition may be less costly than sampling individual days when screening must be done by a traveling screening specialist. The proposed sampling work in this project will extend the theory of time and space sampling.

Sources of empirical data for estimating parameters of cost and statistical error models for sampling elusive populations will be obtained from administrative and research data files from a recent study by the UNC-CH Department of the Maternal and Child Health on the health of migrant Latina women (Watkins et al., 1990). This study gathered data from about 360 women working at several migrant farming operations in North Carolina. We will also seek out data from other population-based health studies of migrant and seasonal farm workers through contacts with the Office of Migrant Health, the US Department of Agriculture, and the National Center for Health Statistics, where useful empirical data sources may be available through the research they fund and conduct themselves. Substantive and cost data for sampling hospitals will be sought for research studies involving sampling and/or data gathering involving hospitals (e.g., the National Hospital Ambulatory Medical Care Survey, the National Hospital Discharge Survey, the Medical Expenditure Provider Survey, the American Hospital Association resource listing of hospitals, the Nationwide Inpatient Sample from the Healthcare Cost and Utilization Project, etc.).

The objectives of the sampling part of this project will be accomplished by completing the following research steps:

Develop a cost-effective (dual-frame) design to jointly sample the transient and settled portions of the population of Latina women in North Carolina following the early-pregnancy prospective research design. Completing this main step will require the following substeps:

(a) Further examine the survey error properties of time and space sampling in order to sample the transient portion. This step will involve extending the theoretical work of Kalsbeek (1988) to more complex designs involving stratification and cluster sampling in time and space. The precise combinations of complex designs that we use for the time and space dimensions will depend on which would be most appropriate for sampling migrant Latina women early in pregnancy. Also considered in deciding on which sampling approaches to use will be the issue of screening for early pregnancy in this setting. Where appropriate we will draw on our previous design research on sampling women during pregnancy (e.g., see Kalsbeek, et al, 1987; Peoples-Sheps et al., 1988, and Peoples-Sheps et al., 1991). The main product of this effort will be one or more mean square error (MSE) models for estimating measures of perinatal outcome in this sampling context.

(b) Examine the survey error properties of approaches to sampling and screening for rare attributes as they relate to sampling the settled. Here we will address the statistical inference issues tied to how one might best sample and then screen for women early in pregnancy, when area sampling, telephone sampling, and other more common approaches to sampling the settled residential population are used. These efforts will draw Kalsbeek’s prior work in sampling the uninsured (Kalsbeek, 1978), African-Americans (Kalsbeek, 1992), and others. The statistical implications of sampling to efficiently screen for rare attributes will be especially important here, thus stressing the need to consider the suggestions by Kalton and Anderson (1986) and the experience derived from NCHS and other CDC studies aimed at oversampling households with young children, ethnic minorities, and other scattered rare attributes. The main product of this effort will be one or more MSE models for estimating measures of perinatal outcome in this sampling context.

(c) Develop cost models for the sample designs for the transient and settled portions. Here we will utilize prior work by Groves (1989) to develop one of more suitable cost models corresponding to each of the design options developed for the transient and settled portions of the pregnant Latina populations. The formulation of these models will be consistent with the survey design that would be applied to each portion.

(d) Separately optimize allocations of the sample designs for the transient and settled portions. The first part of this step will require the assembly of plausible ranges of empirically derived values for parameters tied to the MSE and cost models. Plausible ranges of empirically derived values for cost parameters in these models will be obtained utilizing survey administration data from the Watkins MSFW study and from other available studies that match the design circumstances we are considering in this project. Estimates of the MSE model parameters will be obtained from an analysis of the substantive data from the Watkins study and from other surveys of maternal morbidity. Most of the MSE model parameters will be of three types: (i) measures of intraclass correlation reflecting the statistical effect of cluster sampling corresponding to the time and space sampling of the transient portion of the Latina population or to sampling in the spatial dimension alone for the settled portion; (ii) parameters reflecting the effects of stratification on the sampling error of proportionately allocated samples, and (iii) measures of the effect of disproportionate allocation in stratified samples.

Our general approach will be to bracket all MSE and cost parameter estimates by establishing high, low, and midrange values, with the latter our best estimate based on prior similar studies. Bracketing will be done to enable us to gauge the sensitivity of subsequent optimum results to the values of cost and MSE parameter estimates.

In the second part of this step, optimally allocated designs for settled and transient portions of the population of pregnant Latina women will be separately developed by first solving for relevant design parameters for each portion. Closed-form solutions using classical methods of cost-MSE optimization will be sought first. Failing in this approach, iterative solutions or those obtained by nonlinear integer programming techniques will be obtained. All combinations of bracketed values of the cost and MSE model parameters will be run to enable us to gauge the sensitivity of findings to changes in the models parameter values.

(e) Develop an optimum no-overlap dual-frame design. Finally, an optimally sample-weighted dual-frame solution to the prospective sample design for Latina women will be sought, following Hartley’s original dual-frame optimization framework.

Develop a cost-effective hospital-based time and space sample design for the female Latina population in North Carolina shortly after the birth of their child, following the at-birth research retrospective design. Completing this main step will require the following substeps:

(a) Further develop the approach and examine the survey error properties of a preliminary time and space approach to sampling. Activity in this step will build from the theoretical work on time and space sampling in Step (1a). The unique features here are that sampling in the spatial dimension will involve hospitals since the Latina women sampled under the retrospective design option are chosen just after the birth of their child, which most frequently occurs in the hospital setting. The other distinct element of sampling in this step is the need to sample in time as well given the relatively slow accrual of births to Latina women in some hospitals and the possibility of utilizing traveling teams to sample and gather the data from selected women. As with Step (1a), the main product of this effort will be one or more mean square error (MSE) models for estimating measures of perinatal outcome in the context of sampling and interviewing women in the hospital setting.

(b) Develop corresponding cost models corresponding to the enhanced preliminary design. This activity will be comparable to Step (1c).

(c) Estimate model parameters and optimize allocations for the enhanced preliminary design. This activity will be comparable to Step (1d).

A number of improvements over existing research knowledge on research design will be realized by this project. One is this study’s particular focus on design in the context of studies of Latina women, where to our knowledge little specific attention has been devoted by statistical designers in the past. Hopefully the new knowledge gained will advance the quality of research and thus the future health of this important high risk segment of society. This work will also advance relatively limited design research that has been done in the name of effectively sampling the population of migrant and seasonal farm workers. We believe this attention is long past due and will thus be a major hallmark of this project.

While a number of new and important paths of research of statistical design will be traversed, a number of potential limitations to the work done in this project must be noted. One is the somewhat uncertain availability and utility of statistical and cost data sources for some of the cost optimization work we have proposed. For example, the statistical data from the Watkins et al.(1990) Study of Latina Migrant and Seasonal Farm Workers were obtained from a nonprobability sample of migrant camps, and this study’s administrative data must be sought and may be less that fully useful to the project. In the event that the planned data sources are unavailable or less than fully useful to the project, we will conduct a broad search for health related studies aimed at the migrant community. This search will be facilitated by several contacts that Dr. Kalsbeek has in the migrant health community from his prior design research in this area.

The theoretical basis for our research lies in the work by Sudman et al. (1996), Tourangeau (1984), and others who have investigated the cognitive processes involved in answering survey questions. We will not attempt to extensively review this literature here except to give a brief overview of the question answering process which we will rely on in our research. In general terms, answering a survey question involves comprehension, retrieval of relevant information, judgment based on available information and response formatting/selection. Measurement effects can be introduced at any point in this question answering process. Respondents can retrieve incorrect information or misunderstand the question, which may lead to an inaccurate response. Previous items can influence responses to later ones in an unintended way producing context effects. Respondents also may adjust their answers to reflect social norms. For example, respondents may underreport socially undesirable behaviors, like drug use. A clear understanding of this question answering process and the problems that may arise in the process is crucial to the development of a valid survey instrument. We will refer often to the theoretical framework of the question answering process and the potential for error in it during our questionnaire development and testing efforts.

Following closely from this understanding of the question answering process is the application of verbal reporting or cognitive methods to survey research. Researchers in the psychology literature have reported on the theory and use of verbal reporting methods in psychological research (see Ericsson & Simon 1993; Payne 1994; Wilson 1994; Crutcher 1994; Nisbett & Ross 1980; Nisbett & Wilson 1977). The work by Ericsson and Simon (1993) was noted earlier. There has also been some controversy in this literature regarding the limitations and validity of verbal reporting in various contexts. The introduction of verbal reporting techniques in questionnaire development has been described in the survey literature by various researchers (Sudman et al. 1996; Forsyth & Lessler 1991; Willis et al. 1991). Work by these researchers and others provided much of the background (as described earlier) for the research we propose here. An understanding of the use of cognitive techniques in assessing measurement problems and the limitations of these will provide a critical foundation for the work we do in this project.

As part of the first measurement objective, we will develop questions to gather information about perinatal health outcomes and their determinants from Latina women. In constructing these questions we will rely on the substantive knowledge of the Project 1 subject matter team. They will define a clear set of measurement objectives based on their research interests. Substantive researchers in this project are interested in understanding the Mexican paradox of low rates of preterm delivery and lower infant mortality. Thus, some measures of interest may be the respondent's last menstrual period (to assess gestational age), infant death in the first year of life, use of health care services and health behaviors during pregnancy, and acculturation.

Our two key data sources for examining the effectiveness of cognitive interviewing in addressing the measurement challenges associated with cross-cultural survey research will be results from the cognitive interviews we conduct and results from previous cognitive research reported by federal statistical agencies, like the Bureau of the Census. We will use these two sources to determine what kinds of problems cognitive testing revealed and we will consider how the results of cognitive testing might be used to improve survey measures.

The measurement part of this project will proceed according to the steps outlined below:

Identify specific measurement challenges in obtaining accurate data on perinatal outcomes from Latina women. Completing this main step will require the following substeps:

Apply and adapt cognitive methods for testing general population surveys to address measurement challenges identified in (1). Completing this main step will require the following substeps:

Develop general recommendations for designing and testing survey instruments for culturally diverse populations, like Latinos, based on literature and experience with this project. Completing this final step will require the following substeps:

The following timeline for this project assumes an October 1, 1999 start to the project:

Step	Start	End
Sampling:
1. Investigate optimum dual-frame sampling for the prospective research design	10/99	4/01
2. Investigate optimum hospital-based sampling for the retrospective research design	8/00	7/01
3. Compare sampling approaches for the prospective and retrospective designs	8/01	2/02
Measurement:
4. Identify potential measurement challenges in obtaining data from Latina women on perinatal outcomes	10/99	6/00
5. Apply cognitive interviewing techniques to address measurement challenges identified in (1)	5/00	9/01
6. Prepare recommendations for use of cognitive methods in health research with Latino populations	8/01	2/02
Overall Objective:
7. Determine the most effective approach for sampling and measuring Latina women	3/02	9/02

Throughout the three-year period of this project, we will actively seek opportunities to share our findings with colleagues in the scientific community. In addition to presenting seminars and papers at professional statistical meetings and other colloquia, and publishing in scholarly journals, we will look to share the findings from our research with others doing research on the health of high-risk populations. This may, for instance, involve presenting at the annual Minority Health Conference at UNC.CH and at the conference with compatible subject matter or methods themes to the work on this project.

Adriaans NFP. The practice of snowballing. In Snowball Sampling: A Pilot Study on Cocaine Use (Hendricks VM, Blanken P, Adriaans NFP, eds) 1992.

Albrecht SL, Miller MK. Hispanic subgroup differences in prenatal care. Social Biology 1996;43:38-58.

Aliza B, Brown T, Fine A, Lynch LG. Partnerships for healthier families: Principles for assuring the health of women, infants, children, and youth under managed care arrangements. Washington, DC: Association of Maternal and Child Health Programs (AMCHP);1996.

Andersen R, Aday L. A National Survey of Access to Medical Care. The Center for Health Administration Studies. Chicago, IL: The University of Chicago; 1982.

Anderson DA, Kalsbeek WD, Hartwell TD. The National Head and Spinal Cord Injury Survey: Design and

Anderson L, Calhoun P. Facilitative aspects of field research with deviant street populations. Sociological Inguiry 1992; 62(4):490-498.

Anderson SA. Core nutritional indicators for Difficult-to-Sample populations. Federation of American Societies for Experimental Biology 1990.

Aneshensel CS, Becerra RM, Fielder EP, Schuler RH. Participation of Mexican American Female Adolescents in a Longitudinal Panel Survey. Public Opinion Quarterly 1989; 53: 548-562.

Armstead RC, Gorman JK. Baby love and budget relief: some promising practices in prenatal managed care in Medicaid. Journal of the American Medical Women’s Association 1995;50:178-181.

Bennett T. Monitoring welfare and women's health. In: Chapman AR, ed. Health care and information ethics: protecting fundamental human rights. Kansas City: Sheed & Ward; 1997a. p. 66-87.

Berg S. Snowball sampling. In Encyclopedia of Statistical Sciences (Kotz S, Johnson NL, eds) 1988; 8:529-532.

Biemer P, Akin D. The Efficiency of List-Assisted Random Digit Dialing Sampling Schemes for Single and Dual Frame Surveys. Proceedings of the Section on Survey Research Methods, American Statistical Association 1994; 1-10.

Bienstock JL, Blakemore KJ, Wang E, Presser D, Misra D, Pressman EK. Managed care does not lower cost but may result in poorer outcomes for patients with gestational diabetes. American Journal of Obstetrics and Gynecology. 1997;177:1035-7.

Blanken P, Hendricks VM, Adriaans NFP. Snowball sampling: methodological analysis? In Snowball Sampling: A Pilot Study on Cocaine Use (Hendricks VM, Blanken P, Adriaans NFP, eds) 1992; 83-100.

Bradburn NM, Miles C. Vague Qualifiers. Public Opinion Quarterly 1979; 43: 92-101.

Bradburn NM, Sudman S, Blair E, Stocking C. Question Threat and Response Bias. Public Opinion Quarterly 1978; 42: 221-234.

Braveman P, Bennett T, Lewis C, Egerter S, Showstack J. Access to prenatal care following major Medicaid eligibility expansions. Journal of the American Medical Association 1997;269:1285-1289.

Braveman PA, Egerter S, Bennett T, Showstack J. Differences in hospital resource allocation among sick newborns according to insurance coverage. Journal of the American Medical Association 1991;266:3300-3308.

Brick M., Waksberg J, Kulp, and Starer (1995). Bias in List-Assisted Telephone Samples, Public Opinion Quarterly, 59:218-235.

Brown SS. Prenatal care: Reaching mothers, reaching infants. Washington, DC: National Academy Press; 1988.

Buechley R. A reproducible method of counting persons of Spanish surname. American Statistical Association Journal 1961; March:88-97.

Buekens P, Delvoye P, Wollast E, Robyn C. Epidemiology of pregnancies with unknown last menstrual period. J Epidemiol Community Health 1984;38:79-80.

Buekens P, Hernandez P, Infante C. La atencion prenatal en America Latina. Salud Publica de Mexico 1990;32:673-684.

Buekens P, Kotelchuck M, Blondel B, et al. A comparison of prenatal care use in the United States and Europe. American Journal of Public Health 1993;83:31-36.

Buekens P, Notzon F, Kotelchuck M, Wilcox A. Why do Mexican-Americans have few low birthweight infants? Am J Epidemiol 1999;149:S28.

Buekens P. Barriers and incentives to prenatal care in Europe. Report to the European Commission. Brussels, Belgium: Free University of Brussels; 1997.

Buekens P. Variations in provision and uptake of antenatal care. Bailliere’s Clinical Obstetrics and Gynecology 1990;4:187-205.

Buescher PA, Smith C, Holliday JL, Levine RH. Source of prenatal care and infant birth weight: The case of a North Carolina county. American Journal of Obstetrics and Gynecology 1987;156:204-210.

Byrd TL, Mullen PD, Selwyn BJ, Lorimor R. Initiation of prenatal care by low-income Latina women in Houston. Public Health Reports 1996;111:536-540.

Cantril H, Fried E. The Meaning of Questions. In Gauging Public Opinion (Cantril H, ed.) 1944.

Casady RJ, Nathan G, Sirken MG. Alternative Dual System network estimators. International Statistical Review 1985; 52(2):183-197.

Casady RJ. Telephone Survey Designs. Proceedings of the Section on Survey Research Methods, American Statistical Association 1989; 138-147.

Chiswick BR. A survey of employers of Illegal Aliens: analysis of the survey methodology. Proceedings of the Section on Survey Research Methods, American Statistical Association 1986; 178-183.

Choudhry GH. Cost-Variance optimization of Dual Frame design for estimating proportions. Proceedings of the Section on Survey Research Methods, American Statistical Association 1989; 566-571.

Clark L, Hofsess L. Acculturation. In Handbook of Immigrant Health (Loue S, ed.) 1992; 37-59.

Cochran RS. The estimation of Domain Sizes when Sampling Frames are Interlocking. Proceedings of the Social Statistics Section, American Statistical Association 1967; 332-335.

Cochran WG. Sampling Techniques. (Third Edition). New York, NY: Wiley and Sons; 1977.

Cochran WG. Sampling Techniques. (Third Edition). New York, NY: Wiley and Sons; 1977.

Cowan CD. The methodology of counting the Homeless. Proceedings of the Section on Survey Research Methods, American Statistical Association 1986; 170-175.

Cross, Anne R. and William D. Kalsbeek, "The Challenge of Surveying Nomads on the Move," Intercom, Vol II, No. 42, January/February pp. 11-13, 1983.

Crutcher RJ. Telling What We Know: The Use of Verbal Report Methodologies in Psychological Research. Psychological Science 1994; 5: 241-244.

Cuellar I, Harris LC, Jasso R. An Acculturation Scale for Mexican American Normal and Clinical Populations. Hispanic Journal of Behavioral Sciences 1980; 2(3): 199-217.

Czaja R, Blair J. Using Network sampling for rare populations: An application to local crime victimization surveys. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1988; 38-43.

Czaja R, Snowden C, Casady R. Reporting bias and sampling errors in a survey of a rare population using Multiplicity counting rules. Journal of the American Statistical Association 1986; 81:411-419.

David R. The quality and completeness of birthweight and gestational age data in computerized birth files. Am J Public Health 1980;70:964-73.

Davis D, Collins KS, Morris C. Managed care: Promise and concerns. Health Affairs 1994;13:178-185.

De la Puente M, McKay R. Developing and Testing Race and Ethnic Origin Questions for the Current Population Survey Supplement on Race and Ethnic Origin. U.S. Bureau of the Census, working paper, 1995.

Delgado-Rodriguez M, Gomez-Olmedo M, Bueno-Cavanillas A, Garcia-Martin M, Galvez-Vargas R. Recall bias in a case-control study of low birth weight. Journal of Clinical Epidemiology 1995;48:1133-1140.

Dennis ML, Iachan R. Sampling people who are homeless: implications of multiple definitions and sampling frame Proceeding of the Section on Survey Research Methods, American Statistical Association, 1992; 87-89.

Deyo R, Diehl A, Hazuda H, Stern M. A Simple Language-Based Acculturation Scale for Mexican Americans: Validation and Application to Health Care Research. American Journal of Public Health 1985; 75: 51-55.

Durant S, Vitrano F. Response rates in a Dual-Frame sample design CATI test. Proceedings of the Section on Survey Research Methods, American Statistical Association 1989; 367-371.

Elias-Olivares L, Farr M. Sociolinguistic Analysis of Mexican-American Patterns of Non-response to Census Questionnaires. U.S. Bureau of the Census, Ethnographic Exploratory Research Report #16, 1991.

Ellwood MR, Ku L. Welfare and immigration reforms: unintended side effects for Medicaid. Health Affairs 1998;17:137-151.

Employment Security Commission. Migrant Farmworkers in North Carolina, 1996. Raleigh, NC: NorthCarolina Department of Human Resources; 1996.

Employment Security Commission. Migrant Farmworkers in North Carolina, 1996. Raleigh, NC: North Carolina Department of Human Resources; 1996.

Ericsson KA, Simon, HA. Protocol Analysis: Verbal Reports as Data (Revised Edition). Cambridge, MA: MIT Press; 1993.

Estrada AL, Trevino FM, Ray LA. Health care utilization barriers among Mexican-Americans: Evidence from HHANES 1982-1984. American Journal of Public Health 1990; 80(suppl):27-31.

Faugier J. Looking for business: a descriptive study of drug using female prostitutes, their clients and their health care needs. Unpublished PhD thesis, Manchester University, Manchester 1996.

Faugier J. Sargeant M. Sampling Hard to Reach Populations. Journal of Advanced Nursing 1997 Oct; 26(4):790-7.

Forsyth BH, Lessler JT. Cognitive Laboratory Methods: A Taxonomy. In Measurement Errors in Surveys (Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman SA, eds.) 1991; 393-418.

Franco JN, Malloy T, Gonzales R. Ethnic and Acculturation Differences in Self-Disclosure. Journal of Social Psychology 1984; 122: 21-32.

Frankel MR. A probability sample of the Homeless population of Chicago. Proceedings of the Section on Survey Research Methods, American Statistical Association 1986; 176-177.

Fuller WA, Burmeister LF. Estimators for Samples Selected from two Overlapping Frames. Proceedings of the Social Statistics Section, American Statistical Association 1972; 245-249.

Goldenberg R, Klebanoff M, Nugent R, Krohn, M, Hillier S, Andrews W. Bacterial colonization of the vagina during pregnancy in four ethnic groups. Am J Obstet Gynecol 1996;174:1618-1621.

Graves EJ, Gillum BS. National Hospital Discharge Survey: Annual Summary, 1994. National Center for Health Statistics: Vital and Health Statistics 1995;13:128.

Guendelman S, English P, Chavez G. Infants of Mexican immigrants. Medical Care 1995;33:41-52.

Guendelman S. Health and disease among Hispanics. In: Handbook of immigrant health. Loue S (ed.). New York: Plenum Press, 1998:277-301.

Hartley HO (1962). "Multiple Frame Surveys," Proceedings of the Social Statistics Section, American Statistical Association; 203-206.

Hendricks VM, Blanken P. Snowball sampling: theoretical and practical considerations. In Snowball Sampling: A Pilot Study on Cocaine Use (Hendricks VM, Blanken P, Adriaans NFP, eds) 1992; 17-35.

Hess CA. Managed Care. Washington, DC: Association of Maternal and Child Health Programs (AMCHP); 1993.

Hines AM. Linking Qualitative and Quantitative Methods in Cross-Cultural Survey Research: Techniques from Cognitive Science. American Journal of Community Psychology 1993; 21(6): 729-746.

Homeless and Transient Population. Journal of Official Statistics 1993; 9(4):747-764.

Homeless Population. Proceedings of the Section on Survey Research Methods, American Statistical Association 1993; 468-473.

Hosmer DW, Lemeshow S. Applied Regression Analysis. New York, NY: Wiley and Sons; 1989.

Iachan R, Dennis ML. A multiple Frame Approach to Sampling the Homeless and Transient population. Journal of Official Statistics 1993 ; 9(4): 747-764.

Iachan R, Dennis ML. The Design of Homeless Survey. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1991, 181-185.

Iachan R, Ringwalt CL, Greene JM. Substance abuse among runaway and homeless youth, 1994; 1083-1087.

Iachan R, Ringwalt CL. A national study of runaway and homeless youth. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1993, 845-849.

Iachan, R. , Dennis ML. Estimating the prevalence of substance abuse among people who are homeless Proceeding of the Section on Survey Research Methods, American Statistical Association, 1994; 1094-1099.

Johnson-Webb K, Johnson J. North Carolina communities in transition: An overview of Hispanic in-migration. The North Carolina Geographer 1996;5:21-40.

Johnson-Webb K, Johnson J. North Carolina communities in transition: An overview of Hispanic in-migration. The North Carolina Geographer 1996;5:21-40.

Kalsbeek W.D., S.L. Botman, and J.T. Massey. "Cost Efficiency and the Number of Allowable Call Attempts in the National Health Interview Survey," Journal of Official Statistics. Vol. 10, No.2, pp.133-153, 1994.

Kalsbeek WD. Nomad sampling: an analytic study of alternative design strategies. Proceedings of the Section on Survey Research Methods, American Statistical Association 1986; 164-169.

Kalsbeek WD. Nomad sampling: An analytic study of alternative design strategies. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1986; 164-169.

Kalsbeek, W. D. "Sampling," in Epidemiological Issues in the Oral Health of Black Americans: Methodology for Need and Risk Assessment, E. Taylor (ed.), National Institute of Dental Research, Bethesda, MD, 1992.

Kalsbeek, WD (1989). Design Study of Methods for Sampling Migrant and Seasonal Farm Workers, Final Report Monograph, Milbank Memorial Fund, New York.

Kalsbeek, William D. "Design Strategies for Nonsedentary Populations," Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 28-37, 1988.

Kalsbeek, William D. "Nomad Sampling: An Analytic Study of Alternative Design Strategies," Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 164-169, 1986.

Kalsbeek, William D., and Anne R. Cross, "Problems in Sampling Nomadic Populations," Proceedings of the Section on Survey Research Methods, pp. 398-402, American Statistical Association, 1982.

Kalsbeek, William D., Ophelia M. Mendoza, and David V. Budescu. "Cost Models for Optimum Allocation in Multi-Stage Sampling," Survey Methodology, Vol. 9, No. 2, pp.154-177, 1983.

Kalsbeek, William D., Sharon Y. Cornell, and Patricia S. Tennis. "On Developing a National Sample of Women Early in Pregnancy," Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 720-725, 1987.

Kalton G, Anderson DW. Sampling rare populations. Journal of Royal Statistical Society 1986; Ser. A, 149:65-82.

Kalton G, Anderson DW. Sampling Rare Populations. Journal of the Royal Statistical Society 1986; Ser. A, 149:65-82.

Kalton G. Introduction to Survey Sampling. New York, NY: Sage Publications; 1983a.

Kalton G. Compensating for Missing Survey Data. University of Michigan, Ann Arbor, MI: Research Report Series, Survey Research Center, Institute for Social Research; 1983b.

Kalton, G (1993) "Sampling Rare and Elusive Populations," United Nations Publication INT-92-P80-16E. National Household Survey Capability Programme, Statistical Division of the Department for Economic and Social Informational and Policy Analysis, United Nations, New York

King BF. Sampling from a fluid population. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1990; 332-336.

Kish L. A Taxonomy of elusive populations. Proceedings of the Section on Survey Research Methods, American Statistical Association 1988; 159-186.

Kish L. Taxonomy of Elusive Populations. Journal of Official Statistics, 1991; 7(3): 339-347.

Kish L. Taxonomy of Elusive Populations. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1988; 44-46.

Kotelchuck M. An evaluation of the Kessner Adequacy of Prenatal Care Index and a proposed Adequacy of Prenatal Care Utilization Index. American Journal of Public Health 1994;84:1414-1420.

Lepkowski JM, Groves RM. A Mean Squared Error Model for Dual Frame Mixed Model Survey Design. Journal of the American Statistical Association 1986; 81:930-937.

Lepkowski JM. Sampling the Difficult-to-Sample. Journal of Nutrition 1991; 121(3):416-23.

Lepkowski JM. Sampling the difficult-to-sample. Journal of Nutrition 1991 Mar. 121(3):416-23.

Lessler J, Tourangeau R, Salter W. Cognitive Laboratory Studies of the 1986 Supplement to the National Health Interview Survey Final Results. Proceedings of the Section on Survey Research Methods, American Statistical Association, 1986; 478-480.

Lessler JT and Kalsbeek WD. Nonsampling Errors in Surveys. New York, NY: Wiley and Sons; 1992.

Lessler JT, Forsyth BH. A Coding System for Appraising Questionnaires. In Swartz, N. and Sudman, S. eds. Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco, CA: Jossey Bass, Inc.; 1996. p. 259-291.

Lund ER. Estimators in Multiple Frame Surveys. Proceedings of the Social Statistics Sections, American Statistical Association 1968; 282-288.

Manson AB. North Carolina Latina population indicators by county. Raleigh, NC: NC Office of Minority Health;1998.

Marin G, Sabogal F, VanOss Marin B, Otero-Sabogal R, Perez-Stable EJ. Development of a Short Acculturation Scale for Hispanics. Hispanic Journal of Behavioral Sciences 1987; 9(2): 183-205.

Marin G, VanOss Marin B. A Comparison of Three Interviewing Approaches for Studying Sensitive Topics with Hispanics. Hispanic Journal of Behavioral Sciences 1989; 11: 330-340.

Marin G, VanOss Marin B. Research with Hispanic Populations. Newbury Park: Sage, 1991.

Marks G, Garcia M, Solis JM. Health risk behaviors of Hispanics in the US: Findings from the HHANES, 1982-84. American Journal of Public Health 1990; 80:20-26.

Martin E, DeMaio TJ, Campanelli PC. Context Effects for Census Measures of Race and Hispanic Origin. Public Opinion Quarterly 1990; 54: 551-566.

Maternal and Child Health Branch, Division of Nutrition and Physical Activity, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention. Pregnancy-related behaviors among migrant farm workers -- four states, 1989-1993. Morbidity and Mortality Weekly Report 1997;46:283-286.

Mathews TJ, Ventura SJ, Curtin SC, Martin JA. Births of Hispanic origin, 1989-1995. Monthly Vital Statistics Report 1998; (suppl) No. 6, 46

McKay RB, Aguirre A. The Spanish Translation of the Redesigned Current Population Survey - Lessons Learned. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Danvers, Massachusetts, 1994.

Messeri P, Aidala A, Abramson D, Healton C, Jones-jessop D, Jetter D. Recruiting Rare & hard to reach populations: A sampling strategy for surveying NYC Residents living with HIV/AIDS, using agency recruiters. Proceeding of the Section on Survey Research Methods, American Statistical Association, 1995; 1064-1068.

Moore P, Hepworth JT. Use of perinatal and infant health services by Mexican-American Medicaid enrollees. Journal of the American Medical Association 1994;272:297-304.

Multiple Frame Methodology and Selected Applications. Sankhya 1974; Ser. C 36:99-118.

National Association of Community Health Centers, Inc. (NACHC), The Patient Experience Evaluation Report System (PEERS). Washington, DC, 1996. National Center for Health Statistics. Advance Report of Final Natality Statistics. Vol. 44:3, Suppl 1996.

National Maternal and Infant Health Survey (NMIHS). Washington, DC: The Department of Health & Human Services, US Government Printing Office; 1989.

NCHAC (North Carolina Health Access Coalition) Legislative Update, May 28, 1998, health@ncjustice.org

Negy C, Woods DJ. The Importance of Acculturation in Understanding Research With Hispanic-Americans. Hispanic Journal of Behavioral Sciences 1992; 14(2): 224-247.

Nisbett RE, Ross L. Human Shortcomings of Social Judgments. Englewood Cliffs, NJ: Prentice-Hall; 1980.

Nisbett RE, Wilson TD. Telling More Than We Know: Verbal Reports on Mental Processes. Psychological Review 1977; 84: 231-259.

North Carolina Child Health Insurance Task Force. Final Report of the Task Force on Child Health Insurance to the Secretary of the North Carolina Department of Health and Human Services. Raleigh, NC; November 1997.

Payne JW. Thinking Aloud: Insights into Information Processing. Psychological Science 1994; 5: 241-248.

Peoples-Sheps, M.D. and W.D. Kalsbeek, E. Siegel, D. Dewes, M. Rogers, and R. Schwartz. (1991) "Prenatal Records: A National Survey of Content." American Journal of Obstetrics and Gynocology. 164 (2): 514-521.

Peoples-Sheps, M.D. and W.D. Kalsbeek, E. Siegel, D. Dewes, M. Rogers, and R. Schwartz. "Prenatal Records: A National Survey of Content." American Journal of Obstetrics and Gynocology. Vol. 164, No. 2, pp.514-521, 1991.

Peoples-Sheps, Mary D., William D. Kalsbeek, and Earl Siegel. "Why We Know So Little About prenatal Care Nationwide: An Assessment of Required Methodology," Health Services Research, Vol. 23, No.3, pp.359-380, 1988.

Peoples-Sheps, Mary D., William D. Kalsbeek, and Earl Siegel. (1988) "Why We Know So Little About prenatal Care Nationwide: An Assessment of Required Methodology," Health Services Research,. 23 (3):359-380.

Perez-Stable EJ, Napoles-Springer A, Miramontes JM. The effects of ethnicity and language on medical outcomes of patients with hypertension or diabetes. Medical Care 1997;35:1212-1219.

Pregnancy Risk Assessment Monitoring System (PRAMS) for North Carolina. Raleigh, NC: State Center for Health Statistics; 1997.

Rank O, Snijders T. Estimating the size of Hidden Populations using snowball sampling. Journal of Official Statistics, 1994; 10(1): 53-67.

Rosenbaum S, Darnell J. An Analysis of the Medicaid and Health Related Provisions of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996. Health Policy and Child Health. Center for Health Policy Research. The George Washington University. Washington, D.C. Summer 1996:3(3): 1-12.

Rosenbaum S, Hughes D, Butler E, Howard D. Incantations in the dark: Medicaid, managed care, and maternity care. The Milbank Quarterly 1988;66:661-693.

Sarndal C, Swensson B, Wretman J. Model Assisted Survey Sampling. New York, NY: Springer-Verlag; 1992.

Schejbal J, Lavrakas P. Panel Attrition in a Dual-Frame Local Area Telephone Survey. Proceedings of the Section on Survey Research Methods, American Statistical Association 1995; 1035-1039.

Schulman ED, Sheriff DJ, Momany ET. Primary care case management and birth outcomes in the Iowa Medicaid program. American Journal of Public Health 1997;87:80-84.

Scribner R. Paradox as paradigm - The health outcomes of Mexican-Americans. Am J Public Health 1996;86:303-305.

Shah B. Software for Survey Data Analysis (SUDAAN). Version 6.40. Research Triangle Park, NC: Research Triangle Institute; 1996.

Silberman P, Ricketts TC. How well does North Carolina protect enrollees in HMOs? Chapel Hill, NC: Cecil G. Sheps Center for Health Services Research/North Carolina Institute of Medicine. 1997.

Sirken M. A Cognitive Approach to Designing Survey Questions. Paper presented at the Winter Conference of the American Statistical Association, San Diego, CA, 1989.

Sirken MG, Casady RJ. Sampling variance and nonresponse rates in Dual Frame, mixed mode surveys. Telephone Survey Methodology 1988.

Sirken MG. Household surveys with multiplicity. Journal of the American Statistical Association 1970; 65:257-266.

Skinner CJ, Rao JNK. Estimation in Dual Frame Surveys with Complex Designs. Journal of American Statistical Association 1996; 91(433):349-356.

Solis JM, Marks G, Garcia M, Shelton D. Acculturation, access to care, and use of preventive services by Hispanics: findings from HHANES, 1982-84. American Journal of Public Health 1990; 80(suppl):11-19.

Sparer MS. Devolution of power: and interim report card. Health Affairs 1998;17:7-16.

Spreen M. Rare populations, hidden populations and linktracing designs: what and why? Bulletin Methodologie Sociologique 1992; 36:34-58.

Sudman S, Bradburn NM, Schwarz N. Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco, CA: Jossey Bass, Inc.; 1996.

Sudman S, Sirken MG, Cowan CD. Sampling rare and elusive populations. Science 1988; 991-995.

Sudman S, Sirken MG, Cowan CD. Sampling Rare and Elusive Populations, Science 240: (4855) 991-996 MAY 20 1988

Sudman S. The use of network samples in estimation the incidence of Missing Children. Proceedings of the Section on Survey Research Methods, American Statistical Association 1986; 159-163.

Suthutvoravut S, Hogue CJ, Guyer B, et al. Are preterm black infants larger than preterm white infants, or are they more misclassified? J Biosoc Sci 1989;21:443-51.

Tourangeau R. Cognitive Sciences and Survey Methods. In Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines (Jabine T, Loftus E, Straf M, Tanur J, Tourangeau R, eds.) 1984; 73-100.

Traugott MW, Goldstein K. Evaluating Dual Frame Samples and Advance Letters as a means of increasing response rates. Proceedings of the Section on Survey Research Methods, American Statistical Association 1993; 1284-1286.

Tucker C. Characteristics of commercial residential telephone lists and Dual Frame designs. Proceedings of the Section on Survey Research Methods, American Statistical Association 1989; 128-137.

Van Meter KM. Methodological and design issues: techniques for assessing the representatives of snowball samples. National Institute on Drug Abuse: Research Monograph Series 1990; 98:31-43.

Watkins E, Larson K, Harlan C, Young S, Gilbertson S, Nunez MR, and Wenrich S (1990). Improving the Health of Migrant Mothers and Children, Final Report, SPRANS Grant #373415, Department of Maternal and Child Health, School of Public Health, University of North Carolina, Chapel Hill, NC.

Watkins EL, Larson K, Harlan C, Young S. A model program for providing health services for migrant farmworker mothers and children. Public Health Reports 1990; 105:567-575.

Wellens TR. The Cognitive Evaluation of the Nativity Questions for the Current Population Survey. Proceedings of the Section on Survey Research Methods, American Statistical Association, 1994; 1204-1209.

Willis GB, Royston P, Bercini D. The Use of Verbal Report Methods in the Development and Testing of Survey Questionnaires. Applied Cognitive Psychology 1991; 5: 251-267.

Wilson TD. The Proper Protocol: Validity and Completeness of Verbal Reports. Psychological Science 1994; 5: 249-252.

Wolter K. Introduction to Variance Estimation. New York, NY: Springer-Verlag; 1985.

Zambrana R, Scrimshaw S, Collins N, Dunkel-Schetter C. Prenatal health risk behaviors and psychosocial risk factors in pregnant women of Mexican origin: The role of acculturation. Am J Public Health 1997;87:1022-1026.

Sampling and Measurement Issues in Studying the Perinatal Health of Latina Women