Background

Because of increasing numbers of mobile-only phone users worldwide, currently estimated to be 30.2% in the USA [1], 13% in Canada [2], 14% - 19% across the UK countries [3] and 19% in Australia [4], it has become increasingly difficult to produce unbiased estimates from random digit dialling (RDD) surveys that only target landline phones [5–7]. Consequently there is now substantial international literature on conducting RDD surveys with mobile phone augmentation [8–12] and the American Association for Public Opinion Researchers (AAPOR) Cell Phone Task Force recommended in their latest report (2010) [12]: “Random digit dialling (RDD) surveys without cell phone augmentation should in their methods report how they have produced unbiased estimates without the cell phone only segment”.

In Australia landline telephone surveys have been the method of choice for ongoing population health surveys [13–18]. Although the rate of mobile-only phone users was estimated to be nearly 20% in 2011 [4] the inclusion of mobile-only phone users into these existing landline population health surveys has not occurred. Studies describing the demographic, socio-economic and health profile of mobile-only phone users have been conducted and have shown that mobile-only phone respondents were different to those who had access to a landline phone using face-to-face survey data [19, 20] and internet panel data [21].

Two designs for the inclusion of mobile-only phone users into landline RDD surveys have been discussed in the literature: screening dual-frame design and overlapping dual-frame design [6]. The screening dual-frame design attempts to remove any overlap units usually by screening for telephone ownership prior to conducting the survey and then only interviewing mobile-only phone users from the mobile frame. The overlapping dual-frame design accounts for the overlap in the weighting by using an average estimator and a compositing factor. The overlapping dual-frame design, although requiring a more complex weighting strategy, has been growing in favour because it has been shown that persons selected through mobile frames (even if they have both mobile and landline phones) differ to persons selected through landline frames [7].

Two pilots using a dual-frame design had also been conducted in Australia by Pennay in 2010 (700 respondents) and Lui et al. in 2011 (335 females respondents aged 18 to 39 years) [22, 23]. Pennay [22] provided particularly useful statistics for planning this study including: the expected numbers of telephone numbers required to get an interview in each of the frames (landline 12 numbers and mobile 25 numbers) and the expected percentage of interviews with persons from landline-only phone households in the landline phone frame (14.5%), and percentage of interviews with mobile-only phone users from the mobile phone frame (27.6%).

This paper describes the methods used for the inclusion of mobile phone numbers into the New South Wales Population Health Survey (NSWPHS), an existing ongoing landline RDD health survey in an Australian state [13]. This paper also compares the call outcomes, costs and the representativeness of the resultant sample to that of the previous landline sample.

Methods

Survey methodology

Since 2002 the health and wellbeing of the New South Wales (NSW) population (7.3 million) has been monitored using the NSWPHS. A representative sample of approximately 15,000 persons are interviewed each year, with equal numbers from each of the strata (health administrative areas) using landline RDD computer assisted telephone interviewing (CATI). The questionnaire includes questions on: health behaviours, health status, social determinants, demographics and phone ownership. The survey has approval from the NSW Population and Health Services Research Ethics Committee. The questionnaires and the data collection methods are available on the survey website [13].

In order to include mobile only phone users into this existing landline RDD health survey an overlapping dual-frame design was chosen. This allowed us to examine the representation of the resultant sample for both an overlapping dual-frame design and, by excluding persons with both mobile phones and landline phones from the mobile frame, a screening dual-frame design.

Details about the procedures for sample generation, sample design, eligibility, sample size, questionnaire, data collection, calling protocol, participant selection and probability of selection weighting for the previous years’ landline RDD surveys [24–27] as well as for each of the phone frames are shown in Table 1. As shown in Table 1 the procedures were, where possible, consistent with the previous years’ landline RDD surveys and between frames.

Table 1 Comparison of survey methods, 2011 NSW Population Health Survey and 2012 NSW Population Health Survey
Full size table

Call outcomes and costing

Operational data for the survey were downloaded. The data included telephone number, number of attempts, details of each attempt (including duration) and final disposition. Although the final disposition codes used for the survey are site specific they can be easily mapped to the AAPOR definitions [28]. These final dispositions were then entered into the AAPOR outcome rate calculator [29] and all AAPOR levels of response, cooperation, refusal and contact rates were calculated from the groupings of the final dispositions for each frame. Overall rates were then calculated as described in the Non-response in RDD Cell phone surveys chapter of the AAPOR Cell Phone Task Force Report [12] using the latest ACMA figures for Australia (5% landline-only phone users, 19% mobile-only phone users, and 76% both mobile phone and landline phone users) [4].

The productivity (phone numbers to get a contact, an eligible contact, and an interview) of the sample for each frame was examined. Call costs (including connection fee, if applicable) and interviewer costs (hourly rate multiplied by the calling time) for each sample frame were also calculated and presented as a cost per completed interview.

Demographic parameter comparisons

Interview data for the survey were downloaded. The data included a unique identifier, sample frame, strata, and responses to the health behaviours, health status and demographic questions. Demographic information from the mobile frame sample was compared to the landline frame sample using χ 2 tests. Demographic information from the mobile frame sample, landline frame sample, combined landline sample with the mobile-only sample (equivalent to a screening dual frame design) and the combined landline sample and mobile sample with appropriate overlap adjustment was compared to the NSW demographic profile from the 2011 census using χ 2 tests.

Results

In the first quarter of 2012, 3395 interviews were completed with 2171 (63.9%) being from the landline frame of which 382 (17.6%) were landlines-only and 1224 (36.1%) being from the mobile frame of which 316 (25.8%) were mobile-only.

As shown in Table 2, completed interviews from the mobile frame, compared to the landline frame, were slightly shorter (15.6 minutes v 17.2 minutes), cost 2.3 times more for each completed interview ($74.42 v $31.13) and required more telephone numbers to obtain a contact (2.1 v 1.9), eligible contact (10.5 v 7.0) and an interview (14.4 v 9.8).

Table 2 Call outcome information and rates for by sample frame and overall (combined)
Full size table

Outcome rates

Levels of response, contact, cooperation and refusal rates, calculated as per AAPOR definitions, as shown in Table 2 were similar between frames. Overall combined (with adjustment for the overlap) response, contact, co-operation and refusal rates were 33.1%, 65.1%, 72.2% and 17.4% respectively.

Sample characteristics

Table 3 shows respondent demographic profiles for the mobile frame (mobile-only, both and total), compared to the landline frame (landline-only, both and total). As shown in Table 3 the demographic profile of the landline frame responders was significantly different to respondents: from the mobile frame who were mobile-only for age group (p<0.001), sex (p=0.049), Aboriginality (p=0.049), country of birth (p<0.001), and marital status (p<0.001); from the mobile frame who had both mobile and landline phones for age group (p<0.001) marital status (p=0.003) and income (p=0.001); from the mobile frame for age group (p<0.001), country of birth (p<0.001), marital status (p<0.001) and income (p=0.01).

Table 3 Comparison of the demographic profile of the mobile frame and the landline frame respondents
Full size table

Table 4 shows respondent demographic profiles for the landline frame, mobile frame, the landline frame with the mobile-only respondents from the mobile frame, the combined frames (using λ=0.5 as the compositing factor), and the NSW demographic profile from the 2011 census [30].

Table 4 Sample comparisons to the latest population profile for NSW
Full size table

As shown in Table 4 the NSW demographic profile was significantly different to respondents: from the landline frame for age group (p<0.001), sex (p=0.037), country of birth (p=0.02), marital status (p<0.001) and income (p=0.015); from the mobile frame for age group (p=0.03) and income (p=0.04); from the landline frame plus mobile-only phone respondents for age group (p<0.001), marital status (p=0.01) and income (p=0.02); and from the combined frame for age group (p=0.01).

Discussion

When mobile phone numbers were included in the first quarter of 2012 into the NSWPHS using an overlapping dual-frame design, 3395 interviews were completed with just under two thirds from the landline frame and just over one from the mobile frame. Interviews that resulted from the mobile frame, compared to the landline frame, were slightly shorter, cost 2.3 times more for each completed interview and required more telephone numbers to obtain a contact, eligible contact and an interview. Response, contact and co-operation rates were similar between frames. Overall combined response, contact and cooperation rates were 33.1%, 65.1% and 72.2% respectively. As expected from previous research [19–23], the demographic profile of the mobile-only phone respondents differed most (more that were young, males, Aboriginal and Torres Strait Islanders, overseas born and single) compared to the landline frame responders. The demographic profile of respondents from the two frames combined, with appropriate overlap adjusted, was most similar to the latest NSW population profile.

The inclusion of the mobile phone number was logistically very challenging with the biggest challenge being the lack of geography on the mobile frame which resulted in more time and resources being spent on calling ineligible numbers (persons who reside outside NSW). The inclusion of mobile phone numbers in the NSWPHS however is still cost-effective because of the additional interviews that were conducted with young people, Aboriginal and Torres Strait Islanders and people who were born overseas resulting in a more representative sample. This however may not be the case for smaller states where the cost of excluding ineligible (out of state) persons may be prohibitive.

As this study is mainly descriptive there is a need to further examine if using a different overlap adjustment factors would have impacted on the results. Further work also needs to occur with the sample frame provider to minimise the number of invalid and ineligible number (predominantly business numbers) to improve the efficiency of the data collection.

Early results are now becoming available from stand-alone surveys of the Australian population that are including mobile phone numbers using various designs [31–33] and so we are slowly getting more experience in Australia on conducting RDD surveys with mobile phone augmentation. There is still a need for more detailed methodologies to be provided. So hopefully this study, and the work we are undertaking on weighting strategies for the NSWPHS and an examination of the impact of the design change on the time series, will contribute to a better understanding of how to conduct RDD surveys with mobile phone augmentation in Australia.

Conclusions

The inclusion of the mobile phone numbers, through an overlapping dual-frame design, did not impact negatively on response rates or data collection, and although costing more the design was still cost-effective because of the additional interviews that were conducted with young people, Aboriginal and Torres Strait Islanders and people who were born overseas resulting in a more representative overall sample.

Authors' information

MLB is a PhD student with the Centre for Statistical and Survey Methodology, University of Wollongong, Wollongong, Australia.