Spending wisely: investigating survey mode effects in discrete choice experiment responses

We compared four survey modes: internet panel survey, mail survey, mail invitation to complete an internet survey and in-person interviews andcompared responses to a survey designed to elicit preferences of a healthcare ‘good’ likely to be relevant to all members of the population: the use of community pharmacies for managing minor illness. Preference data were collected using a DCE. For each mode, we considered: 

  1. How representative of the population were respondents to each mode?

  2. Did respondents’ preferences and willingness to pay vary across modes?

  3. Could statistical techniques be used to take account of differences in respondent characteristics?

  4. Did response validity vary across modes?

Outcome and Translation

The mail invitation to complete an internet survey was not taken forward to the main study after an extremely low response rate to the pilot. None of the modes were representative of the general population. Each mode differed from the general population in different ways. For example, while respondents to the mail survey were older, on average, than the general population, respondents to the internet panel surveys were younger. Respondents’ preferences and willingness to pay differed across modes. Response validity also differed across modes. The results provide researchers with a characterisation and quantification of the advantages and disadvantages of each mode and thus allow them make an informed decision about which mode(s) to use in their research.

HERU researchers involved in this research project: Mandy Ryan and Verity Watson

External collaborators: T Porteous (Academic Primary Care, University of Aberdeen)


Porteous, T., Ryan, M., Bond, C., Watson, M. and Watson, V. (2016) 'Managing minor ailments: the public's preferences for attributes of community pharmacies. A discrete choice experiment', PLoS ONE, 11(3), e0152257.

Porteous, T., Ryan, M., Bond, C., Watson, M. and Watson, V. (2016) 'Think pharmacy: making community pharmacy the first port of call', HERU Policy Brief, University of Aberdeen, October 2016.

Watson, V.Porteous, T., Bolt, T. and Ryan, M. (2019) 'Mode and frame matter: assessing the impact of survey mode and sample frame in choice experiments', Medical Decision Making, 39(7), 827-841.


Watson, V., Porteous, T. and Ryan, M. (2013) 'Survey mode effects when eliciting public preferences for health care: a pilot discrete choice experiment study', Health Economists’ Study Group, University of Warwick, June 2013.

Porteous, T., Watson, V., Ryan, M., Watson, M. and Bond, C. (2013) 'Managing minor ailments in the United Kingdom. The public’s preferences for characteristics of community pharmacies: a discrete choice experiment (DCE)', [poster] Royal Pharmaceutical Society Conference, Birmingham, 8-9 September 2013.

Watson, V. (2014) 'Survey responses: the UK mode comparison study', Haindorf Seminar, Hejnice, Czech Republic, 6-9 February 2014.

Watson, V. (2018) 'Survey mode effects on the general public's elicited health care preferences', Network of Alberta Health Economists Health Economics and Technology Assessment Rounds, University of Alberta, Edmonton, Canada, 4 April 2018.