Today, when decision makers need data, they’re increasingly turning to online surveys instead of more traditional telephone surveys. Why? At first glance, the answers seem obvious. Lower costs. An abundance of software on the market making it easy for anyone to conduct an online survey. Quick turnaround times. So why should anyone consider doing things the old-fashioned way?

In March 2007, Market Street Research, Inc. and Massachusetts General Hospital, a teaching hospital of Harvard Medical School, conducted a study to assess the comparability of online and telephone survey methodologies in researching residents’ image of area hospitals. Marketing professionals will find the results eye-opening.

Two methodologies 
The two studies were designed to be as similar as possible.  They used identical questionnaires and screeners, surveyed the same geographic area (eastern Massachusetts and southern New Hampshire), were conducted at the same time of year (early March), and resulted in completed interviews with the same number of respondents (370). 

Numbers for the telephone survey were generated using random-digit dialing.  For the online survey, respondents were selected from online panels compiled by e-Rewards Market Research.  Panels like this are currently a major sample source for online surveys of the general population.  Some researchers claim that up to 90 percent of online studies conducted in the United States in 2006 utilized such panels.

The telephone survey took longer than the online survey to complete, with data collection lasting 18 days, compared with six days for the online survey.  As expected, the data collection cost for the telephone survey was about three times that of the online survey.  The response rate for the telephone survey was 34.5 percent; for the online survey, 12.2 percent.

Demographic differences 
Findings showed significant differences in the demographic characteristics of the two survey populations.  The telephone survey sample was very representative of the population demographically.  Compared with telephone survey respondents, online survey respondents were younger, better educated, more affluent, more likely to be employed, and more likely to have private health insurance.

Weighting is a common statistical technique used to adjust survey results so that groups are represented to the same extent in the survey as they are in the general population.  Even after weighting the data by age and education (two typical parameters), significant differences remained between the two survey populations.  Online survey respondents with more education, more affluence, who are more likely to have private health insurance make a critical difference in a health care study.

Attitudinal differences
The most critical discovery in the comparison of the two methodologies, however, was that even after the data were weighted, people who responded to the online survey expressed significantly different attitudes about hospitals, compared with people who responded to the telephone survey. 

Online survey respondents tended to have attitudes and preferences about area hospitals that are more common among younger, better educated, more affluent residents – exactly how the online survey population was skewed.  Furthermore, online survey respondents were likely to be more technologically savvy.  A very popular source for online panels in the research industry and one used for the panel in this study, e-Rewards partners with airlines and uses frequent-flier lists as a typical source for recruiting participants.  This creates an added dimension to the unique type of respondent that companies are likely to hear from by using online survey panels.  Not only are these respondents likely to be younger, better educated, and more affluent than the general population, but they are also more likely to be “road warriors.”

The most striking difference between the telephone and online survey respondents lies in their attitudes toward community hospitals versus academic medical centers.  Online survey respondents are much more strongly oriented toward academic medical centers:

-When asked to name the first hospitals that came to mind when thinking about New England hospitals, online survey respondents were significantly more likely to name academic medical centers.

-When asked what New England hospital has the best reputation, online survey respondents were more likely to name academic medical centers. 

-When asked what New England hospital had the best reputation for seven different specialty areas, online survey respondents were more likely than telephone survey respondents to identify academic medical centers for every single type of care.  Telephone survey respondents were more likely than online survey respondents to name community hospitals, or else they were unable to name any specific hospitals.

-When asked what New England hospital they would prefer to use if they needed to be hospitalized for a serious or complicated medical problem, online survey respondents were significantly more likely than telephone survey respondents to identify academic medical centers.  Telephone survey respondents were significantly more likely than online survey respondents to say they would use community hospitals.

Clearly, if the research had relied on the weighted online survey results alone, it would have dramatically overstated the market position of academic medical centers in eastern Massachusetts and southern New Hampshire.

Four problems with online surveys 
The most important issue for organizations to understand when choosing a research methodology is that online surveys skew results in favor of Internet users, which at this point in time isn’t everyone.  Use of the Internet is evolving rapidly but, as a Pew Internet & American Life Project study in the first quarter of 2007 showed, about 29 percent of U.S. adults either did not have computers or they have access to computers but don’t use them for e-mail or the Internet.

Second, no method exists for selecting respondents randomly for survey purposes.  There is no universal list of e-mail addresses that researchers can use to obtain a sample for online surveys.  This problem has been solved for telephone surveys through random-digit dialing (RDD), but as one researcher put it succinctly, “There is no analogue of RDD for e-mail addresses.”

Third, the legitimacy of online panelist responses must be carefully considered.  Telephone interviewers are trained to identify respondents who may not meet the survey criteria (such as men responding to a survey of women) or who are not providing serious answers and to elicit opinions from taciturn respondents. 

Fourth, the telephone interview format itself prevents respondents from rushing through the survey.

However, none of these legitimate measures exist for online surveys, where respondents are compensated for each survey they complete.  The current environment increases the risk of obtaining completed surveys from “professional respondents,” individuals who participate frequently in studies and respond in a very different manner than conventional respondents – at best providing responses that are not seriously considered, and at worst providing responses that do not reflect their own views. 

Finally, the single most important problem with online surveys is the extent to which meaningful, actionable conclusions can be drawn from the data obtained from this type of research.  Online surveys use convenience samples.  With a scientific probability sample, the data can be weighted appropriately so that the sample reflects the general population, and a margin of error can be calculated to know to what extent the survey’s findings are trustworthy.  None of this is possible with online surveys.

Online surveys today are akin to the poll conducted by the Literary Digest in 1936 that confidently predicted a presidential win for Republican Alf Landon over Democrat Franklin Roosevelt.  The Literary Digest’s erroneous predictions were based on a sample size of more than 2 million people, but the sample was drawn from lists of the magazine’s own readers and people who owned cars or telephones.  The sample of fairly well-to-do individuals did not represent a cross-section of America in the midst of the Depression.

The Gallup Poll, on the other hand, today predicts election results with consistent accuracy using samples of 1,500 or fewer voters.  Gallup achieves this level of confidence because it polls the nation by telephone, using scientific sampling methods.

The future of survey research
Survey research is clearly facing some fundamental shifts.  Right now, U.S. households’ access to telephones is nearly universal (about 97 percent as of 2000).  Telephone surveys are expensive to conduct, however, and there are social and technological changes that are making it harder to conduct good quality telephone surveys.  For one, people are increasingly unwilling to cooperate over the telephone, in part because they are called regularly by telemarketers posing as researchers, or they heed media and law enforcement warnings not to give out personal information to avoid identity theft.

For another, households’ use of telephones is changing rapidly – the percentage of U.S. adults who live in households with only wireless telephone service (no land lines) has been steadily increasing since 2003 and has reached nearly 14 percent, or one out of every seven homes.  More and more households have two or more telephone lines, which further complicates sampling.

Internet use is changing as rapidly as telephone use, as more and more U.S. households obtain Internet access.  It is quite reasonable to expect that at some point in the future, U.S. households’ access to the Internet will be as universal as telephone access is now.  As the industry matures, researchers will undoubtedly discover methods for surveying people online that can produce broadly accurate results.

Where online surveys make sense today
There are situations in which online surveys are an excellent, cost-effective methodology.  Online surveys would be a very good choice for an organization that wants to survey a captive audience, for which the entire population has a known e-mail address, such as a company wanting to conduct employee satisfaction studies.  An online survey would also be a good choice for surveying certain rare populations, for which a dual-frame, Web-telephone methodology may be the only effective means of finding eligible respondents. 

Online surveys are also a good methodology when the questionnaire includes a visual or audio component or a large number of open-ended questions.  Here, online surveys have an advantage over telephone surveys (where responses may not be captured in full depending on the limitations of the technology employed or the interviewer’s typing speed) as well as mail surveys (which have been shown to elicit much briefer open-ended responses). 

Necessary shifts
For online surveys to become a reliable method for general population surveys, however, there will have to be a significant change in the ways e-mail addresses are generated, comparable to the changes the U.S. telephone system underwent during the last two centuries.  In 1910, reaching a person beyond the limits of one’s city was almost the same as reaching someone whose e-mail address isn’t immediately known.  It meant telling the operator the desired city, the person’s name, and then, if known, a short alphanumeric sequence associated with the person’s building or residence.  Our current system of telephone numbers – seven digits conveniently tied to geography – wasn’t fully established nationwide until the late 1950s.  If the Internet emerges as people’s predominant means of communication (and if no new technology is developed that people think is better), a similar change in the ways people’s e-mail addresses are assigned could make the Web a formidable resource for surveys.

Until that time comes, however, organizations needing data representative of the general population are well advised to consider the problems endemic to online surveys.  The people who respond to online surveys simply do not reflect the characteristics of the larger population.  They tend to reflect the population of Internet users, which at this point means a better-educated, more affluent consumer.  Surveys conducted exclusively through online surveys are very likely to provide skewed data and provide companies with a false sense of consumers’ needs and how they are perceived by consumers.  Traditional telephone surveys are still the best methodology for achieving a reliable understanding of the beliefs, opinions, and preferences of the general population. 

Julie Pokela, PhD, is president and co-owner of Market Street Research, Inc. (www.marketstreetresearch.com), a Northampton, MA-based marketing research firm founded in 1978.  Elizabeth Denny, PhD, is the company’s executive vice president and co-owner, and Ingrid Steblea is the senior research analyst.  Frank Melanson is senior manager for market research and planning at Massachusetts General Hospital in Boston. 

Back to Main Newsletter Page