Conducting Effective Surveys - 'Rules of the Road'
While frequently used for academic and marketing purposes, community or program surveys are increasingly being used as a means for gathering input and informing choices on a wide range of public decision-making topics and concerns. Done correctly, surveys can provide you with a cost effective, unbiased and accurate picture of the perspectives and priorities of your stakeholders - whether they are constituents, taxpayers, or program clientele. Now more than ever community or program surveys are being used to improve decision-making, assess organizational or government performance, assess program or community needs and priorities, gauge satisfaction, increase public participation, and improve civic discourse.
It is critical to recognize, however, that to be reliable a survey must adhere to a number of well-established and research-based principles. Knowing and incorporating these is essential to ensuring your survey serves your purposes and does so in a way that enhances public trust and confidence. Not being aware of, or incorporating these considerations will almost certainly mean that your survey is not as reliable, accurate, or cost effective as you need it to be - and may even cause more harm than good.
Critical Questions
Once you have established the goals and rationale for your survey you will need to confront two primary questions: 1) how much accuracy and reliability do you need; and 2) what type of survey best serves your needs and budget.
What Level of Accuracy and Reliability Do You Need?
In most cases the purpose of conducting a survey is to collect the responses of a small sample of individuals whose responses are then assumed to be representative of the population from which it is drawn. Anything in either the content or process of the survey that works against ensuring that representativeness erodes the confidence we can have in the accuracy of its results. The vast majority of survey flaws can be categorized into two types: selection bias and response bias. In effect, virtually every rule about conducting quality research is to avoid one of these two pitfalls.
Selection bias refers to the innumerable biases inherent in selecting a sample that is not, in one way or another, representative of the population as a whole. Complicating efforts to eliminate selection bias is the fact that there are many ways it can creep into your survey effort. The easiest way around most of these is to make every effort to ensure that the sample is chosen carefully, is large enough and is randomly selected.
Response bias refers to the myriad of concerns surrounding question construction and wording - whether it be in the form of a mailed questionnaire or a telephone survey. Leading questions, non-neutral questions, questions with more than one interpretation, and questions that deal with more than one subject are just a few examples of questions that will cause responses to be biased - and thus cannot be assumed to be representative of the population at large.
Fortunately, the research focusing on effective survey implementation gives us several important tools and techniques for avoiding many of these errors. Since those listed here are the most widely recognized and critical, ignoring even one of them is to call into serious question the effective representativeness and accuracy of your survey effort.
Which Type Of Survey Is Most Appropriate To Your Needs?
Every type of survey method has its advantages and disadvantages depending on the circumstances under which they are applied, the types of information required as well as the budget you have available.
Mail surveys can either be based on a random sample representative of the larger population or sent to everyone in the target group depending on the size of that group. Mail surveys can be as elaborate or brief as necessary depending upon your needs and target population. This type of survey has the advantage of being reasonably easy to conduct - depending, of course, upon the size and complexity of the survey. They also can provide you with the opportunity to gather information from a fairly large number of respondents - or, if done correctly from a relatively small percentage of the total population yet in a way that still accurately reflects the information that would be gathered from that total population.
Telephone surveys share many of the same advantages and characteristics that mail surveys do but are often more expensive primarily because you will likely have to pay survey interviewers. They can also face the challenge of representativeness since gathering a comprehensive and effective list of phone numbers for your population can be difficult.
Internet and electronic surveys. While internet surveys can be relatively inexpensive to conduct they obviously suffer from the fact that respondents will be limited to those with computer and internet access and experience. While the percentage of Americans who meet these criteria is growing every year, we are not yet at the point where this can be effective choice for a survey intended to draw form the population at large. Electronic survey can be constructed in much the same way as a mail survey. In addition software developments are quickly evolving to make this an attractive option for some applications and audiences.
Personal, face to face interviews can also be effective. More than most other methods, personal interviews provide a means for gathering in-depth information from respondents. Personal interviews offer surveyors the opportunity to ask follow-up or clarifying questions. They are generally conducted from a prepared questionnaire form so that questions and responses are consistent and reliable. While interviews may provide the best opportunity to acquire in-depth information, they also suffer from the disadvantage of being time consuming, expensive (particularly if interviewers are being paid) and logistically complex - but by no means impossible - to manage effectively.
Random Selection
Perhaps the most important thing to remember about doing survey research is that the results of a sample survey can only be generalized to the population from which it is drawn. For example, one could not survey or interview a sample of those registered as either Democrats or Republicans and then make the claim that all registered voters feel one way or another about a given issue. The only claim one could make, providing the survey was done correctly, is that its results were representative of voters registered in that particular party - not registered voters in general.
Thus, if the purpose of conducting a survey is to reflect the accurate feelings of an entire population, the sample on which it is based must be chosen randomly from among all those in the population. The underlying assumption is that the differences that exist in the general public (socially, politically, culturally, educationally, financially, etc.) will also be reflected in the sample chosen for the survey. Choosing a random sample, however, must be done carefully to avoid injecting numerical biases into the sample.
Make sure your sample gives each member of the entire population an equal chance of being selected.. A simple example is that if you wanted to select a sample of 50 people out of a total population of 200, you cannot simply choose the first number randomly and then select the next 50 names. For obvious reasons, many within the total population should not have had an equal chance of being selected. In this case, a better way to proceed would be to either choose each of the 50 names out of a hat or choose the first one randomly and then select every fourth name (200/50=4). For large samples, there are computer programs that will generate random selections or companies that will choose them for you.
It is also important to keep in mind many of the other ways that selection bias can enter your sampling process. For example, even if you have chosen your phone interview sample carefully, but you do all your calling in the mid-afternoon, your sample is likely to be over-represented by women in the home, retired people, and the unemployed, and under-represented by working men.
Also use caution when selecting the source from where the sample is drawn. This is a fairly straightforward idea but one that is often difficult to put into practice in the "real world". Samples drawn from various types of tax records, voter registration lists, telephone directories, utility customer lists, and every other source all under-represent some individuals or groups existing in the population at large. The best that can be done is to be aware of these dangers and what they imply for interpreting the results of a particular survey.
Sample Size
Many people beginning to do a survey are surprised to discover that selecting a sample size is one of the easiest choices they have to make. The fact is that above a certain point, any sample size is effective - as long as other factors contributing to problems have been minimized. The essential issue of sample size is related not to validity, but to precision and accuracy - how confident you can be that the results of the survey reflect the real distribution of opinion in the population. This is the essence of sampling error, the error that arises from trying to represent the entire population with a sample. Unlike in the past, there are a number software programs, private sector companies, public sector organizations, web-based calculators and software programs to help you easily determine the best sample size for your project.
Additional Considerations
There are a number of other considerations you will need to take into account including developing an effective strategy, ensuring effective question and survey construction, and where to get the assistance you need. I'll address these in a future blog. For now recognize that as you are thinking about doing a survey, you will need to take these and other considerations into account if you are to avoid the mistakes that limit the effectiveness of far too many survey efforts. In the meantime, don't hesitate to give us a call to help you think through your next project.
