Summary: GAO reviewed the design of two merit systems review and studies (MSRS) questionnaires, one dealing with the merit system and the other with reprisal.
GAO found that the questionnaires' objectives are very broad and address complex issues. GAO questions the appropriateness of using abstract data to meet descriptive and evaluative objectives. Neither questionnaire adequately provides a baseline or the necessary comparison that would permit an assessment of the relevancy of any problem identified. Questions in both address very abstract concepts, though this is less of an issue in the reprisal questionnaire because several key phrases are defined on the cover. In the merit system questionnaire, respondents are asked for views on statements that are vague and use ambiguous words. One way to minimize the effect of abstract terms and phrases is to define them. While respondents may disagree with the definition, at least they should understand them within the questionnaire context. Both questionnaires contain design flaws which limit the usefulness of the results or make the data very difficult to interpret. Some of the flaws are: (1) biased titles; (2) biased questions that could lead some respondents to give different answers than they would give to a neutral wording; (3) the omission of neutral and escape response alternatives; (4) inadequately defined answer choices where the range of choices is not spread out in equal appearing intervals; (5) tendency of respondent to use response sets; (6) mixed timeframes that are confusing; and (7) lack of adequate space to answer questions. GAO believes that these design problems adversely affect the questionnaires' usefulness in achieving major objectives.
