[This page, http://www.psywww.com/selfquiz/aboutq.htm], is about writing multiple choice questions which are fair but hard to guess. It might be of interest to my students--for example, to warn them away from the usual guessing strategies discussed below--or to visiting educators. Comments are welcome.
Multiple choice questions are widely scorned as "multiple guess" questions. Some teachers assume that multiple choice items encourage superficial studying. Perhaps it is true...other things being equal, students do not study as hard or as well for a multiple choice test. Certainly if students are expecting a multiple choice test and they receive an essay test instead, they complain.
However, essay tests have their own set of problems. For example...
Despite all this, essay and short answer tests have many virtues. Students need practice formulating arguments, expressing things clearly, and integrating ideas. Nobody would argue that all testing should be multiple choice. However, for many teachers in many situations, a good objective test is both fairer and more efficient than an essay or short answer test.
I maintain it is possible to construct multiple choice questions which are not readily guessed and which therefore require a student to comprehend basic factual material. Because multiple choice tests lend themselves to machine scoring, students can be offered tests over small chunks of material (which has been shown to promote learning) and students can be offered frequent re-tests or makeup tests (to reduce anxiety and promote mastery of the material).
The key to writing "good" multiple choice questions is to (1) specify objectives so the questions can be specific and focused and the student does not have to guess what to study; (2) reduce frustration for creative students by reducing ambiguities (such as "both a & b" type answers which almost always subject to argument); and (3) defeat the "test-wise" strategies of the student who has not studied and is attempting to guess answers. Let's consider these in turn.
In my opinion, students should not be forced to guess what will be on a test, or "psych out" the teacher to decide what to study. Educational research shows that the less able students are heavily penalized by a failure of a teacher to spell out what is required for a test. The more able students seem to sense what the teacher wants, but the students most in need of help are likely to flounder even more painfully if they must guess what to study.
The obvious solution to this problem is to give students specific study questions, then draw the test from the study questions. But sometimes people criticize the idea of "teaching the test," as if having study questions in itself encourages a superficial approach. That may be true if there are very few study questions. However, if a teacher offers study questions for all the most important ideas in an assignment, then "teaching the test" is "teaching the course." In Keller Plans, an independent study method popular in the 1970s, students were given lots of specific objectives or study questions, and lots of opportunities to take quizzes covering that material, and the results were very good...in fact, Keller Plans were one of the few educational innovations in the entire 20th Century which produced better results that traditional lecture/discussion methods of instruction.
I avoid "all of these" and "none of these" and "both a & b" answers. Over the years I have noticed that my best students hated them. The more a person knows about a subject matter, the easier it is to make arguments in favor of answers somebody else might regard as wrong. True/false questions are the worst of all in this regard. Often the truth value of an isolated statement is quite debatable! It all depends on how it is interpreted, the definition of a key term, or complexities of context. True/false questions also raise the problem of response bias: some people are consistently more inclined to answer True or False. Questions with the "both a & b" type answer put a student in the position of making True/False evaluations. Better to have four or five alternatives with only one correct: a classic "force choice" procedure which eliminates response bias.
The whole point of testing is to encourage learning. If students can guess the answers to a test, they will not study for it. Therefore, to motivate students to study and learn, one must design quiz items that are not easily guessed without good studying. One must also design a test so that answers are not obvious to the student who has merely skimmed the assignment, or studied only highlighted words, or read only summaries.In order not to encourage superficial studying, one must defeat the common "rules of thumb" which students use to guess correct answers. When these are eliminating, learning the material becomes the easiest way to pass a quiz.
Rule of thumb: "Pick the longest answer."
Way to defeat this strategy: make sure the longest answer is right about a fifth of the time (if there are five alternatives for each question)
Rule of thumb: "Pick the 'b' alternative."
Way to defeat this strategy: make sure each answer is used the same number of times, in random order.
Rule of thumb: "Never pick an answer which uses the word 'always'
or 'never' in it."
Way to defeat this strategy: make sure such answers are correct about a fifth of the time
Rule of thumb: "If there are two answers which express opposites,
pick one or the other and ignore other alternatives."
Way to defeat this strategy: sometimes offer opposites when neither is correct.
Rule of thumb: "If in doubt, guess."
Way to minimize the impact of this strategy: use five alternatives instead of three or four
Rule of thumb: "Pick the scientific-sounding answer."
Way to defeat this strategy: use scientific sounding jargon in wrong answers
Rule of thumb: "Don't pick an answer which is too simple or obvious."
Way to defeat this strategy: sometimes make the simple, obvious answer the correct one.
Rule of thumb: "Pick a word which you remember was related to
Way to defeat this strategy: when drawing up distracters (wrong answers) use terminology from the same area of the text as the right answer, but in distracters use those words incorrectly so the wrong answers are definitely wrong.
I draw up study questions which cover virtually every important concept from the chapter. This results in about 110 study questions per chapter. In the "old days" when our university was on the quarter system, with classes meeting daily and a typical student taking three classes, the number was 160. Now, with each student taking five classes, and each class meeting only two or three times per week, 110 questions per week is plenty. That is also enough to give complete coverage of a moderately sized textbook chapter.
I draw up a multiple choice question or two for each study question. If I can't come up with a good test item for a study question, I delete the study question. I use questions with five alternatives, rather than four. That reduces the likelihood of guessing the correct answer.
I avoid "all of these" or "none of these" or "both a & b" type answers for the reasons discussed above (I found that excellent students could often come up with creative reasons to pick the wrong answers). I just use five different answers, and only one is correct.
I use quotation marks and scientific sounding jargon in wrong answers, just as often as I do in correct answers. So these are not effective cues
Using each study question as a starting point, I construct plausible sounding alternatives which are supposed to be clearly wrong...but which might sound right to a poorly prepared student. I also do an item analysis at the end of the term and delete questions which are missed by top-level students. If for some reason a question is inscrutable to the top students in the class, then either there is something wrong with the question or the level of difficulty is unreasonable, or the material is not being explained very well. In any event, the question should go.
For what it's worth, I find that I write better questions if I do so with the book closed, working from a list of study questions alone. If I am looking at the answer in the text while I write the question, the details are right in front of me and I am more likely to write a picky question which requires students to have a photographic memory. With the book closed, I must rely on my own memory of the material. I figure if I cannot remember something myself, it is not reasonable to ask students to remember it. This means I have to double-check later to make sure my own memory of the material was correct, but it is worth the trouble because the resulting questions are more reasonable.
The result of this whole procedure is quiz items which are hard to guess unless the student truly understands the material. My "validation" for this procedure is informal: I get very consistent results term after term, and I notice that students who do poorly on my quizzes generally cannot talk about the material either. Yet if they sit down, book in hand, and compare the quiz item to the study question and the material in the text, the answer is obvious and the student seldom complains that the test item is unfair. This leads me to think the test items are doing their job.
Write to Dr. Dewey at firstname.lastname@example.org.
Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.