What A Test Can Tell You
Using Student Test Answers to Improve the Test (and the Course)
As every teacher knows, developing good test questions is hard work. Even the experts can’t get it right the first time which is why there’s an experimental section on the SAT (a group of questions that doesn’t count towards the students’ exam grades). They’re constantly trying to create new questions to preserve the integrity of the exam but they also need to maintain the same level of validity and reliability that the previous questions had. Analyzing the performance of the test questions that you’ve created (or that you may have obtained from the textbook publisher) is easier for online assessments since much of the data is already pulled together for you. However, you can go through the manual process of calculating the question difficulty level, the effectiveness of each distractor (the choices that aren’t the correct answer), and the discrimination index (discussed in further detail below) using the instructions outlined in Cornell’s Test Construction Manual. Once you’ve gone through the process of aligning your assessments with your Student Learning Objectives (SLOs), you can then start to evaluate the questions themselves.
Basic Tips for Multiple-Choice Questions
- Keep all language as simple and clear as possible
- Move as much of the answer choice wording to the stem (the question itself) as possible
- Avoid All, Always, Never, etc.
- Make distractors plausible and use at least 3
- Avoid having the correct answer frequently being the longest choice
- Shuffle the choices or use alphabetical order to deter unintentional indicators of the correct response
- Check the grammar between stem and all answer choices
- Avoid using the textbook’s phrasing only for the correct answer (this makes it standout from your own personal diction that’s in the rest of the choices)
- Include multiple questions for assessment of each SLO (the number of questions should be reflective of the importance of the SLO relative to the others, they may all be equal but not necessarily)
Analyzing the Questions in Canvas
Below is an example of the quiz statistics data available for a question in Canvas. You can see only 15 people have answered this question so it’s not much information to go on but I can immediately see that one of the answer choices, “Using the same phrasing as the textbook”, isn’t working well as a distractor so I could change that to “” to see if that performs better in future testing. It’s on the easier side (80% correct and difficulty should range between 30 – 80%) but it’s in the acceptable range and the new distractor could decrease this number.
What is very encouraging about this question so far, is the Discrimination Index (which will range from -1 to +1 and should be at least +0.25). You can see by the DI number for this question (+0.42) and the associated graph that the students who performed well on the quiz as a whole (top green bar which represents approximately the top 26%) got this question right whereas those who didn’t do well on the quiz (black bar on the bottom) got this question wrong. Likewise, students who were in the middle got it right sometimes and wrong sometimes (the green and black bars, respectively, in the middle). This is what we would expect since those that understand all of the material should get it right, those who weren’t paying close attention, didn’t study, etc. get it wrong (along with the other questions), and those that are somewhat knowledgeable will fluctuate. Below are examples of other DI graphs. The first “Bad” graph shows the reverse of what we would expect. In this instance you would want to investigate what the top performers are choosing (Are they all picking the same distractor? Is there something about the wording that makes it a strong choice? Is there a conflict between something said in the lecture and what’s in the book that’s leading to this?). In the “Also Bad” example, you’re likely seeing an area that wasn’t covered well in the instructional material so students are just guessing. This will take further investigation by you to figure out what could be going wrong.
Once you’ve optimized your questions, you will start to see that they will look “easier” over time. This is likely because questions will eventually be shared by students that previously took the quiz (although, we all would like to think we’re just turning into amazing teachers!). You can use question banks to help prolong the life of your questions but ultimately you just need to make new ones every year. Asking TAs to make new questions can help breathe new life into your exams.
For more information and further resources, please visit the CITT webpage.