Quizzes using STACK for Physics

Online tests or quizzes had been used in Liz’s courses for several years allowing for constant refinement. She now runs a mixture of in-class and take-home tests in order to assess over a thousand students in Physics 1A (PHYS1121), Higher Physics 1A (PHYS1131), Physics 1B (PHYS1221) and Higher Physics 1B (PHYS1231). There is an online quiz available for students for each week of the course worth 10% in total of their final grade, and there are also two in-class invigilated tests of a similar format worth 10% of their final grade each (20% total) deployed in weeks 6 and 10 to test students on each half of the course. All quizzes are built using the STACK question type and are made available to students via Moodle. 

Liz began by outlining the learning objectives for each lecture in the course explicitly, then wrote five questions based around those learning objectives to add to an overall bank of questions. The bank has since grown to 180 questions that each comprise on average 3 parts, accepting algebraic, numeric and multiple choice responses and can be delivered to students in pre-determined sequences using variables to ensure unique experiences for each student. The questions were programmed up with feedback, this is made available to students during their attempts of the at-home quizzes, for the in class test the feedback is available once the student submits their final answers. The in-class tests use the same bank of questions and give the students four questions for each test with 40 minutes to complete them. They are deployed using the Safe Exam Browser to ensure students are locked to the quiz screen. Students are given a formula sheet and spare paper for their working out. 

Overall, the data shows student performance has improved by as much as 12% using the combined digital approach. 

The initial work of creating the bank of questions in STACK was stage one. Now that data has been collected on average student efficacy for each question in the bank, questions are carefully categorized for the in-class quiz to ensure students all receive an equal amount of questions of varying difficulty. 

The courses are set up as standard in Moodle, and the take-home quizzes are made available each week of the course. Students are free to complete the quizzes at any stage during the week, with each one due by Sunday night. Students receive a small penalty for each incorrect attempt at a question but can retake the whole quiz as many times as they like with only their highest mark being recorded. Dynamic variables ensure that students never receive the same question as each other, and ensures students are learning the method rather than just memorising answers. All questions are automatically marked.

The in-class tests are run twice for each course and are also deployed via Moodle (link only opened at the start of the class and is only accessible from computers in the first year physics lab). Some IT admin will be required before the test day as well as a dry-run to ensure a smooth deployment using the Safe Exam Browser to lock the students to the quiz screen. They are deployed in a standard computer lab and require at least one staff member for invigilation. Each terminal is provided with a formula sheet and spare paper for working out. Students are permitted to use a scientific calculator. A slide is displayed on the projector screen with the rules and instructions for the test: 

Students are required to have their student ID card out on the desk for checking by the invigilator throughout the test. Once students have individually launched the test, Moodle times them for 40 minutes and will automatically submit their tests should the time limit be reached. As for the take-home quizzes, the questions are randomised and use dynamic variables so a student will find it extremely difficult to cheat off another student’s screen. Once students complete the test the results are located within Moodle for analysis and dissemination to students.  

Pros

• Dynamic variables and randomised sequencing ensure unique student experiences and increases the usefulness of every question in the bank; 
• All quizzes are automatically marked with grades stored in Moodle; 
• STACK allows for algebraic responses from students, randomised variables, randomised strings, multi-part linked questions in a bank as well as the ability to use carry on marking when a student makes a mistake; 
• Student performance has improved.  

Cons

• Technical issues have been faced, for example, updates to Moodle may break something in STACK; 
• No ability to categorise questions in the bank based on multiple criteria, for example, based on difficulty and corresponding lecture, the work around is to have multiple copies of each question but this means any updates need to be made in multiple places; 
• Took sizable amount of time initially to write all the questions for the two separate banks – 180 hours for each bank. 
• Requires involvement from UNSW IT for each in-class test to run smoothly via the Safe Exam Browser (SEB). 

• Always provide a practice quiz or quizzes for students to use; 

• Set challenging questions for the students but provide useful feedback so that they can learn from their mistakes; 

• Make sure that your question bank is large enough, small banks encourage students to memorise the answers rather than concentrating on learning the core concepts and how to apply them; 

• Ensure that computer labs are booked far enough in advance; 

• Ensure UNSW IT is given enough time to assist in the set up (at least a few weeks but the more notice the better); 

• Questions can be grouped by “difficulty” and then distributed in equal measure to students, so they all receive a fair amount of each; 

• An example run sheet with instructions for invigilators has been attached below in the “Showcase” section. 

• You can download and view a presentation Liz made on these quizzes here.
• You can also download and view a "run-sheet" document which provides instructions for invigilators here.
• Liz has also made a screencast video showing you what the in class tests look like when deployed using the Safe Exam Browser: