Context
I lecture statistics in the School of Psychology within the Faculty of Science at the University of New South Wales. I have designed, developed, and currently teach a statistics component of a large blended learning first year statistics and research methods course. In addition to teaching statistics, I am a convenor and one of the designers of a new online degree in psychology offered to new students from March 2019, Graduate Diploma in Psychology. This program will enable graduates in other disciplines to complete an accredited UG sequence in Psychology, allowing them to progress to further accredited training in professional psychology (e.g. Honours, Masters and research degrees).
The design and implementation of statistics as a blended course was particularly challenging as our first year students have a varied level of mathematical knowledge and very often bring to my class a great fear of numbers, as well as the expectation that statistics is boring and difficult. As a lecturer and a course designer, I aim to provide my students not only with the knowledge that will enable them to understand statistics and confidently perform their own statistical analyses, but also to show them that statistics is interesting and numbers should not be feared. The online component of the course is designed to allow and encourage continuous engagement with the content through additional reading, interactive modules, online tutorials, formative and summative assessments, and communication with other students enrolled in the course. This is the component of the course where students can learn at their own time and pace accessing content at different levels of difficulty. The face-to-face component of the course is designed to provide students with the opportunity to apply, extend, communicate and assess their understating of statistics in the classroom environment.
In addition to teaching and evaluation of my statistics course, I will perform the evaluation of the Graduate Diploma program. The advantage of creating a new degree from the evaluation perspective is in that the evaluation of the entire program can be planned simultaneously with the content development and implementation, and performed in various stages of the program delivery.
Tool
(1) In the face-to-face tutorial setting: I evaluate the online component of the course in the face-to-face tutorial setting where students are required to practice and apply the concepts they acquired online using new examples and solving new problems. In addition, I evaluate the effectiveness of their online learning by asking them to critically evaluate research papers or design their own experiments.
(2) Student feedback: I use Qualtrics surveys to ask my students to provide me with a specific feedback for individual learning activities, assessments, and my teaching.
(3) Learning analytics: One of the main concerns in blended and online learning is the lack of student engagement with the online materials. For that reason I use the learning analytics collected by Moodle and Smart Sparrow platforms to observe how much time student spend on individual activities, what is the completion rate of these activities, and how frequently students engage with specific activities. I then observe how these measures of students’ engagement and performance in formative assessments are correlated with their performance in individual summative assessments and their overall course grade.
(4) Item analysis: Finally, I am analysing the exam questions using the Rash model for item analysis. In this analysis the probability of a correct answer is modelled as a function of an individual student and the question parameters. This information allows me to evaluate and improve the formative and summative assessments and to provide students with more personalised learning based on their individual needs.
(5) I will evaluate the Graduate Diploma program using Kirkpatrick’s four levels of learning evaluation: (1) Reaction, (2) Learning, (3) Behaviour and (4) Results.
Results
The online content delivered in the blended learning environment allows students to learn new concepts and review the previous knowledge at their own pace and time. The face-to-face component of the course is then efficiently used to practice and apply that knowledge, communicate new concepts in a different context, and encourage students’ active participation in classroom discussions, group and individual problem solving, and assessments. Based on the student feedback and learning analytics I continuously improve my course by adding new supplementary materials and changing delivery methods. Finally, the aim of the item analysis is to provide me with an objective measure of the question difficulty and to help me develop more precise tools for the student assessment and learning.
Impact
By the end of each semester the students who had negative expectations from the course and expressed the fear of statistics, not only overcome their anxiety, but also show increasing engagement with the materials and increasing involvement in the classroom activities. This also leads to improved performance in the assessments and provides a good foundation knowledge for the advanced statistics courses in the second and third year, and enables them to independently perform statistical analysis for their research projects. I have also used the experience with blended learning in teaching statistics to develop content for a number of fully online courses in the Graduate Diploma program.
References
Kirkpatrick, D., & Kirkpatrick, James D. (2006). Evaluating Training Programs: The Four Levels. Chicago/Turabian
Wu. M & Adams, R. (2007). Applying the Rasch Model to Psycho-Social Measurement: A Practical Approach. Educational Measurement Solutions, Melbourne. http://www.edmeasurement.com.au/_publications/RaschMeasurement_Complete.pdf
TAM R package tutorials http://www.edmeasurementsurveys.com/TAM/Tutorials/
Qualtrics: https://teaching.unsw.edu.au/qualtrics
