Observational Assessment App for clinical assessments

Since 2012, UNSW Medicine has been utilising the Observational Assessment Application (OAapp) to greatly assist in the management, deployment and assessment of Medicine students undertaking clinical examinations in our affiliated hospitals, all over NSW. IN recent years, the OAapp has been used for examinations with similar parameters: high stakes (summative) assessments of many students (hundreds), rotating through multiple ‘stations’ to complete certain tasks. These assessments invariably run over several days, and have multiple sessions within each day. Both students and examiners need to be allocated to particular sessions. Secure collection and transmission of assessment data is mission critical.

Within the Medicine program, there are a series of examinations which require examiners to directly question or observe students in completing various tasks across numerous settings ranging from facilities on the university campus to clinical examination rooms in hospitals. A student’s performance is recorded

on an appropriate grading document by the assessor. Traditionally, these examinations required considerable administrative effort in the preparation of the exams, their implementation and the subsequent collation and accurate reporting of grades to the Faculty office. The process entailed the printing and distribution of information and grading sheets to the examiners. Grades were recorded on printed sheets by examiners and would then need to be transcribed to electronic spread sheets. This increased the risk of transcription errors, not to mention the time required for manual transcription of data.

Given that these examinations would take place across a variety of locations including onsite at hospitals, UNSW Medicine required a mobile solution that would allow examiners to assess students at any time within any setting. This solution took the form of the natively built OAapp for iOS devices, allowing examiners to access a secure location for all the required assessment materials, use them during the assessment to record marks and feedback for students, then have them synchronised to Faculty records and subsequently processed for the students. Due to wifi inconsistencies in many locations (esp. hospitals) recording of results by examiners can occur offline, with synchronisation occurring later.

Back end administration is handled via automatic synchronisation between the application and the Medicine LMS eMed, in which digitised assessment items are housed in an Assessment Item Bank. This ensures students and assessors are automatically loaded into the application as well as the correct assessment material and forms. The clinical examination is set up the same way as it was traditionally, for example, with multiple assessors spread across different “stations” in a particular location. Multiple locations can be set up for standardised examinations to occur simultaneously across these sites.

Assessors have access to iPads that have the OAapp installed and has been checked beforehand by an administrator. Students cycle through the stations and are assessed on their performance, whether in observed clinical activities e.g. patient care or on viva stations perhaps involving documentary or image based information to which students have to respond. Assessors use the OAapp to navigate to the correct assessment form for that particular student and input each student’s grade into pre-configured fields. Assessors are also given specific fields to enter qualitative data for each student.

Once all students have been assessed the student’s results can be uploaded to Faculty’s central systems.

Pros

• More efficient and reliable delivery of information about the examination, including essential information for administrative staff, examiners and students and any resources e.g. X-ray images for use during the examination;

• Improves the validity of the examinations by ensuring that all sites use the same information, provides examiners with standardised grading sheets which include guidance on grading;

• Ensures completeness and validation of results entered by examiners (warnings pop up if fields are left incomplete) and negates the risks of transcription errors;

• Does not need a persistent internet connection – only before the assessment to download any required data, and at some stage afterwards to upload results. Data is stored safely and securely on the device while waiting to be uploaded. This is very useful in hospitals with patchy Wi-Fi;

• Using an electronic means for recording feedback results in clear, easy to read content for students, which acts as a further learning opportunity – one that was frequently lost when such comments were handwritten on paper.

• Reduces administrative burden as well as excessive use of paper.

Cons

• The app required considerable time and funding to initially develop, and was built by an external agency albeit integrated with Medicine IT;

• Good communication regarding digitisation of assessments and modest training of staff is required to ensure assessors are happy, and know how to use the app;

• The iPads with the app need to be checked and configured by administration staff before the assessment is run I.e. administrative input is reduced but negated.

• The OAapp is the delivery system, but an existing assessment item bank is required, or needs to be built prior to, or alongside, deployment of the app.

The OAapp has been custom built for UNSW Medicine but a similar solution could be applied to other domains. Many tools currently already exist for the creation of electronic forms on separate platforms, to even simple editable PDFs. Depending on user needs, it may be possible to obtain similar functionality from Qualtrics, Microsoft Forms and others. They are also an increasing number of highly sophisticated, enterprise level system solutions – but at a cost!

• Communication with all stakeholders is important – Faculty, administrative staff at distant locations, their examiners and students.

• Ensure that all assessors have had the appropriate training on using the app (or similar solution) well before the assessment is run.

• More information about using the app and its development is best organised through meeting with Dr Silas Taylor ([email protected]).