Research and Trialling
Pilot research study
- The mode effect study investigated how the performance of items (questions) is affected by the mode in which the test is delivered, that is, whether items perform differently in paper-based and computer-based tests.
- In some schools, researchers conducted cognitive interviews with students. This study was a small qualitative study intended to gauge levels of student engagement with the computer-based assessments.
Trialling studyIn August and September 2013, ACARA trialled online delivery of a tailored test design. The tailored tests provided a multi-branching test consisting of interlocking sets of questions. Students answered an initial set of questions and were then directed to the subsequent sets of questions based on the accuracy of their responses. Students with a high number of questions correct were directed to more challenging questions. Students who had a lower level of achievement in the initial set of questions were directed to questions that were less challenging. The tailored test was expected to provide better assessment of individual student achievement relative to ability.
More than 250 schools across Australia volunteered to participate in the study. The study included remote schools as well as a trial with a small number of home-based and geographically remote students. Approximately 23,000 tests were delivered to students in Years 3, 5, 7 and 9, in reading, numeracy and writing. The research study also investigated the cognitive and behavioural engagement of students with the tailored test and the effectiveness of the proposed test design in better targeting underperforming students. ACARA is now analysing the test data and a range of other information collected, including observations by invigilators and technical issues experienced in schools.
ACARA’s research has also investigated autoscoring technology and technically enhanced items that are designed to assess particular aspects of literacy and numeracy aligned with the Australian Curriculum.
Linking studyIn 2014 a linking study will focus on the linking of paper- and computer-based assessment scales. The scope and precise timeline for this study will be determined by the outcomes of the pilot research study and the trialling study.
Research findings will be made available progressively.
For more information about ACARA’s research into online assessment, please contact firstname.lastname@example.org.