Research and Trialling
ACARA is conducting a broad research program
into the transition from paper-based to computer-based assessments. The
purpose of the research is to provide evidence-based information to
education ministers and the broader education community about the
possibility of delivering the NAPLAN in a computer-based, or ‘online’,
environment.
The research study consists of three elements: a pilot research study, a trialling study and a linking study.
Pilot research study
From September to November 2012, ACARA
conducted a pilot research study into online assessment. Approximately
125 schools across Australia participated in the study. The study used
literacy and numeracy questions from NAPLAN and questions from NAP
sample assessments in civics and citizenship (NAP-CC).
The pilot research study comprised both a mode effect study and a cognitive interview study:
- The mode effect study investigated how the performance of items (questions) is affected by the mode in which the test is delivered, that is, whether items perform differently in paper-based and computer-based tests.
- In some schools, researchers conducted cognitive interviews with students. This study was a small qualitative study intended to gauge levels of student engagement with the computer-based assessments.
Trialling study
In August and September 2013, ACARA trialled online delivery of a tailored test design. The tailored tests provided a multi-branching test consisting of interlocking sets of questions. Students answered an initial set of questions and were then directed to the subsequent sets of questions based on the accuracy of their responses. Students with a high number of questions correct were directed to more challenging questions. Students who had a lower level of achievement in the initial set of questions were directed to questions that were less challenging. The tailored test was expected to provide better assessment of individual student achievement relative to ability.More than 250 schools across Australia volunteered to participate in the study. The study included remote schools as well as a trial with a small number of home-based and geographically remote students. Approximately 23,000 tests were delivered to students in Years 3, 5, 7 and 9, in reading, numeracy and writing. The research study also investigated the cognitive and behavioural engagement of students with the tailored test and the effectiveness of the proposed test design in better targeting underperforming students. ACARA is now analysing the test data and a range of other information collected, including observations by invigilators and technical issues experienced in schools.
ACARA’s research has also investigated autoscoring technology and technically enhanced items that are designed to assess particular aspects of literacy and numeracy aligned with the Australian Curriculum.
Linking study
In 2014 a linking study will focus on the linking of paper- and computer-based assessment scales. The scope and precise timeline for this study will be determined by the outcomes of the pilot research study and the trialling study.Research findings will be made available progressively.
For more information about ACARA’s research into online assessment, please contact onlineresearch@acara.edu.au.
