National Assessment and Surveys Online Program: tailored test design 2013 study
‘We will get better quality results that are returned quicker, which will benefit governments, schools, parents, teachers and students. We expect to see more engaged students and better informed teachers and parents.’
As students concluded the trials of online tailored test design, they reported feeling more positive and accomplished, regardless of their abilities and educational background. The trials also provided an opportunity for students to be assessed by tests catering to their needs, as well as more accurate and timely diagnostic information about student learning needs.
Figure 1: Tailored test design (TTD). This figure illustrates testlets and six test pathways available in TTD
In a related study, students from 16 mainstream schools participated in structured interviews to ascertain how they reacted to the rising and falling pattern of item difficulty, a crucial feature of the tailored test design.
The same methodology was used in a separate study that investigated whether the tailored test design can accommodate the assessment needs of students with socio-educational disadvantage.
- Results of the tailored test design studies show that the delivery of multistage branching tests for NAPLAN online is sound and feasible, and that these tests offer better measurements of student performance, particularly for high- and low-achieving students. The results show that the current measurement model can be used to construct a NAPLAN online measurement scale.
- The psychometric analyses also show that further work is required to finalise the measurement aspects of the tailored test design; in particular, testlet boundaries require further refinement.
- The tailored test design and the proposed branching mechanism work effectively to adapt to the different ability groups. Consequently, well-targeted tests can be administered to different ability groups, thus increasing measurement precision.
- The investigation of cognitive and behavioural engagement of students with the tailored test design showed that multistage testing will provide an opportunity for all students to be assessed by tests catering more fully for their assessment and learning needs.
In August 2014 ACARA began the third phase of its research program with the aim to further refine the measurement aspects of the tailored test design for reading and numeracy tests. This study also includes trialling of grammar, punctuation and spelling test items delivered online, including students listening to spelling items through headphones. The results of this program will be available in 2015.